Storage: Compression Considerations

Nov. 10, 2010

The IP surveillance market has boomed over the past decade with dozens, if not hundreds, of new players emerging and traditional CCTV manufacturers expanding into networked technologies. With the avalanche of new companies and technologies cascading over into the industry, there is no wonder there is confusion surrounding the proper configuration of networked-based security technologies and what is necessary for a successful and reliable deployment.

Resolution, compression, frame rate, retention time and motion in the field-of-view are some of the main factors to consider when configuring an IP-based system. While most of these factors are pretty straightforward, compression is the most confusing and daunting to address. What is MJPEG, MPEG4, H.264 or J2K? Is it lossy or lossless? Just what do all those terms mean and what’s next to come on the horizon?

Defining compression in integration
Video compression is very complex but described simply, it removes redundant data from the field-of-view to make video file sizes as small as possible to enable users to store as much data as possible. Furthermore, video compression technologies save time and money associated with storage and servers.

Each compression scheme is an algorithm used to accomplish data reduction. Each algorithm delivers a different amount of stored data for the exact same raw video. There are two primary types of compression: lossless and lossy.

Determine video needs appropriately
How can you choose which compression is best for a particular application? Two factors come to mind and both ultimately come down to costs. The first cost factor is insurance and is difficult to determine. This insurance refers to the data recorded and if it can be used for evidentiary purposes. If a user requires the original video data for legal reasons then lossless compression is what is needed. If not, lossy compression is an ideal fit.

Lossless data compression is an algorithm that enables the exact original data to be reconstructed from the compressed data. An example of lossless compression is JPEG2000, although there are some versions of JPEG2000 that are lossy compression standards as well.

Lossy data compression actually discards some of the data during encoding to deliver a much smaller file size. This is very useful from a storage perspective, but can be an issue when recreating the video stream. Most of the commonly used compression algorithms are lossy and do a fine job of recreating the data for general purpose surveillance needs. Examples include MPEG4 and H.264.

The second cost factor is the storage and servers required. There is a substantial trade-off in terms of storage costs associated with different compression types. Lossless compression requires more storage than lossy compression. A VGA (640 x 480 pixels) video stream for a single camera at 10 frames per second for a single day of retention, uncompressed, requires more than 700GB of storage. The same configuration with MJPEG requires approximately 75GB; MPEG4 equates to 10GB; and H.264 needs 4GB.

Cost effectiveness explored
The cost per gigabyte has decreased significantly over time and can be less than one dollar per GB for the average end-user. This is due to innovations in software-based RAID protection and use of commodity hardware to accomplish what previously required proprietary hardware and fixed RAID controllers. The flexibility offered to end-users with next-generation storage servers is significant and provides exceptional data and software protection at a fraction of the price.

For the server side, there are two other factors to consider when deploying lossy compression. They are scene motion and object speed. Lossy compression will certainly save money on storage, but may require significantly more processing power (more processors, more money) to perform the task. The more motion in the field-of-view and the speed of the objects, the more processing is required when using a lossy compression algorithm. The algorithm is basically estimating the location of those objects moving within the field-of-view and their speed when compressing (encoding). It then must recreate the stream when accessing data (decoding). In areas of significant changes from static images to high motion/high speed, processing requirements become overwhelming. There are also visual effects that occur when systems are not configured to take higher processing needs into account including ghost images, artifacts, storage spikes, bandwidth overconsumption and video corruption.

It is interesting to discover that each camera manufacturer’s products perform differently depending on the type of video management software and the compression algorithm. The differences can be vast. In order to make certain there is an accurate server and storage configuration, you need to verify these details with not only the video management software provider, but the camera manufacturer. Storage server suppliers are also a great resource as theses companies test and benchmark the various open architecture video management systems available to determine the most effective hardware platform. These vendors typically have more expertise in data payload to disk delivery and processing requirements.

With all of the options available for compression, with price as the primary consideration, the lossy schemes are most common. The surveillance environment helps determine the best compression fit for the needs of the user. Lossless compression provides a higher quality image and better evidentiary data than lossy compression and should be considered in appropriate environments.

About the author: Kevin Klemmer CISSP PSP, is director of Sales, Pacific Northwest/Western Canada, for Pivot3. An ASIS member since 2000, he obtained his PSP certification in 2003. He previously held the position of Chairman of the Puget Sound Chapter for 2007 and 2008 and is currently Assistant Regional Vice President for ASIS International Region 1A. Klemmer holds the CISSP credential: Certified Information Systems Security Professional.