When it comes to the subjects of network data throughput capacities and hard disk storage capacities, the “your mileage may vary” disclaimer applies. What you can achieve in terms of system performance will vary depending upon many factors. However, when planning and designing video networks and video recording storage equipment, you have to have some sound basis for your capacity specifications and design choices.
Q: We bought an 80 TB storage array to upgrade our video storage, but the integrator’s service technician told us that we would get 60 TB of “effective storage”. What does that mean? Is it the fault of the VMS software?
A: It is true that you will be able to store something less than 80 terabytes of recorded video data, and that’s not the fault of the VMS. Similarly, you will be able to transmit something less than a gigabit of video data across a 1 Gbps network connection.
Complex but Understandable Picture
If you have a drinking glass that can hold 10 ounces of water when filled to the brim, it is true that you can fill it with 10 ounces of water. The trouble is that you won’t be able to carry that glass, tip it towards your lips, or even put a drinking straw in it without spilling some of its contents. If you just want to put in a drinking straw, then you only need to leave a little bit of extra room in the glass, so the “actual” capacity may be 9.8 ounces. However, if you want to carry the glass across the room without spilling its contents, you may want to fill it to 80 percent capacity, which makes it an 8-ounce glass for practical purposes. If you have to walk upstairs quickly with the glass, you may want to fill it only halfway to be safe.
Another way to think of it is in terms of “packaging”. If you want to send a letter by overnight courier service, you have to put it in a letter size envelope. A stack of 10 one-page (8-½” x 11”) letters on your desk will take up only a fraction of the space that 10 courier envelopes take on the delivery truck. The letter must also be protected in transit with a cardboard envelope that’s 9” x 12”, along with a plastic pocket attached to hold a multi-sheet shipping slip that manages delivery. Space in the corporate mail room must accommodate the volume of packaged letters that business requires. So you can’t count just the space the letter takes; you have to account for the container as well.
Data Transmission and Storage Requirements
Both of the analogies above apply to the transmission and storage of video data. The video’s electronic data is “packaged” for delivery, storage and retrieval. Think of this packaging as “electronic overhead” required for data handling. The explanations below are oversimplifications, but are sufficient to explain why “raw” capacities and “net” capacities differ.
Network Data Handling
In an Ethernet network, the data is electronically wrapped in an “Ethernet packet”. In addition to the specific data that’s being sent, the data packet contains information that include markers for the beginning and end of the packet, a sequence number to identify the packet, data that specifies the format of the packet (this has evolved over time), identify the hardware address (MAC address) of the sending and receiving computer’s network cards, plus an error-detecting code that allows a network switch or receiving computer to identify a corrupted data packet, discard it, and request retransmission from the source.
There are many more technical details, but this gives you some idea of basic requirements for handling network data. If the communications are encrypted, there is even more overhead involved. For a small bit of original data, the total number of data bytes to transmit it could more than double the overall size of the transmission. If a packet has to be retransmitted just once because of electronic data loss (momentary glitch in the network electronics), then the total number of bytes required to successfully send the packet doubles. This is why the quality of the entire network installation is very important. The network data throughput capacity will be limited by the lowest performing part of the network path. Because Ethernet networks will retransmit damaged packets, even a poor quality network may be successful in getting data 100 percent through to its destination. However, the overall capacity of the network will be diminished due to the overhead of retransmission.
This is why a 1 Gbps network path cannot transmit 1 Gigabit of actual video data per second. In theory it will be closer to 0.8 Gbps, and in practice will be something less than that.
Storage Data Handling
Just like the shoe storage room in a shoe store must be organized in to specific sections and rows, with shoes in labelled boxes, data must also be organized and labeled so it can be easily stored and retrievable once it is stored. This happens at several levels; here are four commonly found:
- hard drive
- hard drive controller
- operating system
- VMS database
The hard drive will have a table containing the list of good (okay to use) and bad (not for use) sectors on the drive. The bad sectors, plus the table to record them, take up drive space. If the hard drive controller is configured for RAID storage (redundant array of independent disks), data is written across multiple disks with error-correcting data. RAID 6, contains a double set of error-correcting data, so that the array of drives can continue to function even if one or two drives has failed. The error-correcting information takes up drive space as well. The operating system formats the drive with a file system (such as the Windows NTFS or Apple HFS Plus), providing a directory of file names with associated file attributes (such as “read-only” and “system” labels), plus file folder information, and information about the locations on the drive where the file’s data is stored. All of this file management information consumes space. Then there is the VMS database itself, which is often a SQL database containing information about the files and where each hour’s worth of video data is stored for each camera.
This is why, by the time an 80 TB set of hard disks is installed and fully formatted, the amount of video data that may be stored could be as little as 60 TB.
The next two columns will provide design rules of thumb for sizing network capacity and drive storage space to achieve the video data throughput and storage retention that your security video system requires.
Write to Ray about this column at [email protected]. Ray Bernard, PSP, CHS-III is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities. For more information about Ray Bernard and RBCS go to www.go-rbcs.com or call 949-831-6788. Mr. Bernard is also a member of the Content Expert Faculty of the Security Executive Council (www.SecurityExecutiveCouncil.com). Follow Ray on Twitter: @RayBernardRBCS.
