Getting the hands dirty with license plate recognition

It's not often I get a chance to "get my hands dirty" with an emerging technology, so I jumped at the opportunity this morning to see one of the newest security product offerings in our industry. Milestone Systems, a Danish company that has created open platform video management systems, is hosting its Milestone Integration Platform Symposium in Puerto Rico this week, and I popped my head into a training class on their new XProtect analytics offering for license plate recognition (sometimes also called Automatic Number Plate Recognition, especially if you like fish and chips and your afternoon cup of tea).

First, a couple notes about this product - it's not officially launched yet, but Milestone is OK with you hearing about it (expect official launch fairly soon). This solution is a tie-in from a company called Dacolian out of Holland. This is Milestone's first real foray at offering a direct-embedded "analytics" system (license plate recognition is most definitely part of "analytics", even though most people think of analytics as virtual trip wires and abandoned object detection). The company is, of course, still strongly partnering with some of the big names in analytics, and that's clear with some familiar faces from Vidient, ObjectVideo and Cernium presenting here at the Milestone symposium. Those companies and others can do a lot more than identify license plates, so it's not really a competitive squabble, in my humble opinion. Nonetheless, it looks like this license plate recognition software add-on is the first in a line of analytics products that would be available directly from Milestone.

So let's get back to when I was getting my hands dirty, courtesy of the Milestone tech guys Morten Lundberg and Rasmus Lund. The Milestone license plate recognition system essentially takes a stream of video (in M-JPEG format) and splits it into a series of images (pretty easy to do, considering this is Motion JPEG). The system then sends that license plate data through the license plate recognition (LPR) administrator program where it identifies plates. The LPR unit then sends that plate info to XProtect Transact's server for storage as metadata.

That's right, XProtect Transact - a Milestone solution that I had thought of as only designed for retail environments - is at its core a system for linking metadata with video. That metadata could be point-of-sale information (as it is typically used in retail environments), but in this case it's the character sequences of license plates. The system requires users to have XProtect Enterprise or XProtect Corporate software (XProtect basic isn't going to cover you here), but it's not particularly processing intensive; in fact, I was running the whole system (XProtect Enterprise, Transact and the LPR add-on) on a rented Lenovo laptop. Admittedly, that's probably not the preferred way of using processing power if you have a number of cameras, but it worked for the purposes of a simple demonstration.

As you might expect, when setting up license plate recognition in the Milestone/Dacolian system, you're able to make a number of configurations. You can set the accuracy threshold (I set a minimum of 60% confidence, but most of my matches were in the mid-90s percentile, and none were below 85% in this small test case). What that does is tell a system that if you've got less than your minimum confidence factor, it doesn't mark that as a recognized plate in the system. Of course, you're still recording the video so you can manually go back and review video where no plate was indicated, but it seems the goal here is to not dump in a bunch of plate recognition "guesses" if, perhaps, you're dealing with a license plate that's been through a mud-wrestling match or is covered with that cold, white stuff we call snow. And while I'm here, I should note the maxim that Rasmus offered: "When we speak of analytics, 100 percent certainty is more theoretical than practical." Consider those as words to live by if you are contemplating analytics.

To match a plate and determine the confidence of the match, the system relies on a number of pre-defined license plate styles for countries around the world and U.S. states. Some of these states have many (sometimes 30) pre-defined templates. That's designed to cover the specialty vanity plates being offered today, and it sounds like Dacolian is staying current with that and trying to add more and more of these bizarre plates that state tax departments are selling as revenue boosters. It's not just the U.S. that is plagued by specialty plates; Morten tells us that in his home country of Denmark, where the total population is 5 million (about the size of the metro Atlanta) they have 42 different types of license plates.

In my example for the demo purposes, I had a number of photocopies of how Texas license plates look on American cars. The blurry photocopies were about as good a testing ground as we could create in the class setting, shy of actually setting up a beta test in front of the hotel.

So, after setting up the country/state definition of the plates I wanted covered, I moved over to a setting where I could define the capture area - after all, there's no sense making your software work hard trying to try to read text way up in the air, when your license plates normally appear on a bumper a couple of feet off the ground. I also looked at settings on video resolution of the cameras incoming stream (I was using a Sony IPELA fixed camera model). I also had an interface to control how long I wanted to store that LPR data (some countries have limits on how long you can store this kind of personal data) and a number of other nuanced settings like a pixel-based setting to set general rules on how big the plates appear in the image. Mostly this was fairly intuitive on how to configure the settings; as a bonus, the GUI gives you a nice still image from your video stream so you can have a point of comparison as you do your configuration.

Quickly enough, I had the system recognizing license plates (both photocopies and some real plates that I could hold in my hands). The Smart Client (the Milestone end-user software/front-end for video management) links the video with license plate capture for your review. Mission accomplished, I thought, but we weren't done yet. A cup of coffee later, we were tackling the challenge of how to set up alerts - after all, one of the chief battle cries from the video analytics camp is that you can actually "DO SOMETHING" with your CCTV system, other than just let the video streams eat up you NVR/DVR/VCR/server storage.

Back in the system, I keyed in a couple plates and set a simple rule on them. If any those plates were recognized, then I had the system mark them as "disallowed." My idea was to set up an LPR system that would block certain vehicles from your parking garage, for example. Soon enough, but not until after realizing that I had been boneheaded and skipped a step, I had a system allowing and disallowing license plates and alerting me when the system had spotted a "disallowed" license plate. That's just one example of how you could set up a rule; I suppose there are scores of possibilities. Maybe you want to look for certain plates and link that with a gate access control system to allow them past a toll booth or into a special parking lot (for your execs, perhaps?); be creative.

I want to give you a final note as you think about applications. The systems are usually best applied for slow or stopped traffic, though that's not to say there's not a vendor out there with a system that can pull license plates off SUV s jam-packed on Interstate 40 when they're moving at 70 mph. Generally speaking, this is going to be better at slow situations like toll booth payments and parking lot access control. Of course, some kid is working right now at MIT or Cal-Tech and might just have that 70 mph system figured out next week. You can hold your breath, but I'm not going to hold mine for that promise.

So what did I learn - other than that if a trade journalist can set up a license plate recognition system, then so can you? I've got a few notes here on tips of the trade in setting up license plate recognition systems, and I think these generally aren't particularly vendor specific. Let's have at it:

1. Get fixed. By that, I mean you shouldn't be using autofocus cameras to try to recognize license plates on moving vehicles. That's a recipe for disappointment, so get a fixed camera instead.
2. Put your eye in the system. Just because the powerful human eye and brain can read a license plate at a horizontal angle of 70 degrees doesn't mean you should yet expect an LPR system to do the same. As Morten says, "A good image for the human eye might not be a good image for the software."

3. Do one thing and do it well. It's probably not advisable at this point to consider making a general surveillance camera capturing a scene also be the camera your system depends upon for license plate recognition. Get a dedicated camera for this.
4. Resolve to be resolute. On any system, there is a minimum number of pixels needed to accurately recognize letters and numbers. Talk to the vendor and find out what that is, and then make your camera choice, zoom, and position applicable to that. The system I was using needed about 130 pixels in width for a U.S. license plate.
5. Don't buy junk. A good, optically correct lens and a camera known for producing good quality images are essential here. In other words: Garbage in, Garbage out.
6. Contrast. When you're examining image contrast and lighting, consider the contrast at the typical plate location. And if you have to weigh contrast versus saturation, Morten suggests you lean towards low contrast if that means you have good saturation.
7. Choose your stream. The system I toyed with used Motion JPEG (MJPEG), which is really a stream of standalone JPEG images. MPEG4, as you probably know, sends only key frames and change data, so there's no standalone frame for a system to compare it to. Most IP cameras come with MJPEG and MPEG4. Select MJPEG for license plate recognition applications.
8. Get the angle. It's hard to read things from an angle, though the systems I've seen have done a surprisingly good job at odd angles. But for best results try to get your camera as "straight-on" as possible.
9. Get lit. Poor lighting isn't a killer, but it's not a recipe for success. The systems today are pretty good at still getting plate data in poor lighting, but it's going to affect accuracy and confidence in the correct capture.

10. Get focused. Focus, like in all CCTV applications, is key. Focus on the plate, and keep in mind a potential focus shift when switching from visible light to an external infrared light source. As a side note, many LPR companies seem to recommend IR cameras for recognition.
11. Move quickly. Get fast with your shutter speed, or at least as fast as you can. Why? Because if you don't have high shutter speeds, your images may going be blurry from motion. Fast shutter speeds stop the motion better and give you a higher quality image from which to recognize the license plate characters. One of the math rules for this is that the maximum time that the shutter should be open is equal to 1/(speed in miles per hour x 18) seconds. For kilometers, it's equal to 1/(speed in km per hour x 11) seconds. So, if the car is moving at 10 miles per hour, the maximum shutter speed is 1/180th of a second.
12. Weather matters. As I suggested earlier, and it's pretty much common sense, plates covered with snow, ice, mud, dirt, and heavy road grime are going to be more difficult to recognize. That's just part of life.
13. Don't lose out. With JPEG imaging, you can set the compression level. But the more compressed it is, the more "loss" you have, and the more likely JPEG compression artifacts (odd pixels) are likely to appear. Those artifacts can influence recognition systems. The 80 percent level was the minimum the Milestone tech guys recommended.
14. Go natural. No augmentations or enhancements here - software or camera-based image enhancements are similar to JPEG compression; they can introduce image elements that just aren't really there. Avoid image enhancements if possible.

That's about it - have fun capturing license plates on America's roads and parking garages.