When the state shut down its online license tab renewal system in April after two legislative audits discovered serious security flaws, legislators blamed department heads, department heads blamed staff, staff and investigators blamed budget cuts, and nearly everyone blamed the rush to pioneer a new convenience for Minnesota motorists.
However, a Pioneer Press review of hundreds of internal state memos found that managers, including high-level officials, knew there were problems with the system but either ignored the many red flags or were hamstrung by cost constraints.
In the end, the Driver and Vehicle Services division made ease of use a higher priority than security, never believing anyone's personal and financial data were at risk.
For the first time, interviews and state documents detail the deep divisions and bitter back-and-forth that simmered for years within the agency a part of the Department of Public Safety over the system's perceived shortcomings.
Numerous staff and top managers knew as early as 2002 of critical security problems with the site, including evidence of hacking, but ignored them or did little more than talk about possible solutions.
Responsibility for implementing and monitoring security software and protocols was dispersed among different divisions, so tasks often weren't done.
Jobs went unfilled and outside security consultants were not hired because of the state's budget deficit and belt-tightening.
Just one manager was placed on administrative leave because of an internal investigation into the matter. She resigned in August.
To date, there is no evidence that anyone's name, address, phone number, credit card information or bank account information has been stolen.
Department of Public Safety Deputy Commissioner Mary Ellison said no one intentionally ignored security.
"You enter into things under the mantra of customer services and then you realize, well, I guess this is something we should have paid more attention to," she said.
In October 2000, Driver and Vehicle Services rushed to roll out the system; it was one of the first applications of e-government in Minnesota. Officials put the cost at less than $1 million, and most of that was staff time. Its popularity grew, and by last year nearly 30 percent of renewals were done online and the system was generating revenue of $30.5 million.
At the time, Minnesota landed in the top 10 states using e-government in an annual survey by Brown University. Today, Minnesota has plummeted to 42.
During the first year of the site's operation, the legislative auditor uncovered 10 security flaws in the system and made 17 recommendations to tighten it.
Officials took action on about six of those recommendations. But other changes were not completed until this past April, just as a second legislative auditor's report was being released, according to state documents. Five problems directly related to protecting consumer information were not fixed.
Pat McCormack, director of the Driver and Vehicle Services division, and her boss, Ellison, still can't or won't say what went wrong between the two audits. McCormack said that she thought everything was fixed but that "as a nontechnical person," she wouldn't know if something was not done.
After an April legislative hearing, Ellison said she was checking to see whether any staff members lied about the work done after the first audit. Now Ellison won't comment on what was found.
Keith Steller, an information-technology consultant and network security teacher at Inver Hills Community College, said the mistake governments and some private companies make is not considering security as part of the customer-service package and ignoring warning signs. The state had the equipment, but not the policies and procedures to make it work, he said.
"How much value do you put on your data?" Steller asked. "How much does it cost to fix it now? And how do they get their reputation back? The lesson is, you should do it right the first time."
Wisconsin, for example, seems to have gotten it right. The state put its license renewal system online in 1990 and established a firm firewall and scrambled the charge card numbers as part of its security system, said Candy Dyhr, financial management supervisor for the Wisconsin Department of Transportation's division of motor vehicles.
The system was audited twice by the Wisconsin Bureau of Audit, which found no problems, Dyhr said.
Minnesota now hopes to have its system back up by later this fall at a cost of $1.1 million. McCormack said she is not certain if any money would have been saved along the way. With constantly changing technology, more upgrades would be needed by now, she said.
There were many missed opportunities along the way to bulk up security.
Daren Mehl, then a St. Paul resident and computer technology specialist for a bank, first warned the division by e-ail in February 2003 that even the mildly computer-savvy could get in and steal the private financial data of anyone who renewed license tabs. He busted into the system himself and left proof he had been there.
Nobody took him seriously, he said. All he received was a generic response that the site was secure but for technical reasons it couldn't display the padlock icon that normally appears somewhere along the edges of the Web browser. Such a symbol is supposed to indicate that intruders can't access any information that users are asked to supply.
"I told all my friends to stay away from it," he said. "I can't believe they weren't hacked."
Mehl was one of hundreds who complained. Robert Bennett, who at the time was a state information security officer, took user complaints and warnings seriously. In late 2002, he fired off an e-ail to a department head questioning whether the site could continue successfully "should the trust in the electronic system which we know is not secure falter from where it even is now."
Bennett said he and Janet Cain, the department's chief information officer, pushed for hiring an outside company in 2001 and 2002 to scan the system for holes and weaknesses.
"We never received budget approval," he said.
By early 2003, McCormack, the division's director, and others again discussed hiring an outside firm. One company offered its service for $1,000 and offered to waive the fee entirely if it failed to find any security vulnerabilities. Ultimately, no one was hired and the work was not done.
"We had some changes in personnel that were going on, and the ball got dropped," McCormack said. "I'm not going to make any excuses for the fact that it should have been addressed I certainly would say I should have been one of the ones to ensure that we tried to keep moving on that. "
Even without outside help, other red flags could have been heeded. The Driver and Vehicle Services computer system had software that recorded when hackers were trying to illegally break into the computer, said Chris Buse of the Legislative Auditor's Office. However, no employee was responsible for checking the log daily, according to the first audit.
Marc Klein, public safety's network operations manager, wrote in a May 22, 2001, e-mail that he would have someone hired by Jan. 1, 2002, to review the computer logs. That didn't happen, according to the second audit, and the logs were sometimes checked days after someone might have jiggled the handle, looking for an unlocked door into the system.
Klein turned down repeated requests for an interview.
However, Cain said Klein hired two security specialists in succession, but each left the job after just a few weeks, and then it was left vacant.
"I don't know if they looked into those logs because they may not have been the highest priority," Cain said.
Resources must be thrown at intruder prevention, Steller said. At the least, a security engineer should have been assigned to spend four hours a day reviewing the logs, he said.
With no centralized technology branch, the public safety department allowed its various divisions, such as the Bureau of Criminal Apprehension and Driver and Vehicle Services, to control their own computer systems.
The result was that computer-safety experts could make recommendations but not enforce them. For example, Bennett worked for the Department of Public Safety and received many of the complaints and warnings about the online renewal process, but he had no authority to order changes.
"The problem with decentralization, it's like dancing with five people at the same time," he said, referring to the five divisions with computer technology staffs.
Security issues were an afterthought for the license tabs computer people, said Buse, the audit investigator, and there is no job category in the state for computer security.
"We don't have a lot of security professionals," he said. "We have database managers who are supposed to be doing some security."
Generally, security engineers can have a lot of titles, and in a pinch they might also be network engineers, computer expert Steller said. But database managers or computer application managers are never security engineers.
Compounding the problems, Bennett said, was frequent turnover in the divisions and in the Office of Technical Support Services where he worked, forcing him to do more with day-to-day operations than strategic security for the agency. He quit in August 2004 to do computer security work at a private company. Budget cuts had reduced the information security unit to just Bennett, and he was not replaced when he quit, the second audit pointed out.
Individuals in each division were responsible for installing security patches, a software program that counters hackers. With no centralized security group monitoring the work, the legislative auditor found, patches weren't installed on many computers, something Bennett pointed out in an October 2003 e-mail.
"Those were not up to date," Cain said. "Why that didn't happen, I don't know."
Digging into how the failures occurred or who was to blame has been a low priority as officials focused attention on fixing the system and getting it running again, deputy commissioner Ellison said.
"I think as a department, we are not trying to dodge responsibility and say everything we did was perfect," she said. "But we're also not trying to look for some villain. We're saying we regret this happened. It was no one's intention to deliberately harm anybody."
However, the division's computer coordinator, Judith Franklin, became the subject of intense scrutiny as the second legislative audit wrapped up last spring. She is the only employee who was disciplined as a result of the security flaws, said Gary Denault, executive director of the Middle Management Association, Franklin's union. She was placed on paid administrative leave in April and resigned Aug. 10.
In an interview with the Pioneer Press, Franklin said that she was made a scapegoat for the system's problems, that she was singled out because of personality conflicts and that she did good work.
Franklin has worked with computers since the days of the large mainframes in the 1970s at university jobs from Oregon to Minnesota. When she joined Driver and Vehicle Services in late 2000, her main role was to oversee the growth of the online renewal system and simplify it for customers. She worked through the night fixing division machines and cleaning up software when the computers were hit by outside viruses, the documents show.
At least three state computer technicians grumbled about Franklin since 2002, according to e-mails. Bennett, for instance, prodded people to fix a security gap and in an e-mail said, "I have informed Judith of it, and have never received a response."
Out of the hundreds of state documents reviewed, about five were complaints about her job performance. Supervisors did nothing until the second legislative audit in 2005 when another five e-mails indicated she had been stripped of responsibility.
She disagreed that the two audits revealed serious problems. She said "most of the things aren't right or wrong" but security had to be balanced against ease of use for customers.
She purchased security software last fall, but it had not arrived before the second audit, Franklin said. Her former boss at the division, Brian Lamb, praised her work, saying she "was brilliant," understood the systems and explained them clearly.
McCormack and Ellison refused to comment about Franklin, even though she has quit the agency, and they refused to say why no one else was disciplined.
"We can't comment on any specific staff," Ellison said. "We are fixing the problem."
The department opted not to perform a forensic audit on the systems that might reveal whether personal information was stolen. Immediately after the computer system was taken offline, officials from Gov. Tim Pawlenty on down said that no Minnesotan's personal data had been stolen or compromised by hackers.
However, auditors and others responded that there was no way to know whether that was true without doing a sophisticated audit of every command and action the computer made before the shutdown.
After researching the issue, McCormack said Driver and Vehicle Services did not hire an outside firm to do such a probe because it would not offer conclusive proof. She said the legislative auditor agreed.
If someone thought his personal data were stolen, he surely would have called the department by now, Ellison added.
"All I can tell you is that people complain all the time," she said. "Because there are millions of people who deal with Driver and Vehicle Services and they complain about everything else and they haven't complained about this."
(c) 2005 Associated Press
