Catastrophe narrowly averted in Florida water plant hack

Feb. 9, 2021
Experts say incident should serve as a 'wake-up call' in critical infrastructure cybersecurity

Cybersecurity professional have warned for years about the dangers posed to our nation’s critical infrastructure by hackers, who with only a few keystrokes, could inflict a wide range of damaging attacks that could lead to widespread chaos and even death. Last week, these predictions came perilously close to becoming reality as an attacker infiltrated the computer system of a water treatment facility in Oldsmar, Fla., and attempted to raise the level of sodium hydroxide, more commonly known as lye, in the city’s water supply to potentially lethal levels.

According to multiple published reports, the hacker was able to carry out the attack by compromising a remote access software program called “TeamViewer” that was installed on the computer of one of the facility’s employees. Fortunately, the employee was monitoring the computer and noticed the activity of the hacker before the attack could be carried out.

Andrea Carcano, Co-Founder of Nozomi Networks, which provides cybersecurity solutions to critical infrastructure operators, says that the hacker in this case appears to have been a relative novice given the attack’s lack of sophistication.       

“The fact that the perpetrator didn’t conceal his visual presence to the personnel monitoring the water treatment operation is the first signal that suggests the relatively low complexity of the attack,” Carcano explains. “Furthermore, according to the reports of the incident, the attacker increased the levels of sodium hydroxide by a significant amount, typically monitored by automated systems, which likely suggests that the threat actor didn’t possess a specific background knowledge of the water treatment process.”

However, Carcano adds that the incident demonstrates how security can often go overlooked in industrial control systems (ICS), especially those owned and operated by small municipalities or other local government bodies with smaller geographic footprints and budgets.

“Remote access, in particular, when not designed with security in mind, is often the beachhead used by remote attackers to infiltrate an ICS network,” he adds. “In this very case, the water treatment plant of Oldsmar has been using a TeamViewer instance, which apparently was accessible from the internet. While it is not known at this stage how the attackers obtained the credentials required, this incident, like many that we’ve documented in recent years, didn’t seem to rely on sophisticated zero-day exploit for its execution.”

Saryu Nayyar, CEO of security analytics firm Gurucul, agrees that situation could have been much different had the attacker been more skilled.

“The cyberattack against the water supply in Oldsmar, Florida, last week should come as a wakeup call,” Nayyar says. “Though this attack was not successful, there is little doubt a skilled attacker could execute a similar infrastructure attack with more destructive results. Organizations tasked with operating and protecting critical public infrastructure must assume the worst and take more serious measures to protect their environments.”

Christian Espinosa, Managing Director at Cerberus Sentinel, which provides managed security and consulting services, questions why remote access would even be permitted at a facility as crucial as a water treatment plant and says that security measures for such locations should match their criticality.

“Normally, risk is the impact if something bad happens times the likelihood of it happening. In this case, the impact (poisoning, possible death) to the population using the water from this facility is quite severe. The overall risk is normally manageable though because controls, such as disallowing remote access, are put in place to make the likelihood of something bad happening very unlikely,” Espinosa explains. “The challenge we are facing with these types of scenarios is that most organizations do not understand cybersecurity risk. In fact, convenience is often the primary driver for decisions with cybersecurity a mere afterthought."  

Mitigation Shortfalls

Tom Garrubba, CISO of Shared Assessments, a member-led risk management strategies, tools, and intelligence organization, says the attack should also serve as a reminder of he need to “consistently review and monitor” critical administrative accounts that have the ability control systems like those found in the Oldsmar plant.

 “With so much emphasis recently placed on hacks for the healthcare and financial services industry, an infrastructure hack such as this tends to hit much closer to home as it regards our physical safety,” Garrubba adds. “Alarms and logs for critical infrastructure systems should be reviewed and attended to constantly, and if such a hack or changes in set tolerances were to occur, a root cause analysis is imperative to mitigate such an event from happening in the future.”

Nozomi Networks Technology Evangelist Chris Grove emphasizes that traditional cybersecurity mitigation measures – vulnerability management, network segmentation, system hardening, identity and access management, etc. – may not have been enough to stop this attack.

“In many cases, and especially during this pandemic, remote administration solutions have been thrown into the mix, sometimes haphazardly,” Grove says. In some cases, the due diligence and compensating security controls haven’t been recognized. In other cases, it has. Either way, facilities should stop thinking like they will prevent cyberattacks and start thinking like they’re already happening. They may not see it, so they should be in a constant state of recovery.

“Typical cybersecurity monitoring would not have really helped in this case if the attacker came from an IP address in the neighborhood,” Grove continued. “Maybe, if the attacker was not located domestically, the firewall could have alarmed about the strange external connection. However, today its TeamViewer, tomorrow it’s an Android phone, the day after its SolarWinds or VMware. There are too many lives at stake to blanket trust all of the vendors to be safe and secure within their products and combined with cyber safe products being abused and misused by attackers, it becomes clear that the monitoring needs to go wide and deep.”

Adopting a Zero-Trust Security Model

The answer for critical infrastructure operators and other organizations, according to Grove, may lie in adopting a zero-trust approach to security.

“Once the operator realizes that nothing is to be trusted, they move towards monitoring the process itself, and the parameters being sent from all of the devices in the control room to the equipment. If the water facility in Oldsmar had this level of cyber security, alarms would have gone off the moment the values were set to anomalous numbers,” Grove says.

Additionally, Grove says many critical infrastructure facilities today do not leverage robust enough monitoring solutions to catch anomalous behaviors that may an indicator of an external attack or internal accident.

“By monitoring deep inside the process for anomalies, it wouldn’t make a difference if the attacker took over the HMI (human machine interface) used by the control room to send commands, the attacker would only be allowed them to send previously used safe values without raising flags,” he says. “Had a facility operator not noticed the moving mouse on the screen, this attack would have gone much further. That level of attention should have been automated.”

Joel Griffin is the Editor-in-Chief of and a veteran security journalist. You can reach him at [email protected].