Many in the cyber world are calling it America’s IT Pearl Harbor, while others see it more like the Greeks using the Trojan Horse to enter the city of Troy in the waning days of the Trojan Wars. No matter how they characterize the event, every expert across the cybersecurity landscape admits that the devastating Russian hack of multiple federal agencies and more than 18,000 government and private networks accomplished by subverting the security protocols of SolarWind’s proprietary Orion network monitoring software was a wakeup call the United States cyber community must swiftly address.
According to a ZDNet blog posted on January 6, the U.S. Department of Justice is confirming that the SolarWinds supply chain hackers targeted internal DOJ networks and accessed an estimated 3,500 agency Outlook365 email accounts. Those accounts have since been blocked, however, the DOJ finds itself in good company as other Federal and private entities that were breached as early as March 2020 but not discovered until early December when the cybersecurity firm FireEye disclosed it had been attacked, which included federal agencies like, the U.S. Treasury Department, the Department of Commerce's National Telecommunications and Information Administration (NTIA), the Department of Health's National Institutes of Health (NIH), the Cybersecurity and Infrastructure Agency (CISA), the Department of Homeland Security (DHS), the State Department, the National Nuclear Security Administration (NNSA) and the U.S. Department of Energy (DOE). Others hacked included three state governments and several city governments, along with companies like Cisco, Intel, VMWare and Microsoft.
"The first part of the breach relates to SolarWinds themselves being breached. While we still don't fully understand how the threat actors gained access to the SolarWinds source code, it is assumed that their Git repository was breached. It is normal practice for companies to review code when committing to the master branch by a second person, but it is not common practice for periodic reviews. Therefore, once the code was changed and committed, it is unlikely the change would be noticed in the code. Once the malicious code was executed on the end customers' servers, there was a delay timer. This is a common tactic called sandboxing used by attackers to avoid detection,” says Danny Jenkins, the CEO/Co-Founder of ThreatLocker, an Orlando-based cybersecurity firm providing zero-trust endpoint security.
Espionage, Plain and Simple
But make no mistake about the motives, says Christopher Painter, the former State Department cybersecurity coordinator under both the Trump and Obama administrations and a globally recognized leader and expert on cybersecurity, cyber policy, cyber diplomacy and combatting cybercrime who talked with SecurityInfoWatch recently, this attack, and others he feels will surely uncover nothing short of good old fashioned Russian espionage in the coming months.
“In terms of finding out who the other victims are, the extent of the penetrations, the nature and volume of the material that was taken, whether, as it appears right now, this is an act of espionage or if there was something more going on like prepositioning. All those things are unclear. But it does look like an act of espionage, and all the reports are it's Russia, and particularly the Russian intelligence service, so that makes it more like an espionage campaign,” explains Painter. “
Paul Joyal is uncomfortably familiar with Russian spycraft and its consequences. Joyal is a long-time consultant on security and Russian affairs. From 1984 to 1989, he was director of security for the Senate Intelligence Committee and worked on Soviet counterintelligence issues during President Ronald Reagan’s period. He worked closely with former Soviet Foreign Minister Eduard Shevardnadze when he was President of the Republic of Georgia on security matters and integration with the Euro-Atlantic Alliance and decorated Georgian Order of Honor for his service. In 2007, Joyal told Dateline NBC that the murder of former KGB agent Alexander Litvinenko served as a warning to all critics of the Putin government. Days after that interview, Joyal was shot and wounded outside his home in Adelphi, Maryland. The attacker(s) have never been found, but he is certain who ordered the hit.
“The use of foreign-owned offshore companies to provide software engineering is a great threat. SolarWinds used firms in Belarus, Poland, and the Czech Republic. JetBrains is a company founded by three Russians. It had a laboratory in Russia. This is so big a breach that re-establishing identity trust will remain an open question. Once an adversary of this capability gets in, they will then burrow in deeper and finding that is extremely dangerous,” asserts Joyal. “As I have written recently, I believe the initial breach goes back to 2019. The main objective was espionage but if any critical infrastructure was touched by a system known to have been breached by this hack then we must assume the possibility of larger spread and danger. It can’t be ruled out.”
Politics Weakens U.S. Response
And it is the Russian angle that Painter maintains deterred the appropriate U.S. cybersecurity response, with the roots of the neglect beginning following the 2016 presidential election. Politics and the lack of serious cyber policy over the last four years exacerbated the nation’s weak security doctrine.
“We have not, in any way taken our eye off the ball, given the amount of money that we've put into this, which is still not really enough, and the amount we've emphasized things like some of the strict DOD activities, the persistent engagement and defend-forward and hunting on networks. We just didn't see this coming. Many of the computer-security companies didn't see this until after the fact, which is what happened when FireEye discovered it after their tools were taken.
“This is a very big failing, and it does indicate that we have not yet done enough. This has been a problem for some time, but even more clearly so during this administration. We have not yet made cybersecurity a core issue of national security and we haven't taken it seriously. We still treat it too much like a boutique technical issue when we need to treat it as something that really cores to our security overall. We haven't resourced it; we haven't prioritized it. Certainly, in contrast with President Trump, who never made this a priority,” laments Painter, citing a scene from the Bob Woodward 2019 book Rage, where, apparently, Tom Bossert, the former Homeland Security Advisor to U.S. President Donald Trump and a Deputy Homeland Security Advisor to President George W. Bush, goes in and says to Trump, "Hey, I want to talk about cybersecurity," and reportedly Trump says, "You know, I'd rather watch the Masters tournament."
Painter continues that, in his opinion, over the last four years, cybersecurity has become more politicized which has mudded the cybersecurity defense of the nation because of the lack of clear messaging from the top. He says that it is crucial to not only prioritize it for your administration and for the public but also to send the message to your potential adversaries that this is unacceptable.
“That's where we've seen real problems with President Trump who has, if anything, undercut whatever else the administration has been doing by second-guessing whether Russia is even responsible or had any culpability in a number of different incidents. That's put us in a weaker position. We've also gotten rid of some of the key positions in our government to deal with these things. The recent firing of Chris Krebs (Director of the Cybersecurity and Infrastructure Security Agency) was obviously not the best timing. But CISA, as good as it has become, and it still has a long way to go, was always playing second, third, or fourth fiddle to the immigration mission at DHS under this administration. My old job at the State Department was downgraded and kind of muddled. Early in Trump's term, the cyber coordinator at the White House was eliminated. So, we really haven't been taking steps to go in the right direction,” charges Painter.
A Hacker Tells Cyber Experts to Look Inward
As one of the most famous global hackers, Chris Roberts was impressed with the skill of the attackers, if not yet willing to jump aboard the Russian blame train, saying that tactics can be learned leaving the door open to other possible suspects. But there is no doubting the severity of the attack and its residual consequences.
“We can go all sorts of different ways on this one because, quite honestly, this was beautifully done. If I look at it purely from a very technical aspect way of doing it, the ability to go, ‘Look, I need to kick in the front door of some specific targets.’ Well, we can't do that as easily these days, so now we go, ‘Okay, what are my options? How do I get in via a third-party supply chain, etc., etc.? Well, I can work on them but some of them are pretty well locked down. So, how do I get into those organization’s supply chains?’ Which is where SolarWinds came in under the microscope. And then to spend a year and a half basically doing recon, doing analysis, doing initial exploit, then dropping something in and only pivoting where and when necessary, I mean, that to me shows they were playing the long
“There are a number of concerning things about this attack. First and foremost, somebody walked into most of our U.S. intelligence agencies left a whoopee cushion, a digital whoopee cushion on the director's chair and got the hell out of there without us seeing. It makes you think that if we’ve got multi-billion-dollar programs that are meant to stop anybody at the front door or the front gate or anywhere along with it and they didn't even burp, fart, or recognize the breach, or if they did, people weren't paying attention, then we have to look at our own architecture and go, ‘What are we missing?’” Roberts adds. “These attackers have introduced instability and our cybersecurity professionals have been unnerved. Somebody's comes in, planted a flag, and walked out. You didn't even know they were there. We got a hell of a warning shot across our bow on this one.”
How to Mitigate the Damage and Prepare for the Next Attack
With that warning shot come the obvious questions revolving around how the cybersecurity community readjusts and how it can implement the necessary safeguards to ensure agencies and organizations are better prepared.
Chris Hickman, the chief security officer at Keyfactor, a leading provider of secure digital identity management solutions, maintains that these attacks are not about FireEye, SolarWinds or Mimecast, rather they are a disturbing and growing trend of habitual breaches.
“The threat actors behind the attacks, whether they are using the SolarWinds backdoor or another, are targeting certificates and credentials. They are leveraging cryptographic assets to gain network access and evade security controls. The current trendline indicates that parts of the industry are still treating certificates as ‘just certificates’ rather than cryptographic assets that play a more important role in hardening network security. Technology alone cannot prevent breaches like this – companies need to ensure that they have in place the right controls, policies and follow industry best practices in order to defend themselves against the evolving threat landscape. Companies need to take a hard look at how they manage and secure digital certificates and cryptographic keys to better protect themselves and their customers,” Hickman says.
Painter says that one characteristic of this particular attack, acknowledging that it is still under investigation, that once the vulnerability has been planted in a large number of systems, it would beacon out and basically say, "Hey, I'm here." That then allowed the apparent Russian hackers to penetrate deeper with a tailored or bespoke package to exploit particular targets. But it is evident they were skilled enough to hide the way that was done, blocking potential flags to alert for anomalous behavior. Besides adjusting the technical safeguards, he insists that policy changes must also follow.
“I don't think there's any way you can guarantee something like this doesn't happen again. What I think you need to do is make sure that you catch it earlier, that there's resiliency and that you have ways of bouncing back, that you have ways of assessing the damage very quickly. I do think there are a couple of things that we can do now. One, expand our resource defenses more than we have before, prioritize this as an issue more than we have done previously. That raises awareness in all boats. There are also some structural changes that must occur. You might want to shake up the U.S. government in terms of how cybersecurity is done and the role of DHS and how much authority it has in dealing with other federal networks. I think those are all probably good changes to look at,” admits Painter.
“But the other part of it is espionage. People will say, ‘Well, you can't really deter espionage.’ However, there has to be a cost for espionage. In the physical world, when a spy gets caught, there's a prosecution, diplomats are expelled, or PNG'd (rendered persona non grata) as they call it in the trade. There are often economic sanctions, among which there are other things, It's not unfair to exact a cost, even for espionage. As I said, we're still trying to figure out if was just that or something more, but being stronger with potential adversaries is a part of it. It's not really a cyber issue, it's a Russia issue. It's not just a cyber issue with China, it's a China issue. We have to put this in the larger context of how we deal with these countries and use all the levers we have to make progress.”
Heed the Warning!
Joyal also says there are immediate steps that can be taken to strengthen cyber defenses.
“Right now, computer systems must be compartmented. Cloud systems should not connect with on-prem servers or vis-a-versa. DHS is providing detailed software to help companies and agencies attempt to mitigate this challenge. All systems must immediately patch, and I would go so far as to completely remove and reinstall new updated software in its entirety.
“I would urge all organizations to take great attention to this warning. I believe the SVR is behind this. If they are in your system, they will key on finding and observing what the cyber incident responders are doing. All communication on steps being taken to repair the system must not be communicated via the system's email. It must be assumed that the response team members and their communications will be compromised,” concludes Joyal. “We must also assume that the attackers are using secondary attacks on the initial target by different means. After information disclosed SolarWinds had a history of weak passwords and not always using additional authentication means for identity verification, I would assume that password attacks are also being used by the adversary. We should max out all encryption options to 256K to make things as difficult as possible for them.”
About the Author:
Steve Lasky is the Editorial Director of the Endeavor Media Security Group and is a 34-year veteran of the security industry. He can be reached at [email protected].
