It was not a good day for the SOC.
Warnings had been emerging over the past few days, but none of them had resulted in crisp defensive action. The team was scrambling, and activity was extreme.
But nothing was happening. No one had seen anything like this before. It was pretty likely that this was a new, active attack, but what exactly was happening? What was the attack vector? What was the target?
No one had seen anything like this before.
And that's the real problem; that and the fact that we're trying to protect an illusion. But we'll get to that later.
The Simplicity of Complex Networks
At their foundation, IP networks are simple: packets flow from source to destination based on next hop analysis by the routers along the way. IP networks are so resilient that they can overcome extensive bombardment. After all, one of the primary criteria for the development of TCP/IP was resilience during wartime. This very characteristic, however, creates no end of complexity when the objective is to make sure that the wrong packets don't get to places where you don't want them to be.
TCP/IP wasn't actually designed to block traffic; it was designed to deliver it. As a result, securing a network is actually quite difficult, and the implications of the multiple paths, failure resiliency, redundancy, organic growth, and the continual introduction of new technologies that create even greater complexity mean that understanding the reality is effectively impossible.
Consider all of the variables that are actively used for network decisions today: far beyond destination address that was originally used for routing decisions, today source address and port, destination address and port, and protocol, are the most basic of variables for network security control decisions. Those variables can be used for routing, for quality of service, and for security decisions. Of course, there are also more complex concepts such as applications and sessions to combine with other variables to give a complicated set of possible input for choices made all the way along the way from source to destination.
On the one hand, this seems to be a powerful set of ideas for making decisions. On the other hand, the truth is that complexity breeds ignorance; there is so much going on that no one really knows what all of the implications are.
Who Knows What?
Network engineers are smart; really smart. They put together incredibly complex systems that deliver billions of packets successfully to allow business to function. When we pause for a moment to consider the reality of the complexity and the reliability of enterprise networks, it becomes clear pretty quickly just how amazing the result is.
The problem is that no one can possibly understand what is actually happening, how all of the possible interactions intertwine, and understand all of the implications of that complexity.
As a simple example, consider the overall network architecture. What does your network look like?
I ask this question to a few hundred network and security engineers every year as I train them in cybersecurity. I ask them how they communicate their network to each other, how they visualize it, and how they decide where and how to deploy their security controls. I also ask them what the two most prevalent products are for documenting networks.
To answer the last question first, they recognize that Visio and Excel the two most widely used tools to document the network. After I pause for effect, I ask them the leading question: have you ever installed a router or firewall? We then talk through the typical steps of an install, including the reality that the Visio diagram that represented the intention for the implementation isn't typically even accurate through the installation. By the time the day is out, the implementation has required variations to get the network to function as needed. The plan is always to update the documentation -- later. Unfortunately, later never comes.
The maps used to communicate the vast majority of networks is a myth; an illusion. It represents the way the network was foreseen to connect and work. However, it's not accurate. It may be off by a little or by a lot. In my experience, network diagrams are typically years old. In more than one case, network diagrams I've seen have been older than major components of the network they document, including entire firewall product lines or companies that didn't even exist when the diagrams were made.
Pause for a moment to consider that truth. The very map that's used to decide where to put security controls and how to configure them is, to be blunt, wrong. It's an illusion. But, it's also the foundation of the defenses for the network. How can the defenses be right?
It doesn't help that many of the software tools used for years to map networks are more and more unable to see the entire network due to the inconsistent way the network handles different protocols and applications. Consider, for example, tools that use PING to map the network and networks that do not allow for ICMP between zones. As a result, the knowledge of the network is incomplete and often based on assumptions and expectations that may very well be false.
This reality means that the understanding necessary to protect the network is absent: no one really knows the actual layout and connectivity of the network they are working to protect. Unfortunately, they are also completely unaware that they don't know it! Most of the really smart people running networks are so busy working on protecting what they think they have that they cannot come up for air and figure out what they really have.
And this isn't going to change.
From Knowing to Doing
Now that we've admitted our existing navigation tools are inaccurate at best, let's also look at the reality that many of the people who make an effort to understand the network in order to protect the assets have limited experience and knowledge about the precise possibilities across a modern network. Many security engineers have not had extensive involvement with network or system engineering. They may understand some aspects of the overall networked system very well, but have very little if any understanding of other aspects. Security crosses all of the technology boundaries, including systems or all kinds, wireless and wired network connections, routing, firewalls, and load balancers. Consider the complexity of each of these areas individually, and then consider the multiplied complexity of mixing them all together and using the precise and arcane configuration languages of each time component of the system.
Each security professional needs to be an almost impossible combination of experts to do the role justice. This leads to the reality of a skills shortage.
The shortage of cybersecurity skills is leading to a critical situation in which the attackers significantly out-gun the defenders. From recently analyzed breaches we know, for example, that attackers wait with their break-in already complete, sometimes for months, before they destroy or steal anything.
They sit and watch. They learn. In fact, it's very likely that the attackers know far more about the compromised network than the network's engineers do. The attackers devote themselves to creating an accurate picture of a network they know they don't know. Most enterprise engineers are too busy changing, monitoring, reviewing, and planning to be able to be aware of the actual reality of their networks.
We are already seeing these issues behind the scenes of many recent breaches. It should frighten everyone that it is still the case that the majority of network attacks use known issues in known ways. The methods used in attacks show us that even simple and understood weaknesses continue to provide the majority of attack paths.
Learning
The reality of the complexity and state of skills within cybersecurity leads to two integrated approaches to address the lack: first, to develop the capabilities of those already involved in security and second, to address the limited automation currently in use by security professionals.
In order to provide capabilities so staff can actually accomplish what they set out to do, skills development is vital. It is surprising how many security professionals have limited knowledge of network protocols and the specific capabilities of network security controls. Security analysis requires such a broad range of knowledge to understand and analyze potential issues that it is very uncommon to have developed them over the course of a career, especially one that has been relatively focused on one area. Using industry training and certification, organizations should raise the level of skill they are able to bring to bear on their network and system security challenges. In addition, moving security professionals into and through parts of the operational teams of an organization will expand their experience and allow them a broader perspective as they deal with potential issues.
In addition, it is the rare organization that has the tools and systems necessary to effectively address the continuous analysis of the network environment. Networks today change constantly, and a daily change window is very common. As a result, operational processes, tools, and systems must be integrated with the input from proactive analysis to get a clearer, more defensible perspective. It is still rare for organizations to use the automated analysis to determine all possible paths through a network instead of looking at just the currently active paths, yet this analysis is essential to protect the network from the inevitable attacks as the state of the network changes due to unexpected outages, maintenance, or attack.
When we become willing to face this reality, we can then begin to decide on the paths to improve both skill sets and automation. The complexity now inherent in networks means that software analysis of the reality is an essential component of the information plan for understanding and defending the network.
Every enterprise network is already too extensive and continues growing in both size and complexity too rapidly for any human to grasp. Only accurate automation analyzing the reality and providing clear remediation priorities give an organization any hope of overcoming the resource gaps with automation even while helping cybersecurity staff to continue to expand their knowledge and experience.
About the Author: Steve Hultquist is an expert in cybersecurity, customer success, technology, communications, marketing, and leadership. He currently serves as Chief Evangelist at RedSeal, Inc., which helps Global 2000 companies defend their critical digital assets. Prior to serving as Chief Evangelist, Steve served as CIO and VP of Customer Success, with responsibility for all customer facing services and support. Previously, he has been a consultant, a journalist, and a hands-on network, systems, and software engineer deploying bleeding edge technologies for customers worldwide.
