Beyond Natural Disasters

Oct. 27, 2008
Business continuity issues you haven't thought of before

There's trouble brewing on the horizon. Your business could be headed for disaster and you don't even know it. The weather looks good, the terror alert level hasn't changed, but some things in IT are creating what could be one of your worst business outages ever.

When the business continuity and disaster recovery conversation comes up, we often think about things such as site disasters, environmental catastrophes, and hardware failures. These are certainly real threats and need to be considered, but there are other, less obvious, technologies and IT processes that can create just as much of a business nightmare.

Operating System Configurations and Maintenance
Many of the server operating systems I test for security weaknesses such as Windows, Linux, and NetWare, etc. are only one disgruntled employee away from being taken down. The problem is that many servers are not properly hardened from attack nor are they being patched like they should be. In fact, within a matter of minutes, someone with malicious intent on your network can discover a vulnerability and then run an exploit tool against it to gain full administrative level access to the system - often without anyone ever knowing about it. Figure 1 shows how the free Metasploit tool can exploit a missing patch on a Windows server to provide anyone on the network with remote access to the system.

Not only do misconfigurations and missing patches create serious business headaches, they also put sensitive information at risk. This is often information that may not be properly backed up or replicated elsewhere thus extending business downtime.

Web Applications
Another IT-related element that many people don't think about creating business continuity risks is Web applications. Be it e-commerce, intranet sites, data center monitoring systems - you name it - Web applications are everywhere are usually susceptible to attack. The four main areas posing risks within Web applications are:

Cross-Site Scripting where an attacker can modify a Web page or trick unsuspecting users to click on a malicious link thus redirecting traffic from your legitimate site to a rogue third-party site. Not only has sensitive information been compromised using this attack but many well-known Internet sites have been taken offline for extended periods of time creating big problems for the businesses affected.

SQL injection where an attacker can send database query commands to the Web application that are then redirected to the backend database. The result can be relatively benign in the business continuity sense where the attacker "only" reads what's in the database. However, with SQL injection, an attacker can also delete the contents of the database as well - certainly not good for keeping the business going.

Login attacks where an attacker attempts to crack the passwords of legitimate users resulting in locked accounts or cracked passwords and full access into the system.

Denial of Service attacks where an attacker sends large amounts of traffic (page loads, login requests, SQL injection attacks, etc.) to the Web site that creates a situation where legitimate requests cannot be handled.
These vulnerabilities are typically the result of people deploying Web sites and applications in a hurry and not using good tools to test them for security flaws on a consistent basis.

Wireless Networks
Like Web applications, wireless networks often pop up in various places across the enterprise as well. The convenience they offer in the physical security and data center arenas can easily be negated with a traffic redirection or denial of service attack. With traffic redirection, an attacker sets up a rogue wireless access point and tricks your wireless hosts into connecting to it. The result can be mean loss of video surveillance, loss of data center system controls, and more. In a wireless denial of service, the attacker can use common wireless hardware and software to effectively jam the airwaves knocking your wireless offline indefinitely. It can be next to impossible to track down the culprit much less prevent this type of attack.

These vulnerabilities are typically the result of people being sloppy with wireless deployments including not hardening wireless systems from attack. It's also a matter of placing critical business systems on a wireless network - something that can be risky even when encryption and/or a wireless intrusion prevention system are used.

Homegrown Applications
Many IT shops have what I call homegrown applications that often serve as critical business defenses. The need is there but the budget not so much. And when this happens, what's an IT professional to do? Build the application in-house of course! I've seen everything from intrusion prevention systems to firewalls to network management systems written (and supported) in-house.

The problem is not with the software necessarily. In fact, many homegrown applications work really well and they're "customized" to fit right into the environment they're in. The issue is with how the applications are managed. In most cases I've seen, there's usually only one person that knows how to manage and maintain the system. They know it better than anyone else, but this if often to their own detriment. It means they can't take off work and they always have to be in touch in the event of a problem.

Say, for instance, a homegrown intrusion prevention system stops working and is configured in such a way that all inbound network traffic is denied. It's the middle of the night with no one onsite and no way to access the internal network remotely for troubleshooting. On top of that, the one person who knows the system happened to be in a car crash the day before and is now incapacitated. So, there's no one to fix the problem and (of course) there's no documentation or response procedures to help others get the system back online. All electronic business communications cease. Not good in today's world.

Gartner Group has found that operator error is the second most common cause of unplanned downtime. This is ahead of the traditional factors what we commonly associate with business continuity. Oversights such as not knowing how to start up and shut down certain types of hardware and improperly racking systems creating cooling and hardware reliability problems are not only careless but they can usually be traced back to ignorance and lack of training. I often ask network administrators what type of training budget they're allotted each year. The answer is almost always a resounding "There is no budget.". I also hear "I don't have the time even if we had the money.". The logic is backwards. IT and security staff are responsible for administering critical business systems yet they're not given the budget and time to learn what they need to do their jobs and stay current. Arguably one of the greatest business continuity risks of all.

Change Management
Many software failures and system crashes that lead to business outages can be traced to communication breakdowns. In particular, it's people not knowing who's doing what or who did what. In other words change management. Effective change management is about keeping everyone in the loop at all times. All people involved with managing a certain host or application must aware of who's doing what, when, and why. However, this is often not the case. It's the lack of this critical communication that often leads to downed systems at all the wrong times.

I experienced this first hand recently. I was brought in by a client to assess a situation where some IT team members made changes to a critical network host at the wrong time and caused it to go down for quite an extended period of time. Because of this, all electronic transactions and customer communications into a critical e-commerce environment were shut off for hours. Executives were notified, customers were unhappy, and tens of thousands of dollars were lost all because a handful of people didn't follow the proper change management procedures. So much for the thousands of man-hours they spent adopting the change management-centric ITIL framework the previous year!

Everything from improper system testing to overlooking certain incompatibilities in operating system configurations can create big time business trouble. These problems all fall back on change not being managed properly within IT. I'm not a big fan of red tape in IT and security, but if that's what it takes to make sure changes are properly implemented and keep critical business systems from going down, then so be it.

Lack of Buy-In
A problem that contributes to most of these business continuity issues is the fact that a lot of managers don't really understand what it takes to implement and maintain effective information systems to support the business. Be it lack of personnel or not enough money to buy the right technologies to help support business continuity (i.e. test systems, storage replication, and server virtualization) the priorities are all out of whack. They may or may not contribute to the business bottom line, but it's virtually guaranteed that if IT and information security are not given the resources they need, serious business consequences can result.
Business continuity and responsible information systems management go hand in hand. A traditional business continuity plan will only go so far - especially if your focus has been solely on what Mother Nature can dole out. Step back and take a look at your technologies and IT processes to see what else is creating weak links for your business. By stepping out of the traditional mindset we've had for so long, you'll see your business operations in a new light - a light that's impossible to keep burning when your priorities aren't where they need to be.

Kevin Beaver is an independent information security consultant, speaker, and expert witness with Atlanta-based Principle Logic, LLC. He has more than 19 years of experience in IT and specializes in performing information security assessments revolving around IT compliance. Kevin has authored/co-authored seven books on information security including Hacking For Dummies and Hacking Wireless Networks For Dummies (Wiley) as well as The Practical Guide to HIPAA Privacy and Security Compliance (Auerbach). He's also the creator and producer of Security On Wheels - security learning for IT professionals on the go. Kevin can be reached at kbeaver @