Metrics that Matter for Everyone

March 26, 2009
The second of a two-part series on how to make security metrics work for your organization

In last month’s issue of Security Technology Executive, we discussed the importance of internal security metrics programs and laid out a six-step plan for creating one (“Building a Metrics Program That Matters,” Feb 2008, p.22). If you have not started working toward a metrics program yet, now is the time.

Budgets are tightening — a metrics program will help you identify and present tangible savings and return on security investment. Layoffs are skyrocketing — a metrics program will enable you to better secure your organization in the face of an increased threat of workplace violence and insider theft. Management is looking for internal partners who are motivated to help the business succeed in a tough market — a metrics program will elevate your stature in the company by showing how you align your programs with the goals of the business. (Check out last month’s article for more specifics on what metrics can accomplish and how.)

While it is clear that stand-alone metrics lead to significant internal benefits, their value increases exponentially when they are paired with the metrics of others. Comparing your metrics results with the results of other businesses in your sector and across sectors enables you to rate your security program’s performance, which could help you identify security gaps as well as gain funding and executive-level support. Unfortunately, the security industry has often shown itself unprepared and unwilling to develop a framework to accommodate benchmarking on a significant scale.

How Is Benchmarking Useful?

Comparing your security metrics results with others’ can give you insights beyond those you receive from studying your own metrics without context. Say, for instance, that you have established a metric to show security’s cost as a percentage of the company’s revenue. You have collected all the data and done your calculations; however, as you prepare to present the results to management, you realize that you have no way of proving to your CFO that the numbers you have arrived at are evidence of cost efficiency.

The percentage may look great to you, because you may have talked with peers about their overhead and heard some statistics at seminars that convince you that you are way ahead of the curve. But your CFO is a tough sell these days, and he or she is going to think your cost percentage looks awfully big sitting out there all by itself. Without solid data showing that you eat up a smaller percentage of revenue than your competitors’ security programs, your numbers may be more of a hindrance to you than a help.

Without metrics results from other companies, you have no way of identifying what is missing from among your own security measures and programs. Your internal metrics may show that you are meeting your own and your management’s standards in all your existing initiatives, but they cannot tip you off that your standards are lower than the standards of 80 percent of the other companies in your sector.

Reliable benchmarking data also enables you to label your program “good,” “better,” or “best” — or “bad,” “worse,” or “worst” — and you cannot honestly do that with only your internal metrics. If you could prove your program was good, better or best, you may have an easier time securing support and funding for security initiatives, and you would be able to easily show due diligence in case of a breach or lawsuit. If you could prove your program was bad, worse or worst, you would be able to see exactly where you needed improvement, and you would again have an easier time showing senior management the need for funding and support.

Sector-Specific vs. Cross-Sector Benchmarking

Benchmarking with “like” organizations within your sector is an extremely valuable endeavor. Since each sector has unique risks, requirements and controls, benchmarking within your sector enables you to make specific comparisons without having to account for the differences between industries. Yet, with security becoming increasingly sophisticated and branching out in new directions yearly, should sector-specific benchmarking be the highest level of commonality we seek to achieve?

If we aspire only to benchmark within sectors, we will end up with hundreds of different sets of metrics that cannot be easily translated across industry lines. While that would be preferable to what we have today — which is very little benchmarking at all — it is not the ideal. Ideally, there should be a single, common set of metrics that could be used across all markets or sectors, that would apply to security at any company of any size, regardless of what business it is in. Then, those common metrics could be modified within individual sectors to accommodate the peculiarities of the given market.

In other words, perhaps we are doing this backwards. The security industry has begun by developing metrics at the company level, then attempting to find commonalities within a sector — and we are not even seriously considering cross-sector benchmarking because by the time we get to that level, our metrics have become so specific and complex that it is impossible to find shared elements to compare.

Instead, shouldn’t we be trying to develop a baseline of security metrics that apply in every sector, and then moving down the chain from there? This would enable us to engage in meaningful benchmarking at a national and global level. It would also provide an easier entry into the metrics and benchmarking scene for sectors that currently do not engage in these activities to any great extent, because they could work from the baseline to develop sector-specific metrics instead of having to begin from scratch.

In order for this type of high-level benchmarking to take place in security, some entity or a partnership of entities must first take the initiative to identify a set of measures that can be commonly used across sectors. So far, no such entity has emerged in either the public or the private sector, for myriad reasons.

Will the Public Sector Do It?

We know the government is willing to enter the metrics conversation at some level — it has done so through information security legislation like FISMA and related NIST special publications, as well as other information security and security-related regulations that require some sort of measurable assessment of risk or compliance.

In a more recent move that touches a little closer to home, the Department of Homeland Security is advancing a security metrics initiative by asking all 18 of its defined critical infrastructure sectors to develop metrics that demonstrate security within each sector. DHS partnered with the Department of Energy and the American Gas Association last April to conduct a workshop aimed at starting a list of useful security metrics for the oil and natural gas sector.

The hosts of the event, organized by the Joint Energy Metrics Working Group under the DHS Critical Infrastructure Partnership Advisory Council (CIPAC), stated emphatically that the metrics initiative was to rely on the voluntary participation of companies and would not evolve into a mandatory reporting program. DHS is very careful to position this effort as a partnership with industry, not a regulatory exercise.

Clearly the agencies involved want to ease industry fears that any government intervention will lead to mandatory compliance. The concern is that metrics imposed from outside will not be relevant or appropriate, at least not initially, and the road to relevant and appropriate metrics might be filled with fines and painful, expensive mistakes. But if the agencies walk on eggshells, the project may result in a set of metrics that is too vague or too general to provide any meaningful information for the industry or the government.

While the government is clearly showing more interest in security metrics, it does not appear that the public sector will have the momentum or the support to develop common metrics suitable for sector-wide or cross-sector benchmarking in the near future.

Will the Private Sector Do It?

The private sector has the potential to develop common metrics for cross-sector security benchmarking, but so far, no leader has emerged to propose it.

Security associations, organizations and publications have all stepped into security measures benchmarking on a micro scale. Over the past couple years, security practitioners have found their e-mail inboxes increasingly jammed with invitations to participate in this or that group’s short survey to gather data on a single topic that can then be analyzed and charted to allow participants to see where they stand among their peers. The results of these efforts can be useful for informal purposes, but they are generally not controlled enough to offer results that are appropriate for cross-sector benchmarking. Also, these initiatives have generally been proprietary to a single organization or publication, leading to limited respondent pools and limited usefulness.

No private industry group has yet announced an effort to take the first step of developing security metrics that can be used across all sectors in the security industry. It is likely that such an effort could only succeed if industry groups worked together and encouraged all their members and constituencies to participate in a joint effort.

A Starting Point

In preparing this article, George Campbell, emeritus faculty member of the Security Executive Council, drafted a set of metrics he believes could be used to measure security in any business. He expected the task to be a difficult one, but, he says, “I was struck by how many common denominators there were. I was amazed to find a prevalence — not an absence — of commonalities.”

The metrics Campbell chose focus on two areas that security has to address in any type of organization: how security interacts with the business (meeting goals, contributing value) and how security deals with risk.

In the coming months, the Security Executive Council is going to be focusing on all the relevant areas of security where metrics can provide significant insight or can sway management or demonstrate value to the organization. Do you have ideas on metrics that would be useful to security in all companies? If so, we would like to consider these for inclusion in our list. Send your thoughts to [email protected].

George Campbell is emeritus faculty of the Security Executive Council, former CSO of Fidelity Investments, and the preeminent expert in the field of security-related metrics. See page 20 for the rest of his bio.

Marleah Blades is senior editor for the Security Executive Council (SEC). Prior to joining the SEC she served for six years as managing editor of Security Technology & Design magazine.

The Security Executive Council is a member organization for senior security and risk executives from corporations and government agencies responsible for corporate and/or IT security programs. In partnership with its research arm, the Security Leadership Research Institute, the Council is dedicated to developing tools that help lower the cost of members’ programs, making program development more efficient and establishing security as a recognized value center. For more information and inquiries on membership requirements, visit