If we aspire only to benchmark within sectors, we will end up with hundreds of different sets of metrics that cannot be easily translated across industry lines. While that would be preferable to what we have today — which is very little benchmarking at all — it is not the ideal. Ideally, there should be a single, common set of metrics that could be used across all markets or sectors, that would apply to security at any company of any size, regardless of what business it is in. Then, those common metrics could be modified within individual sectors to accommodate the peculiarities of the given market.
In other words, perhaps we are doing this backwards. The security industry has begun by developing metrics at the company level, then attempting to find commonalities within a sector — and we are not even seriously considering cross-sector benchmarking because by the time we get to that level, our metrics have become so specific and complex that it is impossible to find shared elements to compare.
Instead, shouldn’t we be trying to develop a baseline of security metrics that apply in every sector, and then moving down the chain from there? This would enable us to engage in meaningful benchmarking at a national and global level. It would also provide an easier entry into the metrics and benchmarking scene for sectors that currently do not engage in these activities to any great extent, because they could work from the baseline to develop sector-specific metrics instead of having to begin from scratch.
In order for this type of high-level benchmarking to take place in security, some entity or a partnership of entities must first take the initiative to identify a set of measures that can be commonly used across sectors. So far, no such entity has emerged in either the public or the private sector, for myriad reasons.
Will the Public Sector Do It?
We know the government is willing to enter the metrics conversation at some level — it has done so through information security legislation like FISMA and related NIST special publications, as well as other information security and security-related regulations that require some sort of measurable assessment of risk or compliance.
In a more recent move that touches a little closer to home, the Department of Homeland Security is advancing a security metrics initiative by asking all 18 of its defined critical infrastructure sectors to develop metrics that demonstrate security within each sector. DHS partnered with the Department of Energy and the American Gas Association last April to conduct a workshop aimed at starting a list of useful security metrics for the oil and natural gas sector.
The hosts of the event, organized by the Joint Energy Metrics Working Group under the DHS Critical Infrastructure Partnership Advisory Council (CIPAC), stated emphatically that the metrics initiative was to rely on the voluntary participation of companies and would not evolve into a mandatory reporting program. DHS is very careful to position this effort as a partnership with industry, not a regulatory exercise.
Clearly the agencies involved want to ease industry fears that any government intervention will lead to mandatory compliance. The concern is that metrics imposed from outside will not be relevant or appropriate, at least not initially, and the road to relevant and appropriate metrics might be filled with fines and painful, expensive mistakes. But if the agencies walk on eggshells, the project may result in a set of metrics that is too vague or too general to provide any meaningful information for the industry or the government.
While the government is clearly showing more interest in security metrics, it does not appear that the public sector will have the momentum or the support to develop common metrics suitable for sector-wide or cross-sector benchmarking in the near future.
Will the Private Sector Do It?
The private sector has the potential to develop common metrics for cross-sector security benchmarking, but so far, no leader has emerged to propose it.
Security associations, organizations and publications have all stepped into security measures benchmarking on a micro scale. Over the past couple years, security practitioners have found their e-mail inboxes increasingly jammed with invitations to participate in this or that group’s short survey to gather data on a single topic that can then be analyzed and charted to allow participants to see where they stand among their peers. The results of these efforts can be useful for informal purposes, but they are generally not controlled enough to offer results that are appropriate for cross-sector benchmarking. Also, these initiatives have generally been proprietary to a single organization or publication, leading to limited respondent pools and limited usefulness.