Here Be Dragons

While both Berry and Goldberg agree there can be much hidden value in conducting ROI analyses of security technology, they say the landscape is replete with potential stumbling blocks.

Goldberg says organizations questioning the ROI of security tend to make two common mistakes. The first to focus on a purely quantitative analysis, which risks failure to recognize the technology's business impact. For example you can't tell whether your security risk profile has changed just by knowing the number of monthly alerts generated by your intrusion detection software. If you received 500 alerts one month and only 300 the next, does that mean that the intrusion detection is working well, or that it isn't working as it should be, since it may have not even noticed some intrusions in month two? And an ROI analysis certainly can't tell you if the one alert you should have responded to in month two, when your numbers were down, was an extremely dangerous one.

"So there's some problems with trying to pull this stuff out of the technology, and a lot of organizations go into this monthly reporting cycle trying to show that all of this stuff is doing something, but not drawing a link between that and actual impact on the business," he says.

But the other mistake is to not keep the metrics dynamic enough, Goldberg says. Once you have some software working and you're getting some value add out of the investment, you need to consider whether you should change the metric because the risk profile or the business has changed.

And Berry warns that numbers in an ROI analysis can be extremely rubbery. "In an ROI analysis you can come up with any number that you want to," he says. "You know, this idea we are going to avoid all these costs, and it's going to mean a $6 million cost avoidance if we invest in this. You extrapolate or rationalize any kind of figures that you want; that doesn't mean that they're accurate and that's one of the pitfalls of all of this.

"Many organizations get into the inherent difficulty in calculating cost avoidance which is very much more art than science."

The only way to tip the balance in favour of science is to gather as much independent empirically driven information and data as possible, Berry says. For instance, the value of a reduction in viruses infiltrating the organization entirely depends on the potential cost to the organization of the viruses in question. To get a head start on doing those calculations, you might see what data is available from the carrier companies that do risk insurance and the analysts houses. It's true that many of these breaches are so novel that the carriers won't have deep experience over a number of years to draw upon which would let other organizations make reasonable extrapolations about the calculations they should make within their own organizations. Berry concedes. Nonetheless, using independent third-party information as the basis of your calculations whenever possible provides you with a powerful analytic tool.

But Berry also cautions organizations against over-enthusiastic use of that tool. Plenty of organizations suffer "analysis paralysis", he says, getting so bogged down in such calculations that they never make the investment.

"Given the fact that a lot of companies don't even bother to do ROI analysis it is not that much of a problem in the organizations, but it can be a risk," he says.

But it is also vital to have the expertise to do the return on the investment analysis, he says. Without such experience, you can have the greatest third-party information in the world, but still make a lot of calculation mistakes.

Above all, remember that when it comes to security, value stems less from what's delivered to the buyer and more from the costs and negative impacts the buyer is able to avoid in the event of a catastrophe.

"A thorough economic analysis is essential to achieve a complete understanding of the entire range of costs, risks, rewards, and resource demands," Berry says.