Of Metrics, Magic, and Failure

As everyone knows companies have been tightening their belts lately, and IT spending is harder to come by no matter what Gartner and the other analysts are saying. You and I know full well that getting money out of management, even if it's a great cause, is like pulling the cell phone out of a teenager's hands... possible but unlikely.

Now, over the last couple of weeks, and more recently today via Jeremiah Grossman on Twitter) folks have been talking to me about metrics, and more importantly measuring "web based exposure to risk" ... right. We can talk about this all we want, but talking it to death won't make it any more real.

At the heart of the problem is the fact that we're essentially being asked the same question your parents asked you when you asked, begged, and pleaded for $toy: "what will happen if you don't get $toy ... right now? Can you wait until Christmas?"

There are two camps that have formed over the last several years, when it comes to answering that question ...

1. The "FUDs" - You know who you are. "We'll be hacked if we don't buy product X" ... sound familiar? Seeing Fear|Uncertainty|Doubt only works for so long before they catch on!
   
2. The "Franks" - With the mentality of an accountant and the numbers to back it up, the "Franks" typically don't give any emotion into their work, it's all risk and numbers

The problem here, of course, is that neither is very effective. Let me re-phrase- neither group has been very effective so far. Selling FUD may work for a while but eventually Chicken Little will realize the sky isn't necessarily always falling ... and will start asking questions. On the other hand, you can't really do InfoSec as a numbers game. This isn't a casino where you can simply play the odds and hope to win ... there aren't any established odds and no one else (except maybe for you) is playing by any rules.

So we've got a problem. There are 3 things that are going to save Info Security ... web app security more specifically - metrics, magic, and failure. Don't laugh.

• Metrics: You can measure damn near anything these days, and if you look hard enough and network with enough people eventually you'll get some pretty decent numbers on things like how many times the average credit card authorization site for a mid-market company gets attacked (actually attacked, not "scanned") in a given quarter. You can even scam your way into getting people to giving you metrics on how many times companies in your industry, maybe even your competitors, (anonymously of course) have been attacked and maybe even hacked. You may even get someone like Gartner, Forester or IDC to publish a report on the metrics of being hacked and how much it costs per record lost, or hour of downtime. This is all great information but it's entirely worthless if you don't have the right context. By context I'm referring to the specific context that makes sense to the metrics you're collecting ... clear as mud right? Here's an example -if you're a very large online retailer it only makes sense to collect metrics of like context ... from other businesses that are of similar size, market, and exposure. In this instance collecting metrics from an industrial business won't help your cause any ... or will it?
• Magic: When metrics alone just don't make the case, make the case with a little bit of home-brew magic. Security folks are known for our passion and ability to manufacture a reality that fits our ends. I'm not saying to make things up, that's what the FUDs often do, I'm simply telling you to get creative. Make the numbers pop without making them up ... it's harder than it sounds. Being a magician also involves getting numbers from places that you can't talk about ... friends in your network, your own systems or other sources that can't clearly be cited for anonymity. Adding a pinch of magic to your pitch will make the metrics get someone's attention and scream "You can't ignore me!"
• Failure: It's a fact, nothing helps along a security agenda like a catastrophic hack. Once you've been breached your executives will write checks for amounts you've never seen before... in record time. The problem is once you've had a major breach your job may be on the line and then things get really crazy (see my post on the dangers of a disaster-driven security program) while everyone is running around with their hair on fire. The trick is to not be the victim of a major breach but something big enough to spark attention and make people inside your company paranoid, that's when you get to work your magic. (see the Magic above). Some great person once said "Without failure, we cannot know what it is to succeed" ... and that's very true even today in InfoSec.

There you have it, Frank meet FUD. I will argue that there are no clear [security/risk] metrics that will win the mind of a sufficiently uneducated executive... so we must educate them and always make sure they understand the risks...

Speaking of risks ... how do you measure that, exactly? Do you measure what you find ($found_vulns) against the function of the application ($function) and the value ($value) combined with exposure ($exposure) ... ? And riddle me this Batman... what about those vulns which aren't found? How do you measure risk with that many unknowns?

Sadly, if I had the answers to those questions I think I would be writing this post off a yacht in the South Pacific, but alas, I'm sitting at my desk in Chicago... (damn it's cold for September).

For starters, and maybe as a spring-board for a good risk formula I have used the following elements successfully ...

• Dollar-value of the asset being assessed/analyzed -should be the easiest metric to gather
  
• Relative exposure to defined threats (notice I say defined threats) - exposure is relative to the industry, type of business, type of technology, type of asset, etc ... this can have multiple components within it (to be discussed at a different time)
  
• Relative complexity of asset - remember, complexity is the enemy of security
  

These are the simplest-case metrics you can gather that will work for building a business-case for risk-based analysis.

Good luck! More coming soon...