Now that the dust has somewhat settled from the Target breach, and the subsequent law-suit madness is hopefully over I feel like it's safe to write about this topic, as much as it can ever be to discuss a touchy subject. Much of the writing and rhetoric, and finger-pointing for blame, around that breach centered around the fact that a 3rd party was hired to 'find faults' in the 1st party, and the 3rd party apparently failed to do so. Or ... something like that.
I wish I could say that this is the first time I've heard a confused understanding of what penetration testing is, but it's not. I also wish I could say that the purpose, limitations, and actual best-use of penetration testing is well understood amongst enterprises - but again - it's not.
Penetration testing as TVM
An organization I'm familiar with basically used penetration testing by a 3rd party as their stand-in TVM (threat & vulnerability management) program. The case was that the internal security team's ability to identify weaknesses and toolset were weak, and the CISO believed that the best way to identify threats that his team should focus on, in order to best position his defenses, was to be regularly penetration tested. So, four times a year his organization would undergo a structured, scoped and time-boxed penetration test which - of course - Information Security was ready and prepared for.
I'm sure you can already pick out the few glaring issues with this approach, but it continues to disturb me that the defensive posture of an enterprise is allowed to be determined by the testing capability and talent of another organization. Not to take anything away from the company that is currently charged with the penetration testing contract - because I have no reason to doubt their talents - but it's foolish to think that they'll find "all the issues", or even the most important ones. While I think penetration testing is important to identify the things that are glaring, and obvious from a complete outsider's perspective - it should be in no way (in this blogger's humble opinion) authoritative on what you should consider important. Third-party penetration testing does not replace a threat and vulnerability management program, period, end of story. It just can't.
There are too many variables here. The thing that's most important to understand is that penetration testing is ultimately too limited, in the way it's implemented by CISOs, and has a very little chance of being holistic enough. Penetration testing will definitely identify some externally visible, exploitable vulnerabilities if you hire a good crew. Otherwise you'll get what you pay for, the output from a Nessus scan copied and pasted into a PDF. The problem here is that you need a more complete picture. There are nuances. Different testers look for different things, they have different approaches, and will likely have different results. You need a consistent, repeatable, and continuous approach to identifying your vulnerabilities supplemented by penetration testing. You simply can't swap out a TVM program for even regular penetration testing. It won't work.
Penetration testing leads to security
An organization, any organization, cannot simply test itself secure. That's as insane as an auto manufacturer crashing cars until they stop failing crash tests. You still have to actually fix the issues! And we all know how that goes. How many of you have stories where you go out and test one of your clients, only to discover that nothing, or barely anything, has been 'fixed' from the last round of testing?
While penetration testing is definitely a good way to identify exploitable, visible security issues in your enterprise when done right, it's not going to make you more secure unless you do something about the problems. Therein lies the challenge... too many CISOs are looking for someone to come in and find nothing wrong and move on. We call this the compliance with penetration testing requirements.
Good security leads to good security. Whether you're hiring outside firms to perform penetration testing or not. There is no substitute for sound strategy, executed well and with purpose and executive leadership's backing.
What's the point then?
You may think I'm down on penetration testing, at this point. You're wrong. I think there is a time and place for one of the most important validation activities a security program can perform. I stress that this is a validation activity - once you've shored up your issues you seek to validate your posture with a good and thorough testing.
For those enterprise CISOs who are building or optimizing their security program penetration testing is a validation exercise. First and foremost, you need to know what your high-value assets are. There is no substitute for this, and penetration testing nor crystal ball will not help you here. Identification of critical assets is a primary activity of any security program, and everything you do will be based from that point. Next make sure you've built a solid TVM infrastructure, with good policies and practices. Ensure you have a workable definition of critical, and how you make go/no-go decisions when it comes to remediation, deferring a fix, or simply accepting a risk. Then make sure you have the necessary backing to ensure that you can execute when it's time. Once you've done all that, and you're sure you've done enough internal test-fix rounds have someone perform a thorough penetration test on your organization to show you all the things you've missed or simply not thought about. It's amazing how many times someone can get at a high-value target through what we perceive is a low-value asset...
Lastly, don't get too mad at your 3rd party penetration testing organization for failing to identify the avenue of infiltration that caused your big breach. There are a lot of factors that go into what is considered a 'good' penetration test - and many of the failings fall on the shoulders of the client...but that's a discussion for another time.