Friday, February 1, 2008

The Psychology of Patching...

I read an interesting article on ZDNet, and thought I would add my own spin on it. Giving credit to the original source ( I think Larry Dignan found an interesting topic. Why? Because there really is a psychology to patching, and it's not unique to Oracle, or Microsoft, or any particular software vendor. Here's why.

Eric Maurice of Oracle brings up an interesting point about how patching is really a negative experience... Russian roulette if you will - with your production environments. How many of us have had our production environment stop working, seemingly randomly, due to an unintended consequence of a patch? Think about this for a second, if at least one instance pops right into your head... you know where Eric Maurice is coming from. What I don't think I agree with are Eric's two possible solutions. First solution is to make patching mandatory, period. Second solution is to make it a measurable metric which would hopefully bring about better patching entitlement...interesting concept but I don't agree that these are the only methods of looking at patching. In fact, I don't actually see patching as a 100% must-do. I'll pause a second while you clean the soft drink which you just sprayed on your monitor...

You're wondering how, as a security veteran, I can say that patching isn't always critical. Well, it's precisely because I've been doing this for so long that I can say that. It's the exact same reason a critical vulnerability in a web application isn't always critical... you understand right? Let me elaborate here... and maybe clear things up a little and hopefully disspell your thoughts of me as a lunatic.

First, I think of patching as a last-line of defense. If your systems is so exposed so as to have to rely solely on patching, you may have a problem to start with. This is a bit controversial, as a significant number of my colleagues would argue that a patched system should need no other defenses - but allow me to make my point. I'll make no argument that front-line systems must be patched, but then... those are likely more heavily defeneded by things like firewalls, IPSes, and so on ad nauseum. Patching is important here, definitely, but since these systems are likely clustered it's likely that you can test one or a small cluster without taking down the entire system at once. When you talk about things like databases, and other such critical pieces of infrastructure - you should have multiple layers of defenses so that patches should only be a piece of the overall defense strategy. Even an internet facing design, if it's done properly, has n+3 tiers (at least 3...) and mitigants in-between the layers such as IPSes, firewalls, database front-ends possibly (look into Imperva's database cloaking technology), on top of properly secured access rights, encryption and the whole of things that are security. If all the other pieces are working right, then patching should be but a small component of the puzzle.

I'm not saying patching isn't critical and important, don't get me wrong - but it shouldn't be as "all-important" as some people from our security realm would dictate. If you don't patch to the latest Oracle patch-bundle, the sky won't fall if your application design is security-conscious.

I will admit, I have alterior motives here as well, ones that are not security-related. Keep in mind that in order for IT to be supported, as it is likely not a profit center for the company, the business must make money. If a business is to make money, downtime must be minimized, and performance must be streamlined. If you throw "must-do-it-now" patching into that mix, you will have yourself a recipe for disaster, I can virtually guarantee that. I have spent many hours on calls trying to figure out and then later explain why the latest patch set for product X my security team required is now impacting production systems - and the conversation never ended well. The CIO will always play the "it has to stay up and be functional" card over "it has to be 100% bulletproof", always. It's almost irrelevant how fool-proof your strategy is to test patches before they go into production systems, it's almost impossible (almost, because with a vast budget anything is possible) to duplicate every aspect of your production systems on a test system. Sometimes things happen you can't even predict, and BAM! You're on an outage call, but I digress...

So to my original point - patching should not be seen as a thou shalt do directive in every case. Treat your systems with the proper front-line precautions and it'll be a significantly lesser burden on you when Microsoft or Oracle, or IBM releases that next critical bug.

Feedback, as always, is welcome!

No comments: