Tuesday, June 23, 2009

Microsoft Security Essentials: Road Test

What better way to test the effectiveness of a malware scanner than to go download random binaries from the dirtiest part of the Internet... the P2P networks. Even worse, to really test Microsoft's Security Essentials I decided I would download, install and run LimeWire... and download binaries (.exe files) that I would normally avoid like the plague.

It's simple to find malware on the 'net these days... pop open LimeWire and search for something like "Photoshop crack" or "{random app here} keygen"... you'll find all the malware testing you could ever want.

As a control to Microsoft's Security Essentials I used VirusTotal.com. If you've never used VirusTotal it's a service that uses the major scanners out there (~40'ish or so) to scan your uploaded file and give you a verdict... pretty neat utility. Since not every Anti-Malware (A/M) program catches all threats it's best to run the binaries I've harvested through this handy-dandy little tool to ensure that I have a good idea of what the competitive products are finding on the binaries I'm working with.

I will admit the results are a little... shocking, even for Microsoft's standards.

Let the games begin!

-----
Testing Method: Download random [suspect] binaries from LimeWire
Keyword Search: "keygen" "crack"
File Types: Windows .exe files
Control: VirusTotal.com
-----
  1. Name: "Office Mac Keygen" | Verdict: Obvious | VirusTotal Link: Here ( 89.47%) | MS SecEssentials: Fail

  2. Name: "All Sony Products KeyGen 1.2" | Verdict: Obvious | VirusTotal Link: Here ( 92.69% ) | MS SecEssentials: Detected - TrojanDownloader:Win32/Tonick.gen (removed)

  3. Name: "ALL_Xilisoft_Products_Keygen_v_1" | Verdict: Obvious | VirusTotal Link: Here ( 90.25% ) | MS SecEssentials: Detected - TrojanDownloader: Win32/Tonick.gen (removed)

  4. Name: "berry white incl keygen by REVENGE" | Verdict: Obvious | VirusTotal Link: Here ( 87.81% ) | MS SecEssentials: Detected - 2 Threats (in 2 files) TrojanDownloader: Win32/Tracur.A & Tracur.B (removed)

  5. *Name: "conficker_including_keymaker_by_T" | Verdict: Average | VirusTotal Link: Here ( 66.67% ) | MS SecEssentials: Detected - TrojanDownloader: Win32/Tracur.A

  6. Name: "solo_le_pido_dios__including_crack" | Verdict: Obvious | VirusTotal Link: Here ( 92.31% ) | MS SecEssentials: Detected - TrojanDownloader: Win32/Tracur.A

  7. Name: "umidimmi_var_KeyGen.All_Versions.zip" | Verdict: Average | VirusTotal Link: Here ( 74.36% ) | MS SecEssentials: Fail

  8. Name: "SRS_Audio_SandBox_1.9.0.4_with_Keygen.zip" | Verdict: Obvious | VirusTotal Link: Here ( 90.25% ) | MS SecEssentials: Detected - TrojanDownloader: Win32/Tonick.gen

  9. Name: "y_hubo_alguin_crack-serial-keygen.zip" | Verdict: Average | VirusTotal Link: Here ( 70.74% ) | MS SecEssentials: Fail

  10. Name: "registry_clearner_from_TSRh_team (cracked).zip" | Verdict: Average | VirusTotal Link: Here ( 60.53% ) | MS SecEssentials: Fail
Looking at the results, one could conclude that Microsoft's SecurityEssentials did not fare well compared to other anti-malware scanning engines. That being said the Security Essentials detection engine broke down on 1 obvious piece of malware (90% detection rate) and then choked on another 3 pieces of malware that had 60%, 70%, and 74% detection rates respectively. Ordinarily that's pretty bad but when you consider that Microsoft Security Essentials is free... one has to wonder.

Overall some things that I noticed is that the engine's real time protection is a little lacking, as it rarely (only once) caught the piece of malware as it was being unzipped, and typically only when I attempted to actually run the file. This obviously isn't optimal, but not an entirely show-stopping failure given that most of the active pieces of malware require you to activate them somehow... such as double-clicking to execute the file.

---
Bottom Line: The verdict, unfortuntaely folks... is that Microsoft's Security Essentials is essentially lacking on the detection front. In a world where Internet-borne threats are polymorphic, stealth, and ever-changing the Security Essentials tool fails to deliver real protection against the nasty things that go bump on the 'net. Even when compared against other freeware detection engines (such as AVG) Microsoft's engine still competes poorly, since every single piece of malware that Security Essentials missed, AVG's scanner caught.

Sorry to say - but I recommend spending the cash for a decent anti-malware scanner boys and girls, "Code-name Morro" (Microsoft Security Essentials) isn't up to the task of protecting your computer.

I would like to stress that this is a test of static file analysis, and not of "invading malware" from a drive-by download or something... I downloaded files and then had MSE (Microsoft Security Essentials) check to see if it could detect malware hidden inside the ZIP files they came in. Your results may vary!

Interestingly enough - Steve Ragan over at The Tech Herald had exactly the opposite results. Odd... not sure what to make of this yet... but rest assured more analysis is happening as you read this. Check out Steve's absolutely comprehensive analysis (complete with video!) here... http://www.thetechherald.com/article.php/200926/3926/Review-Microsoft-Security-Essentials

6 comments:

Rob Fuller (mubix) said...

So these numbers are slightly skewed. By using Virus-Total.. hold on, just so I don't rant about this topic:

This fellow does a much better job at it.
http://anti-virus-rants.blogspot.com/search?q=virus+total

To summarize: VirusTotal uses stripped down versions of the "Anti-Malware" tools. So, pitting them (the 40 or so stripped down versions) against a fully installed anti-malware suite (I use the term loosely), makes the facts even worse.

I would suggest using some of the other sites out there to test further:
http://anubis.iseclab.org/?Anubis
http://virusscan.jotti.org/

However, they did just release the tool today, other vendors have been at this game for a long time, just as with Google's release of Chrome, it might take them a bit to get their feet under them, so I will reserve final judgement on them. Can't wait to see what they have in-store. They do kinda have an ace up their sleeve (kinda created the OS), lets see how they play their hand.

Raf said...

@Rob: I would agree with you, ordinarily given that Microsoft *just released* Security Essentials... but consider this - they've had experience with this sort of thing! Remember "OneCare"? How about "Windows Defender"?

I think that by now MS should have the anti-malware thing right, or at least we need to acknowledge we've given them ample chances to not fail as miserably... I mean - hey as you point out - they did create the OS so they should have an unfair advantage... right?

idodialog said...

Sorry but this isn't a test at all. Just because YOU think that a crack file or a keygen is malware does not make it so - at all. Keygens by their very nature, i.e. inserting code in an exe, or reverse engineering an exe (I use these terms for simplicity) incorrectly flag as malware or as a trojan or the like. In fact many do no more than they claim to do causing no harm, not acting as a trojan or worm etc etc.
As they are not "malware" neither should AV/AM programs find them. If they do it is ACTUALLY a false positive.
So your criticism of MSE may actually be a commendation.
Cracks and Keygens may of course be malware but your tests end up telling us nothing.

Raf said...

@idodialog:
What you'll notice is this isn't just my "interpretation" of these files. They were submitted to various (I published VirusTotal because it's the best known) binary analysis websites; which then detected various versions of Trojans and such inside the actual .exe file.

What's interesting in your defense of Keygens and Cracks is that most of them (as either a side effect, or main purpose) actually trojan the machine of the poor sucker who's stupid enough to download and run the untrusted file...

While I don't think my analysis is complete or comprehensive by any means; I don't think attacking the fact that I used suspect binaries of a questionable nature which I then verified as malware with 40+ malware detection engines is the right criticism.

Thanks for your continued readership, I appreciate you taking the time to comment!

idodialog said...

@Raf - OK I had another, more careful look at what you said. I guess it was your second para which caused my reaction and partial misreading. I've not ever used limewire - you may be right!
Next: Mine is by no means a defence of Trojans. On the contrary, my hope is that AV Engines will accurately detect an actual trojan rather than one that simply has some of the characteristics of a trojan.
Finally I looked at the interesting VirusTotal results. I'm unconvinced. The variation in the range of results and detection of apparently different infections by different AV engines in the same file is alarming. The failure of other leading engines to make detections would need to be plotted to see where MSE falls.

Anonymous said...

I would also challenge the findings of this report. One, as a user suggested, detection of keygens as malware is a highly debated subject. We have seen many keygens detected as malware when they do no apparent malicious activity. The only way to truley test this would be to create a keygen and run it through something like VT, I et you would get detections. 2. Criticizing MSE detections on files that had low detection rates across the board is unreasonable. Anti-malware has false negatives, its part of the game. AV Comparatives has given MSE Advanced ratings for some time now, I certainly trust their range of tests. Just my opinion.

Google+