An informal snapshot of virus-detection rates on some fresh real-world malware

[mechBgon]

I did a little real-world malware-detection test that I mentioned in another thread recently, and I thought it might interest some readers here. The main point is not to say which antivirus is best, it's to point out that none of them are an excuse for you to get cocky and ignore common sense, risk avoidance, and other best practices.

Main points:

• Antivirus software doesn't detect all malware, so it should be just one layer of your defense strategy. This concept will get "well duh, Captain Obvious :roll:" reactions from many of you, but might be informative to others, so bear with me other layers of defense available to you.
  
• Antivirus programs may have options you can enable to increase detection rates, such as heuristics; archive/compressed-file scanning; scanning of all filetypes; and enabling optional threat categories like dialers, joke programs, adware/spyware, etc.

I collected 95 malware samples, including exploits, rootkits, trojans (Zlob, DNSChanger, LoadAdv, VideoAccessCodec and others), backdoors, password-stealers, PUPs/adwares (including some DLLs, BHOs and EXEs harvested from a few live installs on my honeypot), a malicious HOSTS file, a QuickTime exploit, and a couple email worms, plus a really tough one: sneaky Frogexer images used to smuggle malicious code through the firewall. This is fresh real-world malware, hot off the bad guys' servers today.

I ran antivirus scans on folders containing the malicious files. Online scanners are noted as such; other scanners were run as fully-installed products, except the McAfee command-line scanner.

Caveats:

• dude, this isn't, like, a scientific test. Focus on the big picture (antivirus doesn't make you invincible).
  
• it isn't necessarily indicative of the softwares' performance on every category of threats. Most of the samples in my test were harvested from sources identified by using Microsoft Network Monitor 3.1 to track a highly-vulnerable Win2000 virtual machine being slammed around like a pinball after deliberately sending it to known malicious sites that use batteries of exploits, so this test doesn't have a lot of relevance to, say, detection rates on email-borne malware.
  
• it doesn't account for some of the proactive-defense capabilities that some of the softwares would provide in real-life scenarios.

In no particular order...

• F-Secure's online scanner detected 60 of the 95 samples.
  
• BitDefender 10 AntiVirus Plus, in its default configuration, detected 49 of the 95 samples.* The BitDefender contextual scanner appears to use all BitDefender's optional capabilities by default, so there were no additional tweaks to use. It missed all but one HTML exploit, some Trojans, at least one of the rootkit files, and the adware/fraudware files.
  
• ClamwinPortable detected 44 of the 95 samples.**
  
• NOD32, in its default configuration, detected 59 of the 95 samples**. Heuristics and compressed-file scanning were already enabled by default, and enabling detection of "potentially-unsafe programs" didn't result in any additional detections. NOD32 missed all the HTML exploits, the Frogexer pics, a rootkit, some Trojans and adware/fraudware files.
  
• Panda ActiveScan online scanner detected 47 of the 95 samples* if I'm reading the report correctly. They missed the HTML exploits but had relatively good detection of the adware/fraudware files.
  
• Microsoft's Live OneCare online scanner detected 43 of the 95 samples. It was unable to delete some of them, for unknown reasons, but it detected more than I expected.
  
• Kaspersky AntiVirus 7 detected 67 of the 95 samples using maxed-out settings (which is how I normally run it). Looking at the scan results, the reason KAV7 scored higher than F-Secure (which uses the KAV engine and sigs) appears to be the new heuristic-detection capabilities added to v7. KAV7 nevertheless missed all of the HTML exploits, all of the Frogexer pics, about half of the DLLs from actual adware/PUP installs, and some of the Trojans.
  
• Computer Associates (CA), in its default configuration, detected 26 of the 95 samples. There did not appear to be any additional scanning capabilities to enable, since heuristics were already enabled. CA missed nearly all of the HTML exploits, lots of Trojans, all the crafty .GIFs, the PUP/adware files, an email worm, a Spambot, Trojan-Downloaders, and some of the rootkit files. Evidently signatures and heuristics are not CA's main selling point
  
• Symantec's online scanner detected 39 of the 95 samples.
  
• Symantec AV Corporate 10.1.5.5000 detected 43 of the 95 samples in John's testing. Glancing at a screenshot of what was missed, it looks like it missed all the HTML exploits, some Trojans, the Frogexer pics, the adware/fraudware files, some Trojans, and the malicious HOSTS file.
  
• McAfee's command-line scanner was run with all detection capabilities maxed, using the hourly beta 4100 DATs. It detected 36 of the 95 samples. Notably, it failed to detect any of the rootkit files or any of the HTML exploit samples, and it failed to detect the current versions of the Frogexer pics.
  
• AntiVir PersonalEdition Classic, in its default configuration, detected 60 of the 95 samples. With all settings maxed out in Expert Mode (heuristics at maximum, all filetypes scanned, and all optional threat categories enabled) it nailed 71 of the 95 samples, including all but one HTML exploit, but still missed most of the files from live adware/PUP installations, some Trojans, and the crafty .GIF files.
  
• AVG Free Edition, in its default configuration (which already includes heuristics and archive scanning), detected 47 of the 96 samples. With the Scan all files option enabled, it detected on one additional file, the malicious HOSTS file, leaving the HTML exploits, adware/PUP files, the crafty .GIF files and lots of Trojans undetected.
  
• Avast! Personal Edition, in its default configuration, detected 45 of the 95 samples. Setting the protection to High instead of Normal did not get any additional detections. Avast missed the same sorts of stuff that AVG did.
  
• AOL Kaspersky, in its default configuration, detected 59 of the 95 samples. With all settings maxed out, it detected 61 of the 95 samples thanks to the enabling of PUP detection. Detection pattern was similar to KAV7 except it didn't have the heuristic detections.
  
• SUPERAntiSpyware detected 42 of the 95 samples. That was pretty impressive considering that the majority of the samples don't really fit its target genre.
  
  
• I got an interesting question from John... of the files missed by KAV7 in fully-maxed configuration, how many of them were detected by SUPERAntiSpyware? I tested and found that SAS detected 9 of 28 files that KAV7 had not detected. John did further tests with antispyware apps and reports these results**:
  
  • Counterspy v2 - 43 out of 95 (trojans, spamtool, vpp, antispystorm)
  • AVG Anti-Spyware - 39 out of 95 (trojans, hosts file, rootkit, ultimatedefender)
  • a-squared free - 25 out of 95 (trojans, spamtool, virusprotectpro, antispystorm)
  • Spy Sweeper 5.5.7.48 - 24 out of 95 (detected trojans, ultimate cleaner, maxifiles)
• Oh, and Windows Defender detected 3 of the 95 samples *golf clap*
  
• My suggested layered defense works against 95 of the 95 samples
  

I'm not trying to imply that security software is useless, but don't bet the farm on its protection alone when you can add some proactive layers to your security strategy.


How about now, 2 weeks later?

• AntiVir detects 73 of the 95 samples with maxed-out Expert Mode settings
  
• Kaspersky AntiVirus 7 detects 74 of the 95 samples with maxed-out settings including the optional heuristics
  
• AOL Kaspersky detects 70 of the 95 samples with default settings, and 72 of the 95 samples when the potentially-dangerous software option is enabled
  
• SUPERAntiSpyware detects 44 of the 95 samples
  
• Spyware Doctor Starter Edition from Google Pack detects 21 of the 95 samples

*This was about one day after the other tests, so Panda and BitDefender had a potential time advantage to get samples, analyze them and create signatures for them.

**This was almost two days after the first round of tests, so these programs had a potential time advantage to get samples, analyze them and create signatures for them.

[shawn130c]

Wow, those are are some interesting results. Are you still doing testing? I would love to see how BitDefender 10 AV Plus performs in this test. I know there is a 30 day fully functional trial available online, or I also have a download link if you need it.

[mechBgon]

Originally posted by: shawn130c
Wow, those are are some interesting results. Are you still doing testing? I would love to see how BitDefender 10 AV Plus performs in this test. I know there is a 30 day fully functional trial available online, or I also have a download link if you need it.

Sure, I'll grab the BitDefender trialware and get the results added. Since the samples used here are "frozen in time" from August 14th, BitDefender might have an advantage since they'll have had more time to get these samples, analyze them and get the definitions created. But I think the "bigger picture" will be the same in the end... there's so much malware, and some of it is updated so frequently, that it's not realistic to expect antivirus software to singlehandledly protect us from all of it.

edit: BitDefender's result has been added. Panda ActiveScan online scanner has also been added.

[John]

Feel free to add the results of Symantec AV Corporate 10.1.5.5000

How did WinAntivirus 2007 do in the detection rate dept.?

[CalvinHobbes]

Nice work with the testing. One thing that would be interesting to know is what percentage of the malware was detected when using multiple applications.

For example, I run at least 5 antispyware applications on a daily basis. I'd be curious to know what percentage the 5 apps detected. I'd be pretty happy if the combined percentage was above 75% or 80%.

[mechBgon]

Originally posted by: CalvinHobbes
Nice work with the testing. One thing that would be interesting to know is what percentage of the malware was detected when using multiple applications.

I did add one combo result near the end there (KAV7 plus SUPERAntiSpyware).

For example, I run at least 5 antispyware applications on a daily basis. I'd be curious to know what percentage the 5 apps detected. I'd be pretty happy if the combined percentage was above 75% or 80%.

I can remark that a non-Admin user account on Vista, running IE7 Protected Mode, fully patched up, and backed by a Software Restriction Policy, is a powerful proactive spyware-prevention combo with a kill ratio way higher than 75%-80% Free, no performance hit, no need to run anything daily.

[Schadenfroh]

I have been waiting for you to post this!

[MadAmos]

Thanks for the time you spent on this it gives me a little more "ammo" when I am helping friends and family to NOT just click on it. If I hear one more time "I just opened it for a second to see what is was" or " Nothing can infect me I have (insert you favorite Virus software name here)" I will scream isgust; It amazes me how many people think they can just close a window if it doesn't "look" right before any thing bad happens. :roll:

Amos

[Lemon law]

Where is NOD32 which is often rated along with Kaspersky as a top rated active AV program?

But great results even though its very hard to create a scientific replicating test in this area. But you touched on what amounts to the million dollar question when you start to investigate what combination of programs are effective and implied what programs essentially overlap---meaning two programs could be little better than one alone. And the object of the game is to have a small set of programs that gets the most total pieces of typical malware.---and hopefully stop all the more serious variants.

But two jokers in the deck would be process control programs and a windows software restriction policy in place.

[mechBgon]

Originally posted by: Lemon law
Where is NOD32 which is often rated along with Kaspersky as a top rated active AV program?

I should've included it, but I was up until past 3AM as it was. And my main point actually isn't to compare the detection rates of one company to another, it's to demonstrate that none of them make people invincible. Still, I'll add NOD32's result shortly, although they now have a two-day working advantage.

If I hear one more time "I just opened it for a second to see what is was" or " Nothing can infect me I have (insert you favorite Virus software name here)" I will scream

Botmasters everywhere are grateful for their cooperation :evil:

I have been waiting for you to post this!

Thanks bro

[John]

I just installed Antivir free and with all of the settings maxed it detected 75 out of 95. Out of the 20 files that were left over SAS Pro caught another 6 items. Spy Sweeper (without antivirus) 5.5.7.48 detected 1, then a-squared free, AVG Anti-Spyware, and CounterSpy detected 0.

Individual results:

Counterspy v2 - 43 out of 95 (trojans, spamtool, vpp, antispystorm)
AVG Anti-Spyware - 39 out of 95 (trojans, hosts file, rootkit, ultimatedefender)
a-squared free - 25 out of 95 (trojans, spamtool, virusprotectpro, antispystorm)
Spy Sweeper 5.5.7.48 - 24 out of 95 (detected trojans, ultimate cleaner, maxifiles)

[mechBgon]

Originally posted by: John
I just installed Antivir free and with all of the settings maxed it detected 75 out of 96. Out of the 21 files that were left over SAS Pro caught another 6 items. Spy Sweeper (without antivirus) 5.5.7.48 detected 1, then a-squared free, AVG Anti-Spyware, and CounterSpy detected 0.

Individual results:

Counterspy v2 - 43 out of 96 (trojans, spamtool, vpp, antispystorm)
AVG Anti-Spyware - 39 out of 96 (trojans, hosts file, rootkit, ultimatedefender)
a-squared free - 25 out of 96 (trojans, spamtool, virusprotectpro, antispystorm)
Spy Sweeper 5.5.7.48 - 24 out of 96 (detected trojans, ultimate cleaner, maxifiles)

Great info Per Lemon law's suggestion, I just added NOD32 results to the OP as well.

[shawn130c]

Originally posted by: mechBgon

Sure, I'll grab the BitDefender trialware and get the results added. Since the samples used here are "frozen in time" from August 14th, BitDefender might have an advantage since they'll have had more time to get these samples, analyze them and get the definitions created. But I think the "bigger picture" will be the same in the end... there's so much malware, and some of it is updated so frequently, that it's not realistic to expect antivirus software to singlehandledly protect us from all of it.

edit: BitDefender's result has been added. Panda ActiveScan online scanner has also been added.

Thanks for the quick testing, I was hoping for better results, but it didn't do that bad. I hate to ask again but I have always wondered about Clamwin, especially the portable version located here: Clamwin Portable from Portable Apps.

Also to get your samples did you take your virtual machine to known bad sites and just downloaded the infected files or did you use the network monitor to capture the traffic while it was being effected? I would really like to learn how to do this, so I can try some of my own tests, when I have the time.

[mechBgon]

Originally posted by: shawn130c

Thanks for the quick testing, I was hoping for better results, but it didn't do that bad. I hate to ask again but I have always wondered about Clamwin, especially the portable version located here: Clamwin Portable from Portable Apps.

I will run the ClamWin test and put up the result shortly.

Also to get your samples did you take your virtual machine to known bad sites and just downloaded the infected files or did you use the network monitor to capture the traffic while it was being effected? I would really like to learn how to do this, so I can try some of my own tests, when I have the time.

It was a mix. I already had a collection of active malware URLs and bad sites that have scripted exploits, plus I know where to find some common Trojans that get updated several times a day (including some warez/keygens/cracks sites). I installed some of the Trojans on a physical Win2000 system to harvest their actual as-installed files (the installers are also present in the samples too). To get the majority, however...

1) log onto a separate non-Admin user account on Vista x64 with some extra safety precautions added.

2) use Microsoft Network Monitor to watch a Win2000 virtual machine get the stuffings kicked out of it when sent to the exploit sites. The Win2000 VM has some configuration tricks such as certain folders turned "sticky" (stuff can't be deleted from them, due to Permissions tweaks). I generally use out-of-date IE6, but sometimes use FireFox 2.x to see what they've got in store for the FF users.

3) harvest malware files either from the Win2000 VM itself, or by using IE7 Protected Mode on the Vista host system to go fetch them using the info from Network Monitor, whichever is faster (the Win2000 system was almost unusable at several points). Searching the victim OS for files/folders created/modified in the last 1 day is often revealing, as well as looking in the "sticky" folders for new files.

4) also use Vista to go grab the web pages that contain the scripted exploits themselves, which generally use obfuscated JavaScript. View > Source > opens in NotePad > Save As > All Files and append the proper filetype to it (php, htm, etc).

[Schadenfroh]

Originally posted by: shawn130c
I hate to ask again but I have always wondered about Clamwin, especially the portable version located here: Clamwin Portable from Portable Apps.

Thanks for the link, I will have to add that to my automated malware removal script!

[lusher]

50% average Fits with the data using similar methdology (testing only new malware, immediately) at the following sites

http://winnow.oitc.com/AntiVirusPerformance.html
http://www.shadowserver.org/wi...ki.php?n=Stats.Viruses

These are generally dynamic "scanned as found" type of tests or on new malware only, as opposed to monthly scan huge database tests like http://www.av-comparatives.org/ or http://www.av-test.org/index.p...ub=2006&menue=1&lang=0 where antiviruses generally do 90%+

[Jschmuck2]

I'd love to know how you collected this stuff.

[ForumMaster]

interesting. i wonder how ZoneAlarm Internet Security Suite 7.0.337.000 does. it uses the older Kaspersky 6 engine i believe.

[mechBgon]

Originally posted by: ForumMaster
interesting. i wonder how ZoneAlarm Internet Security Suite 7.0.337.000 does. it uses the older Kaspersky 6 engine i believe.

I'd speculate it does the same as AOL Kaspersky, as far as the virus detection goes, if it uses the KAV6 engine. ZoneAlarm might or might not detect some of the malware components being pulled through the firewall; some of the malware is downloaded using the Background Intelligent Transfer Service (BITS). Crafty.

I'd love to know how you collected this stuff.

Expand this pic to full size for a visual. I explained the collection routine somewhat about 4 posts up in this thread.

[ForumMaster]

Originally posted by: mechBgon

I'd speculate it does the same as AOL Kaspersky, as far as the virus detection goes, if it uses the KAV6 engine. ZoneAlarm might or might not detect some of the malware components being pulled through the firewall; some of the malware is downloaded using the Background Intelligent Transfer Service (BITS). Crafty.

yeah i guess. that's actually good. all i know is that it updates itself about every hour or two like the regular kaspersky.

[corkyg]

Excellent report, Mech . . . It is very useful info. I still believe that the ISP is the first line of defense for all malware - and in two years - they have proven their worth. Nationwide uses Postini. I get to delete it from the server after they quarantine it - that includes Spam as well as real malware.

NWN

[mechBgon]

Originally posted by: corkyg
Excellent report, Mech . . . It is very useful info. I still believe that the ISP is the first line of defense for all malware - and in two years - they have proven their worth. Nationwide uses Postini. I get to delete it from the server after they quarantine it - that includes Spam as well as real malware.

NWN

My previous ISP had malware filtration at their end, not just for email but for all network traffic. It didn't work very well, judging by the 8GB stash of files in my FMalware folder. I urge everyone to consider using proactive protection in addition to security software, with non-Admin user accounts being #1 on the list.

Bank Of India website ha><0red, serving exploits all day. Welcome to 2007, everyone :evil:

[corkyg]

Agree fully! But, in the past two years none of that stuff has left my POP3 box. I delete it all there - basically I.m operating on a white list only - everything else gets zapped.

A good second line of defense is MailWasher Pro 5.2. That lets you inspect everything in your mailbox before downloading, and you can delete or bounce gleefully. Only email from people I know gets downloaded and accepted.

BTW - Comcast and AOL both have white list only options. You know of any others?

[mechBgon]

Originally posted by: corkyg
Agree fully! But, in the past two years none of that stuff has left my POP3 box.

None of the malware in the sample set here is normally found in email. Sorry if that wasn't clear

I delete it all there - basically I.m operating on a white list only - everything else gets zapped.

A good second line of defense is MailWasher Pro 5.2. That lets you inspect everything in your mailbox before downloading, and you can delete or bounce gleefully. Only email from people I know gets downloaded and accepted.

BTW - Comcast and AOL both have white list only options. You know of any others?

I haven't researched which ISPs have whitelist-only options, but that's good to know. As I'm sure you know, a whitelist won't save you from being sent an infectuous email if one of your whitelisted senders gets infected with an email worm.

[corkyg]

Originally posted by: mechBgon
As I'm sure you know, a whitelist won't save you from being sent an infectuous email if one of your whitelisted senders gets infected with an email worm.

Oh, yes. Am well aware of that. But, whitelist just about eliminates spam, and evil messages that contain links to web malware - i.e., the current greeting card and sexy photos scams.