An informal snapshot of virus-detection rates on some fresh real-world malware

[mechBgon]

I actually don't worry as much about infectuous Spam. I think the security vendors have a much better "feed" of email-borne malware, for the obvious reason. Also, email clients such as Outlook or Outlook Express would put HTML email in the Restricted Sites zone, and restrict possibly-dangerous filetypes by default. The user can wave off on opening an attachment or the email as a whole, and there's plenty of Spam filtration going on to help subdue that angle of attack, too.

The stuff in my sample set varies, but a considerable amount of it is the type of stuff you might encounter if, say, you browsed http://forums.anandtech.com and it turned out AnandTech's advertising-banner supplier was hacked, or the site itself.

The hacked-advertiser scenario happened at Tom's Hardware Guide this year, and I could reel off more instances of compromised sites, including http://pics.bbzzdd.com, Asus.com, The Register, Microstar, one of Mozilla's mirrors, a page at Microsoft.com, and some of the >10,000 sites reportedly hacked using MPack. So IMHO the folks who think "oh, but I never visit dangerous websites" should be prepared in case a dangerous website visits them because that is one of the bad guys' new business models. And as I think the results of my test show, reactive protection alone is not necessarily going to stop an attack.

 

[dunringill]

I would be interested in seeing how eEye Blink Personal does in your tests. Any chance of that?

** By the way, this would only be relevant if you are testing with XP since Blink is not yet Vista compatible.

 

[SunnyD]

Hey mechBgon, I know this is a "task" that would probably take longer, but I would love to see your test results for a CURRENT fresh VM install of XP SP2, with IE7, all updates and roll-ups, along with each of the respective AV's and AntiSpyware solutions to see how they would do directly against some of these malware battery sites live.

Please note, this test would be WITHOUT any sort of pro-active measures on your part - just a typical home end-user setup with an administrator account, firewall, antivirus and antispyware solution that grandma would use.

I'd suggest starting with the BIG names in AV, Norton, McAfee, Kaspersky, and Microsoft (since these are the most readily recognized RETAIL options). Then I'd throw in Avira, AVG and Avast!.

 

[mechBgon]

Originally posted by: dunringill
I would be interested in seeing how eEye Blink Personal does in your tests. Any chance of that?

** By the way, this would only be relevant if you are testing with XP since Blink is not yet Vista compatible.

At this point the set of samples is obsolete, but if I do another test, I'll try to include that. But for about the 40th time, the point here is simply to show that security software alone is not an infallible defense, so it's more of a Pass/Fail test in my viewpoint. And nobody passed.

Hey mechBgon, I know this is a "task" that would probably take longer, but I would love to see your test results for a CURRENT fresh VM install of XP SP2, with IE7, all updates and roll-ups, along with each of the respective AV's and AntiSpyware solutions to see how they would do directly against some of these malware battery sites live.

Please note, this test would be WITHOUT any sort of pro-active measures on your part - just a typical home end-user setup with an administrator account, firewall, antivirus and antispyware solution that grandma would use.

I'd suggest starting with the BIG names in AV, Norton, McAfee, Kaspersky, and Microsoft (since these are the most readily recognized RETAIL options). Then I'd throw in Avira, AVG and Avast!.

Hmmm. While the result would be interesting, it would be more my style to test a live default Vista system with IE7 Protected Mode, Windows Firewall, and no security software. Realize that it would be stupendously difficult to mimic Grandma's system for testing because the bad guys aren't just targetting Windows, or FireFox, or IE, or a specific version of IE... they're also gunning for WinZip, WinAmp, Java, QuickTime, Acrobat Reader, and so forth. So the attacks that my test system would experience would depend on what-all it's got installed, and there are zillions of permutations, and I don't even get the same attacks every time on the single system that I use now. Not good scientific repeatability there.

If that's not enough, polymorphic JavaScript has been reported, so visiting the same site 10 times might net you 10 different HTML exploits that all do the same thing overall, but with different code. Again, not repeatable in "live-fire" testing.

And then there's the user as a vulnerable point. If grandson Randy is using Grandma Prudence's computer while she's out buying a cake for his 15th birthday... well... he'd better be on a Limited or Standard account when he finds the video that says "You must install Video ActiveX Object to view this movie." :evil: Because the average detection rate for Zlob trojans is mighty low.

Bottom line, my message is to make the jump to non-Admin operation, abandon risky behaviors (warez/etc), expand software updating to include third-party software, and not assume that security software is infallible. Doing a proper AV comparison, even if I had the time and resources to do it properly, is at cross purposes with that goal.

 

[NYCSTE2003]

this thread is amazing and you did some great work all of you. please feel free to message me on AIM if you need more testing done. i have billions of programs installed on my computer and am currently trying to fix an issue i got hehe.

 

[SunnyD]

Originally posted by: mechBgon

Hey mechBgon, I know this is a "task" that would probably take longer, but I would love to see your test results for a CURRENT fresh VM install of XP SP2, with IE7, all updates and roll-ups, along with each of the respective AV's and AntiSpyware solutions to see how they would do directly against some of these malware battery sites live.

Please note, this test would be WITHOUT any sort of pro-active measures on your part - just a typical home end-user setup with an administrator account, firewall, antivirus and antispyware solution that grandma would use.

I'd suggest starting with the BIG names in AV, Norton, McAfee, Kaspersky, and Microsoft (since these are the most readily recognized RETAIL options). Then I'd throw in Avira, AVG and Avast!.

Hmmm. While the result would be interesting, it would be more my style to test a live default Vista system with IE7 Protected Mode, Windows Firewall, and no security software. Realize that it would be stupendously difficult to mimic Grandma's system for testing because the bad guys aren't just targetting Windows, or FireFox, or IE, or a specific version of IE... they're also gunning for WinZip, WinAmp, Java, QuickTime, Acrobat Reader, and so forth. So the attacks that my test system would experience would depend on what-all it's got installed, and there are zillions of permutations, and I don't even get the same attacks every time on the single system that I use now. Not good scientific repeatability there.

If that's not enough, polymorphic JavaScript has been reported, so visiting the same site 10 times might net you 10 different HTML exploits that all do the same thing overall, but with different code. Again, not repeatable in "live-fire" testing.

And then there's the user as a vulnerable point. If grandson Randy is using Grandma Prudence's computer while she's out buying a cake for his 15th birthday... well... he'd better be on a Limited or Standard account when he finds the video that says "You must install Video ActiveX Object to view this movie." :evil: Because the average detection rate for Zlob trojans is mighty low.

Bottom line, my message is to make the jump to non-Admin operation, abandon risky behaviors (warez/etc), expand software updating to include third-party software, and not assume that security software is infallible. Doing a proper AV comparison, even if I had the time and resources to do it properly, is at cross purposes with that goal.

See, that's exactly what I'm saying. A typical "grandma" system (out of the box) shouldn't be too hard. Honestly, it doesn't matter much if there's specific malware targetting specific software at this point... but you take the "ubiquitous" installations, which are Windows, IE, Java, and Acrobat, throw them on a machine and not worry about what granny's grandson is going to put on the machine. It's not so much what kind of compromising has occurred on a machine already, it's what can someone expect with a typical brand new machine without being educated.

 

[mechBgon]

Originally posted by: SunnyD

Originally posted by: mechBgon

Hey mechBgon, I know this is a "task" that would probably take longer, but I would love to see your test results for a CURRENT fresh VM install of XP SP2, with IE7, all updates and roll-ups, along with each of the respective AV's and AntiSpyware solutions to see how they would do directly against some of these malware battery sites live.

Please note, this test would be WITHOUT any sort of pro-active measures on your part - just a typical home end-user setup with an administrator account, firewall, antivirus and antispyware solution that grandma would use.

I'd suggest starting with the BIG names in AV, Norton, McAfee, Kaspersky, and Microsoft (since these are the most readily recognized RETAIL options). Then I'd throw in Avira, AVG and Avast!.

Hmmm. While the result would be interesting, it would be more my style to test a live default Vista system with IE7 Protected Mode, Windows Firewall, and no security software. Realize that it would be stupendously difficult to mimic Grandma's system for testing because the bad guys aren't just targetting Windows, or FireFox, or IE, or a specific version of IE... they're also gunning for WinZip, WinAmp, Java, QuickTime, Acrobat Reader, and so forth. So the attacks that my test system would experience would depend on what-all it's got installed, and there are zillions of permutations, and I don't even get the same attacks every time on the single system that I use now. Not good scientific repeatability there.

If that's not enough, polymorphic JavaScript has been reported, so visiting the same site 10 times might net you 10 different HTML exploits that all do the same thing overall, but with different code. Again, not repeatable in "live-fire" testing.

And then there's the user as a vulnerable point. If grandson Randy is using Grandma Prudence's computer while she's out buying a cake for his 15th birthday... well... he'd better be on a Limited or Standard account when he finds the video that says "You must install Video ActiveX Object to view this movie." :evil: Because the average detection rate for Zlob trojans is mighty low.

Bottom line, my message is to make the jump to non-Admin operation, abandon risky behaviors (warez/etc), expand software updating to include third-party software, and not assume that security software is infallible. Doing a proper AV comparison, even if I had the time and resources to do it properly, is at cross purposes with that goal.

See, that's exactly what I'm saying. A typical "grandma" system (out of the box) shouldn't be too hard. Honestly, it doesn't matter much if there's specific malware targetting specific software at this point... but you take the "ubiquitous" installations, which are Windows, IE, Java, and Acrobat, throw them on a machine and not worry about what granny's grandson is going to put on the machine. It's not so much what kind of compromising has occurred on a machine already, it's what can someone expect with a typical brand new machine without being educated.

To start with, I'd hope brand-new machines will not be available with WinXP for much longer. Symantec investigates current threat survivability on a default Vista installation ~95% kill ratio without any security software.

< Darth Vader > Impressive. < / Darth >

Vista is the right Windows for the uneducated user running a default out-of-the-box setup.

At any rate, for the test to be meaningful as a means of comparing Brand A to Brand B (as opposed to simply demonstrating that neither Brand A nor Brand B are an infallible defense), it needs to be scientific and repeatable. I got as close to that as practical by feeding the exact same HTML and PHP exploit files to all of the software tested here, but I'm not equipped to set up my own captive MPack server for exhaustive live testing If I do come up with a way, and the time to do it, then I'll certainly post the results.

 

[SunnyD]

Originally posted by: mechBgon
To start with, I'd hope brand-new machines will not be available with WinXP for much longer. Symantec investigates current threat survivability on a default Vista installation ~95% kill ratio without any security software.

< Darth Vader > Impressive. < / Darth >

Vista is the right Windows for the uneducated user running a default out-of-the-box setup.

At any rate, for the test to be meaningful as a means of comparing Brand A to Brand B (as opposed to simply demonstrating that neither Brand A nor Brand B are an infallible defense), it needs to be scientific and repeatable. I got as close to that as practical by feeding the exact same HTML and PHP exploit files to all of the software tested here, but I'm not equipped to set up my own captive MPack server for exhaustive live testing If I do come up with a way, and the time to do it, then I'll certainly post the results.

Excellent, that's all we can ask for as time is valuable to anybody. I'd certainly like to see the results. Now, if I could just get myself to stop using an admin account for my daily tasks... (Vista is getting reinstalled tonight anyway, so I'll probably set it up the right way this time!)

 

[NYCSTE2003]

soo let me ask a question.

dont use admin account to do all your normal tasks because if you get infected the virus thingy has full reign on your system?

so instead make a user account and give it full or limited access? but setup a password maybe that if i remember like linux allows you to alter admin files?

 

[mechBgon]

Originally posted by: NYCSTE2003
soo let me ask a question.

dont use admin account to do all your normal tasks because if you get infected the virus thingy has full reign on your system?

I don't know a better way to explain this than the handgun analogy. How dangerous is a handgun with no ammo? However, if the bad guys convince you to download and run something that has malware inside, like that keygen you downloaded and ran, that's a Trojan Horse. They simply convinced you to load the ammo, point the gun at yourself and fire it.

so instead make a user account and give it full or limited access? but setup a password maybe that if i remember like linux allows you to alter admin files?

Read the page again. You want to set up a new user account that's an Administrator, then switch your user account down to Limited. Then use your Limited account for whatever you can, and use the Admin account only when really necessary (and not for anything risky).

If you run P2P programs, which it sounds like you do, you could also set up a separate Limited account that has a password, and set your P2P program's Services to run under that account, so that if they get exploited, the bad guys not only have an unloaded handgun, but it's in a different room in the building than you are, so to speak.

 

[NYCSTE2003]

Originally posted by: mechBgon

Originally posted by: NYCSTE2003
soo let me ask a question.

dont use admin account to do all your normal tasks because if you get infected the virus thingy has full reign on your system?

I don't know a better way to explain this than the handgun analogy. How dangerous is a handgun with no ammo? However, if the bad guys convince you to download and run something that has malware inside, like that keygen you downloaded and ran, that's a Trojan Horse. They simply convinced you to load the ammo, point the gun at yourself and fire it.

so instead make a user account and give it full or limited access? but setup a password maybe that if i remember like linux allows you to alter admin files?

Read the page again. You want to set up a new user account that's an Administrator, then switch your user account down to Limited. Then use your Limited account for whatever you can, and use the Admin account only when really necessary (and not for anything risky).

If you run P2P programs, which it sounds like you do, you could also set up a separate Limited account that has a password, and set your P2P program's Services to run under that account, so that if they get exploited, the bad guys not only have an unloaded handgun, but it's in a different room in the building than you are, so to speak.

very cool i fully understand never used login accounts really myself so i might need alittle help from you sometime to set something like that up maybe after i reformat sometime really soon cuz it appears i cannot fix my problems. but we shall see i do enjoy trying to fix them.

 

[soonerproud]

Any chance of you updating this test (and thread) for current products and threats? People constantly ask me what security products they should use and this information has been crucial to my suggestions. Thank you for all the hard work you have done on this. It is greatly appreciated.

 

[law9933]

Has one of the pros compared MalwareBytes to say SuperAntispyware or SpyBot? It is a great free program, that seems to fix what the others do not.

 

[MadScientist]

Nice work Mech. The security setup on my computer is similar to yours., but I have found it almost impossible to convince the owners of infected computers that I have cleaned /re-installed the OS that this is how their security should be set up. I have given them your website link and even printed out your "How to secure your Windows PC" page for them. These are people, the grandpas, grandmas and others, with very limited/no computer knowledge running Windows XP.

Given your results, this brings us back to the much discussed question: To clean or re-install the OS? It takes an arsenal of AV and AS weapons to clean a computer today, and how can you be certain it's totally clean.

 

[mechBgon]

Originally posted by: soonerproud
Any chance of you updating this test (and thread) for current products and threats?

I'm afraid I can't. It calls for broadband and now I'm stuck on dial-up (long story)

but I have found it almost impossible to convince the owners of infected computers that I have cleaned /re-installed the OS that this is how their security should be set up.

You have my sympathies. It's not easy to convince them sometimes

 

[Chiefcrowe]

Thank you sir, that was fascinating!


Now, has there been any testing done on Kaspersky 2009 yet?

 

[bsobel]

But for about the 40th time, the point here is simply to show that security software alone is not an infallible defense, so it's more of a Pass/Fail test in my viewpoint. And nobody passed.

Agreed, but the test is flawed, static file scanning is only part of almost all of the security products you tested...

it doesn't account for some of the proactive-defense capabilities that some of the softwares would provide in real-li

I realize you pointed that out, but it has a *huge* effect on the detection rates you are using and making conclusions based on.

 

[danzigrules]

Originally posted by: mechBgon

Originally posted by: soonerproud
Any chance of you updating this test (and thread) for current products and threats?

I'm afraid I can't. It calls for broadband and now I'm stuck on dial-up (long story)

but I have found it almost impossible to convince the owners of infected computers that I have cleaned /re-installed the OS that this is how their security should be set up.

You have my sympathies. It's not easy to convince them sometimes

Any chance of an update now?

/me slaps mechBgon with a TGB trout

 

[mechBgon]

Originally posted by: danzigrules

Originally posted by: mechBgon

Originally posted by: soonerproud
Any chance of you updating this test (and thread) for current products and threats?

I'm afraid I can't. It calls for broadband and now I'm stuck on dial-up (long story)

but I have found it almost impossible to convince the owners of infected computers that I have cleaned /re-installed the OS that this is how their security should be set up.

You have my sympathies. It's not easy to convince them sometimes

Any chance of an update now?

/me slaps mechBgon with a TGB trout

I was getting a bit OCD with the malware hunting so I stopped doing it, and therefore I don't have a ready supply of malware anymore

/me "fishes" around for...

ahhh yes, the mIRC logs! Random snippet for you, ol' timer

*** danzigrules has joined #teamanandtech
<mechBgon> hey danzig
<danzigrules> hey mech
<charrison> http://www.silverchat.com/~sil...ybabtu/019_kelster.jpg
<mechBgon> lol!
<danzigrules> anyone here care to help me burn an iso file so that i can install linux on a different pc?
<mechBgon> set up them the missle, or set up them not the missle, there is no "try"
<danzigrules> ezcd creator 4 software
<mechBgon> no experience with EZ CD creator myself
<danzigrules> this will be the 4th cd burnt too
<danzigrules> nero?
<mechBgon> ottawanker are you around?
<mechBgon> I don't see any other Linux dudes except LD and he's off playing DF
<mechBgon> or wait, maybe TwoFace knows
<danzigrules> no not TF
<danzigrules> he's been driving me nuts
<danzigrules> hehe
<mechBgon> haha!
<danzigrules> told me to find winoncd and all i can find are foriegn versions
<mechBgon> does EZ CD Creator's Help have any info?
<danzigrules> not for a smchumk like me
<danzigrules> well idiot then
<danzigrules> hehe
<danzigrules> I have tried to burn the iso file itself, tried to change the files with winimage and burn
<danzigrules> and when i get the boot disk to boot it says the cd doesn't contail the distro
<danzigrules> contain

 

[Scouzer]

Too bad you can't update this with all the 2010 suites