Internet Security software test, i'll keep you updated

[ak47gen]

Good evening ladies and gentleman. I was getting tired of bias AV/AS reviews. There are lots of websites which have many opinions and almost every one are different. My goal is to create a fair test which tests 4 categories, possibly 5. The categories are;
1. Resourcefulness
2. Scan Speed
3. Scan detection
4. False Positives
5. (if I find the right program) test the responsiveness of the OS (program execution benchmark tool)

A buddy of mine showed me a bunch of malware he collected when doing freelance computer repair and I had a collection from the same means. In total I now have around 40,000 variants of malware.

The test consist of the following Internet Security software which I choose IS instead of AV to give it a fair test,excluding ClamAV (ClamAV doesn't have an IS suite). The list consist of;
Avast!
Avira
AVG
BitDefender
ClamAV
Kaspersky
Mcafee
Nod32
Panda
Trend Micro
Webroot

The computer is a Fujistu lifebook n3530 with 1 GB of memory.

The OS of testing is a fresh install of the windows XP pro with SP3 installed and customizations ran on it to speed things up. Which has a folder injected into the OS that contains all the malware. After that I imaged it with Acronis true image to ensure reliability. Each test is restarted from the same image which has only the OS and the malware.

I will keep you updated, but as of now any suggestions on a execution benchmark program?

Thank you
-AK

 

[bsobel]

So your not including Symantec which is the fastest/lightest in the market with the 09 releases? Second, what background do you have in testing AV/AS products. What methodology are you using, actual infectors, scan only, heuristics, behaviour blocking?

 

[ak47gen]

I do apologize I did leave NIS out from the list but I did download it and had it part of the image. I will correct the above statement. Because I am curious to see the results, since 2003 norton seem to be a slower and more resourceful software. But I heard good things about the new one and I must try it.

 

[bsobel]

Originally posted by: ak47gen
I do apologize I did leave NIS out from the list but I did download it and had it part of the image. I will correct the above statement. Because I am curious to see the results, since 2003 norton seem to be a slower and more resourceful software. But I heard good things about the new one and I must try it.

05 and 06 where the worst. 07 was an improvement, 08 was good. 09 seriously is amazing (However, I'm biased, I work for SYMC)

 

[ak47gen]

I do have to admit they are good at detecting and always have been, at least to http://www.av-comparatives.org/ which is the only source i trust so far.

 

[ak47gen]

and about your second question on your first post, that i over looked. I have no background on virii.

But here is how the test go, as of now I am gonna keep it simple. I plan to just run a drive scan with default setting (i keep it on default because if the developer believes that should be the standard then I shall use his standard), the only settings i will change will be the prompt to remove, so it won't affect the scan time.

I will measure the time and amount of the scan and then look through the log and see if any know false positives come up.

The other thing I will test is after the install of each program I will pull up task manger and see the difference of memory use.

 

[lxskllr]

I'd like to see Comodo's new A/V tested also. It didn't do particularly well while in beta. I was wondering if it's improved at all.

 

[ak47gen]

If i plan to do a another go at this i'll go ahead and do that program. As of now I am mostly doing this to test scan speed and resources, the detect rate i realized won't be as comparable to major websites like av-compartive, the malware is mostly in there to give the AV obstacles to show scan time efficiency. Most sites seems to not care for scan speed and resourcefulness testing, but when you buy the software I expect it to functional as a whole program not just a scanner.

 

[ak47gen]

Current stasis I have 7 programs done avast!, AVG, Avira, Bitdefender, Nod32, Trend, and Panda. I am running into some issues, the last 2 scans seem to take an extreme time. After this test i plan to run DFT i think my HDD is dying out. The other issue is norton didn't like the image i create, it was a install with customized services (to speed things up), so far no other program came across the issues, but norton won't install, i might have to make another image.

The info I gather is, never use AVG, by far the worst, it uses 8.4x the memory then nod32, avg picked up 1402 on a quick scan while nod32 did 47283 on a quick scan. There is no doubt in my mind i would rather pay the $40 instead of the AVG. On the other note Avast didn't do so bad and I was quite surprise with avira, its was about as compromisable as Nod32.

Thanks for reading this I hope you enjoy I will have more done and once complete i will have .png's of all the image (scan results and taskmanager)
-AK

 

[ak47gen]

I am currently doing default scans then i will be doing max setting scans. Resources is physical memory use in KB. Another note since these are all fresh installs system has little (~10) cookies and all programs are internet security Suites. I will keep you updated with more when they come in.

Stand system
resources 116344

Avast!
Resources 307260
Scan Detect 39834
Scan speed 30:41

AVG
Resources 589184
Scan detect 1402
Scan speed 27:35

Avira
Resources 214292
Scan detect 43006
Scan speed 25:36

Bitdefender
Resources 288052
Scan detect 71034
Scan speed 3:58:51

Kaspersky
Resources 217880
Locked up on 2 scans

nod32
Resources 169244
Scan detect 47283
Scan speed 01:19:00

Panda
Resources 375740
Scan detect 21672
Scan speed 38:00

Trend Micro
Resources 298504
Scan detect 72733
Scan speed 09:04:00

Webroot
Resources 275100
Scan detect 24524
Scan speed ~40 mins (lost log)

 

[CuriousMike]

So, according to your data, for pure detection, Bitdefender and TrendMicro are the clear winers. By, like, A LOT.

 

[ak47gen]

the simple anwser to that question is a no. I ran so far all tests on default scans the reason why nod32 only ran for an hour instead of trend running for 10 is because trend defaults settings are maxed.

 

[CuriousMike]

nod detected 47k pieces of malware.

bitdefender and trendmicro both detected > 70k pieces of malware.

For me, raw detection is more important than speed.

 

[ak47gen]

Your not getting what i am saying. If i turned off heuristics on trend then the scan time of trend will still be more and won't detect as much. Wait for round 2 when i have everything on max settings.

 

[bsobel]

Originally posted by: CuriousMike
nod detected 47k pieces of malware.

bitdefender and trendmicro both detected > 70k pieces of malware.

For me, raw detection is more important than speed.

And here is the problem with tests like these, unless you know exactly what is being tested and how, its not really a useful test. A couple questions, points to consider.

a) The OP says "40,000 variants of malware", yet is reporting 70k+ detections. Did he magically get another 30k samples, if so, from where and why weren't they disclosed originally.

b) Who decided these 40k (or 70k) items were indeed malware?

c) Some AV scanners, when they see lots of detections go into a mode where the heuristics really get turned up (slowing the scan overall). So the 'scan speed' may not be a fair comparison vs an entire drive scan if only 1 or 2 bad files where found.

 

[ak47gen]

Bsobel is correct there are probably lots of false postive when heuristics are on. Though i did state over 40k in maware variants. There is also repets of virii. Yes i know its unfair to do scan defaults because programs sometime have heuristics on by default. But I do this for a reason. The AV companies had them default so i shall leave the first round as default to show that when (most) people scan they will not change those setting.

But testing is still to early, round 2 will begin once i get my virtualbox linux OS up in running. I do this to speed up time of scan (heuristics) take longer, and yes round 2 will be all max settings.