Has Anandtech posted flawed test results for AthlonXP 3200+?

[Brunnis]

Hi guys!

I have e-mailed Anand Lal Shimpi several times during the last two weeks, but I've yet to recieve any response. Maybe my mail hasn't reached him? Anyway, here's the problem:

I read the review of the Pentium4 3.46EE that was posted October 31st. In this review, Anand had an AthlonXP 3200+ system and when reading the review I noticed that the performance of this system was extremely bad. Here's a link to the test:

http://www.anandtech.com/cpuch...ts/showdoc.aspx?i=2261

Now look at the results from WinRAR 3.40:

http://www.anandtech.com/cpuch...oc.aspx?i=2261&p=9 (scroll all the way down)

263kB/s for an AthlonXP 3200+? Hmm... Luckily I own a system that can be setup exactly like Anands AXP system:

AthlonXP-M 2600+ @ 3200+ (11*200)
Abit NF7-S 2.0
2*512MB Crucial Ballistix PC3200 @ 200MHz 2-2-2-10 (dual channel enabled)

Now, mine and Anand's systems have equal CPUs, Nforce2 mobos and 2-2-2-10 memory. So, how do the results from WinRAR 3.40 turn out on my machine?

Anandtech's results: 263 kB/s
My results: 398 kB/s

Now that's a difference of more than 50%!

Also, take a look at the gaming results:

http://www.anandtech.com/cpuch...c.aspx?i=2261&p=13

The AXP 3200+ performs horribly in most of the games and some simple calculations show that it would require an AthlonXP running at 2.7-2.9GHz to even match a P4 3.0E in most of those games! If Anand used the default Doom3 "Demo1", those results are way off. I know for a fact that an AthlonXP 3200+ is capable of approximately 70FPS if the graphics card doesn't limit performance.

So, that's a lot of text and you may find it stupid to bother with all this, but I find correct results to be much more interesting! Anyway, I hope that somebody from Anandtech will notice this thread and actually take some time and explain how they got the results. Maybe there's something I have overlooked? If not, the tests need to be performed again.

Ohh, almost forgot! I have a theory on what Anand did wrong. It seems as if the tests were run with a memory divider that divided the mem frequency down to 133MHz (or 266MHz DDR). I did my own tests and my WinRAR 3.40 score went down from 398 kB/s to 256 kB/s. In other words, almost like Anands results. I also tested with Doom3 and my score was within three FPS of Anands score. With memory at 200MHz, as it should be, my score is around 12 FPS higher than Anands result.

EDIT: Sorry if I've posted in the wrong section. I didn't really see any board for site feedback or suggestions.

 

[DAPUNISHER]

Which version forcewares and what vid card are you using?

 

[Brunnis]

Originally posted by: DAPUNISHER
Which version forcewares and what vid card are you using?

Chipset drivers are version 5.10 and vid card is Radeon 9800 Pro 128MB. Motherboard drivers don't affect WinRAR scores from what I've seen (I have also used versions 2.03, 2.45 and 3.13). Graphicscard drivers are 8.07 "Hot fix" or Catalyst 4.12b. Those didn't raise performance from Catalyst 4.8 too much at low resolution/CPU limited situations, though.

Anyway, to test Doom 3 I didn't run the same res as Anand, since it obviously would make my Radeon bottleneck performance pretty heavily. Instead I ran it at 640*480 "High details". It doesn't provide for a direct comparison with Anand's results, but it gives a rough estimate.

Anyway, Anand's WinRAR result is definately not correct (actually far from it), so something is wrong with that setup. Considering that the gametests are way under where they should be, I'd say that those results need to be looked at again. It's strange that noone else seems to have noticed this...

 

[LTC8K6]

I looked around at other benchmarks and the Athlon XP's always seem to do poorly with WINRAR. All of the other benches seemed to agree with Anand. The usual WINRAR settings used were best compression and dictionary 4096KB.

 

[Zebo]

Ya I get 478 with 2.7 AXP and 659 w A64 @2650

 

[Brunnis]

Originally posted by: LTC8K6
I looked around at other benchmarks and the Athlon XP's always seem to do poorly with WINRAR.

Yes, AthlonXP is slow with WinRAR, but fact remains: AthlonXP 3200+ performs close to 400kB/s, not 260kB/s.

Originally posted by: LTC8K6All of the other benches seemed to agree with Anand.

No, I withhold that the results are incorrect. Just look at the initial "Doom3 CPU Battlegrounds" article, where the AthlonXP 3200+ was just 9% slower than the P4 3.0E. Now in the new tests the 3200+ is 24% slower. Additionally, the P4 3.0E is faster in the new tests (76.2 vs 74.9 fps), whereas the 3200+ is much slower (58 vs 68 fps). So, the much lower result of the 3200+ in the newer tests can't be attributed to a less taxing test since the P4 3.0E actually gets a speedup.

Finally, I repeat: Since when does one need an AthlonXP @ 2.7-2.9GHz to match (but not beat) the performance of a P4 3GHz Prescott in games? It's just crazy... Isn't there someone from Anandtech on these forums that can shed some light over this little "problem"?

 

[DAPUNISHER]

Originally posted by: Brunnis
Anyway, to test Doom 3 I didn't run the same res as Anand, since it obviously would make my Radeon bottleneck performance pretty heavily. Instead I ran it at 640*480 "High details". It doesn't provide for a direct comparison with Anand's results, but it gives a rough estimate.

I can't comment on the WinRAR but the above is nonsense, no offense intended. Unless you are testing with the same vid card, drivers, res. and eye candy settings then your D3 scores are pointless due to a huge variance in the testbed config. Your config for the WinRAR test is similar enough to warrant a closer look at that though.

 

[LTC8K6]

No, I withhold that the results are incorrect. Just look at the initial "Doom3 CPU Battlegrounds" article, where the AthlonXP 3200+ was just 9% slower than the P4 3.0E. Now in the new tests the 3200+ is 24% slower. Additionally, the P4 3.0E is faster in the new tests (76.2 vs 74.9 fps), whereas the 3200+ is much slower (58 vs 68 fps). So, the much lower result of the 3200+ in the newer tests can't be attributed to a less taxing test since the P4 3.0E actually gets a speedup.

Are all of these tests with the same boards, drivers, vid cards, vid drivers, res, etc.?
Was it the same DOOM3 bench both times?

Did you use the same WINRAR settings?

The new test used a bit faster hardware for the Pentiums, it seems:

LGA-775 Intel Pentium 4 and Extreme Edition CPUs
2 x 512MB Crucial DDR-II 533 Dual Channel DIMMs 3-3-3-12
Intel D925XECV2 Motherboard (we used the same board for 1066MHz FSB and 800MHz FSB tests
ATI Radeon X800 XT PCI Express

 

[Brunnis]

Originally posted by: DAPUNISHER

Originally posted by: Brunnis
Anyway, to test Doom 3 I didn't run the same res as Anand, since it obviously would make my Radeon bottleneck performance pretty heavily. Instead I ran it at 640*480 "High details". It doesn't provide for a direct comparison with Anand's results, but it gives a rough estimate.

I can't comment on the WinRAR but the above is nonsense, no offense intended. Unless you are testing with the same vid card, drivers, res. and eye candy settings then your D3 scores are pointless due to a huge variance in the testbed config.

Actually, the performance of Doom3 at 1024*768 with either 6800U or X800XT is fast enough to make the CPU an extreme bottleneck. Therefore, neither drivers nor small variations in graphics card performance would affect performance. And it's not like we're talking a small performance drop in the case of the AthlonXP. Its performance just plummels with the use of an X800XT. There's no sense in it. And just look at the rest of the game tests. Even though my test with Doom3 can't show anything concrete, the tests themselves pretty much tell the story! Anybody that has followed the CPU market the past two years should see that those results are off. I say it a third time: Since when does it take a 2.7-2.9GHz AthlonXP to beat a stock P4 3.0E?

BTW, if you don't believe my reasoning behind my own Doom3 test, just take a look at these two Anand articles:

http://www.anandtech.com/cpuch...c.aspx?i=2261&p=13
http://www.anandtech.com/cpuch...oc.aspx?i=2149&p=7

They use different resolution and different graphicscards, yet performance between all processors are very close, as it should be when the CPUs are acting as bottlenecks. All CPUs except the AthlonXP 3200+. Everything points to the fact that something was wrong with the way that system was set up. And, as I said, it seems to correspond very well with memory running async, both with WinRAR and Doom3.

Originally posted by: LTC8K6
Are all of these tests with the same boards, drivers, vid cards, vid drivers, res, etc.?

Nope, but what I wrote above still points to errors in the config of the AXP 3200+ system.

Originally posted by: LTC8K6
Was it the same DOOM3 bench both times?

It would seem so, since the results for all the CPUs (except AXP 3200+) are very similar (within a few fps). I've tried to get Anand to answer this too, but he refuses to answer.

Originally posted by: LTC8K6
Did you use the same WINRAR settings?

There're no settings to be done, since the test is standardised. There are really now doubts that the result from this test is wrong. The important question is: did the same thing that cause the abnornally low WinRAR result hinder performance in other tests? Also, the fact that WinRAR ran this slow would indicate a problem with memory speed, since WinRAR is extremely dependant on memory performance.

Don't anybody know anyone on this forum that writes for Anandtech that could explain a thing or two (or tip a person that could)? It would seem like it's in Anandtech's best interest to look into this.

 

[LTC8K6]

Well, I don't think you've made a case but I await clarification.

 

[Brunnis]

Originally posted by: LTC8K6
Well, I don't think you've made a case but I await clarification.

Do you know anybody at Anandtech that is a member of this forum that I coud PM and point to this thread?

 

[LTC8K6]

Wesley Fink and Evan Lieb come to mind. I have seen both of them post here.

Wesley is motherboard and memory, and Evan is SFF at Anandtech.

 

[acivick]

One thing about the CPU Battlegrounds article vs. the P4 article that I noticed was that one used a 6800Ultra and one used an X800XT. This could be an important fact when comparing the performance in the two articles. Due to the 6800 dominance in Doom3, you can more clearly see the performance difference between processors when using cards based on this architecture. I'm willing to hazard a guess that if you were to benchmark HL2 instead with the X800XT systems that performance differences would be similar to the Battlegrounds article, due to THAT card's particular affinity to HL2. I think the mistake you're running into with your argument on these differences between reviews, at least as far as Doom3 is concerned, is that you assume both systems would have similar differences in performance, even though the systems have different hardware under the hood, which may not be the case.

You still may have something with that WinRAR benchmark though, since its performance is much more directly influcenced by fewer factors. There might have been an anomaly there that wasn't immdiately obvious. I couldn't immediately find another article here that runs the WinRAR benchmark to compare its accuracy with though.

 

[Brunnis]

Originally posted by: acivick
One thing about the CPU Battlegrounds article vs. the P4 article that I noticed was that one used a 6800Ultra and one used an X800XT. This could be an important fact when comparing the performance in the two articles. Due to the 6800 dominance in Doom3, you can more clearly see the performance difference between processors when using cards based on this architecture. I'm willing to hazard a guess that if you were to benchmark HL2 instead with the X800XT systems that performance differences would be similar to the Battlegrounds article, due to THAT card's particular affinity to HL2. I think the mistake you're running into with your argument on these differences between reviews, at least as far as Doom3 is concerned, is that you assume both systems would have similar differences in performance, even though the systems have different hardware under the hood, which may not be the case.

Actually, if you look at both articles, all CPUs systems are indeed performing very similar between the two tests. This is natural, since the different graphicscards won't affect performance. Both the 6800U and X800XT are capapble of framerates well below those attained by even the fastest CPUs at 800*600 or 1024*768 in those tests. The only thing stopping framerate from reaching the maximum capabilities of the graphicscards are the power of the CPU and memory systems. There is only one single system of all the ones that perform very different between the tests and that's the AthlonXP 3200+. We know from the first Doom3 test that it is indeed capable of 68 frames per second if the graphicscard is not the bottleneck. What you're saying is basically that the X800XT is indeed bottlenecking the performance of an AthlonXP 3200+.

It still hasn't ceased to amaze me that not one single person in this thread seems to be able to stop for a second and actually contemplate if it's reasonable to think that the P4 Prescott suddenly has become almost as fast, clock for clock, as the AthlonXP in gaming...

Originally posted by: acivick
You still may have something with that WinRAR benchmark though, since its performance is much more directly influcenced by fewer factors. There might have been an anomaly there that wasn't immdiately obvious. I couldn't immediately find another article here that runs the WinRAR benchmark to compare its accuracy with though.

Actually, there's no need to verify the result. I have run the test on my AthlonXP Barton @ 3200+ and it yields 390-400kB/s every time. I've also run the test on my old 1700+ @ 2.3GHz and it also scored somewhere between 350 and 400kB/s.

Also, what needs to be considered is that if there's some kind of error in the system setup that causes such low performance in WinRAR, it's very likely to affect other programs too. The gaming results really do suggest bad memory performance (just like WinRAR does), but only the folks at Anandtech could tell really.

I'm gonna get in contact with Wesley Fink and see what he has to say about it.

EDIT: I decided to PM Evan Lieb about this issue. Let's hope he get's back with some information!

 

[acivick]

Actually, if you look at both articles, all CPUs systems are indeed performing very similar between the two tests. This is natural, since the different graphicscards won't affect performance. Both the 6800U and X800XT are capapble of framerates well below those attained by even the fastest CPUs at 800*600 or 1024*768 in those tests. The only thing stopping framerate from reaching the maximum capabilities of the graphicscards are the power of the CPU and memory systems. There is only one single system of all the ones that perform very different between the tests and that's the AthlonXP 3200+. We know from the first Doom3 test that it is indeed capable of 68 frames per second if the graphicscard is not the bottleneck. What you're saying is basically that the X800XT is indeed bottlenecking the performance of an AthlonXP 3200+.

What I was trying to say, which I may not have been completely clear about, is that you should not compare numbers between two different systems in a gaming environment, even if (understandably) the only difference is the video card, especially top end cards. There just might be something going on under the surface, for instance optimizations that are targeted to run on P4 and A64 machines, but not AXP machines. These sorts of optimizations can be at the driver level or even in the game itself, as has been covered before in many articles. On the other hand, you may be correct in that it was some sort of misconfiguration, such as running the RAM at a ratio, which the reviewer may or may not have known about. All I'm saying is that it is tough to say with the information we are provided with.

Actually, there's no need to verify the result. I have run the test on my AthlonXP Barton @ 3200+ and it yields 390-400kB/s every time. I've also run the test on my old 1700+ @ 2.3GHz and it also scored somewhere between 350 and 400kB/s.

With respect to verification, what I was suggesting was to look for another article to see if there was something wrong with the AXP configuration they used in the P4 article. For instance, if the same or similar machine gave different results in another article, you could easily point to a configuration error in the most recent article, whereas if similar numbers are given in some other article, further investigation might be needed to find the exact problem.

But essentially, you are correct, only the people who ran the test and know how the configuration was set up can answer for sure, at least insofar as knowing the exact conditions in which the test was run.

EDIT: And I do think that newer Intel chips, while they probably do not quite compete clock for clock with an AXP, still have come a long way from being trumped by it in gaming, especially with recent advances in the bandwidth department (something the P4 eats up like candy). I haven't done any hard core calculations for proof, just my .02.

 

[RussianSensation]

Sometimes benchmarks are different from real world usage.

For example:

I have 2.6@3.06ghz P4 "C" memory running at 5:4 ratio (235FSB:188) - 2-2-2-5

I just ran WinRAR 3.40 "default" bench and i get 299 KB/sec

Now my friend has A64 3000+ with a 36Gig Raptor HDD

I have 80GB WD 8mb HDD instead.

Task at hand:

Extract 593MB .iso archived Office 2003 using WinRAR 3.40 onto the desktop

My computer: 1 min 52 seconds

His computer: 2 min 57 seconds (After system reformat onto clean HDD)

Explanation - given that A64 is supposed to dominate WinRAR and his HDD is faster - none.

Basically - if you are happy with your system performance and it performs better than what others have tested, then be happy

 

[VirtualLarry]

Interesting issue, at least the WinRAR scores. My system is an Athlon XP TBred-A running at 166Mhz x 10.0, RAM/FSB sync at 166Mhz too. Without shutting down my background programs, WinRAR 3.40 gave me a score of 163, although the number under it bounces between 160-odd and 180-odd. However, if I take my calculated score, and extrapolate linearly for both CPU and memory FSB, and assume that the AT testbed rig was running 200Mhz x 11.0, then the numbers work out. Using such gross scaling assumptions may not be valid though, I would have to do more WinRAR benchmarking to test that theory.

 

[stratman]

Good on ya for discovering this Brunnis, I hope you manage to talk to the ATech'ers =)

 

[dug777]

It will be kinda interesting to see if they shout u a reply....it would be good manners

 

[Brunnis]

Originally posted by: dug777
It will be kinda interesting to see if they shout u a reply....it would be good manners

Yeah, let's hope they atleast say something! I have yet to recieve an answer in my inbox.

BTW, I tried CS Stresstest yesterday and I get better results than Anand at 1024*768, despite that the fact that my 9800 Pro severely bottlenecks performance. Lowering the resolution a little, to see how the CPU really performs, I got over 120 fps. Now, just as with Doom3, this is merely an indication that things weren't quite working as they should. I may have a newer version of CS Stresstest that's been massively optimised, but I somehow doubt that.

 

[LTC8K6]

Evan is the small form factor guy, so I don't know if he'd be able to do much.

Wesley would probably have an answer if anyone would. Since he's the mobo and mem guy around here he might be able to answer your question about the mem divider.

 

[Brunnis]

Originally posted by: LTC8K6
Evan is the small form factor guy, so I don't know if he'd be able to do much.

Wesley would probably have an answer if anyone would. Since he's the mobo and mem guy around here he might be able to answer your question about the mem divider.

Okay, I'll send him the PM too. The reason I picked Evan is that he seems to post at the forum a whole lot more...

 

[Anand Lal Shimpi]

Thanks for the manner in which you brought your question to light, as far as I remember there were no issues with the nForce2 setup we used for the Athlon XP 3200+ and the memclock should have been DDR400 but I can definitely double check it. On first glance the Doom 3 numbers do not look suspect but I'll look at them again as well as the WinRAR issue. Please be patient with me however as I'm in the middle of a handful of Half Life 2 articles as well as a shipping SLI performance article as well. I won't be able to get to it until after Thanksgiving most likely.

Take care,
Anand

 

[Fern]

We'll I've been following this since you first posted it Brunnis. See the rig in my sig.

I'm very impressed Anand got back to you.

Hope I catch it when he responds back.

Fern

 

[Brunnis]

Originally posted by: Anand Lal Shimpi
Thanks for the manner in which you brought your question to light, as far as I remember there were no issues with the nForce2 setup we used for the Athlon XP 3200+ and the memclock should have been DDR400 but I can definitely double check it. On first glance the Doom 3 numbers do not look suspect but I'll look at them again as well as the WinRAR issue. Please be patient with me however as I'm in the middle of a handful of Half Life 2 articles as well as a shipping SLI performance article as well. I won't be able to get to it until after Thanksgiving most likely.

Take care,
Anand

Ahhh, finally! I'm very glad you answered! I'm looking forward to see what get from the double check. I've never seen an AthlonXP 3200+ score under 300kB/s in WinRAR, so I'm fairly confident you'll find something!

BTW, looking forward to the HL2 articles!

EDIT: Hey, Fern, you're running an AthlonXP @ 3200+! What do you get in the WinRAR test?