Cool, Anand finally comments on SYSMark 2002

[SuperCyrix]

Forgive me if this is a repost of a repost of a repost. I have little idea what people in the general hardware section is talking about.
Anybody care to sum up this SYSMark 2002 thing?


Here

08/23/2002 1:22 PM (NEW!) Message OptionsSubscribe to ThreadE-Mail ThreadPrint Thread


I just thought I'd chime in here now that data seems to be leaking out about this.

Here's where things stand currently:

1) AMD went around to reviewers and distributed the PDF that you've seen posted in this thread. The data AMD produced is verifiable as I have done my own verification of the tests in house.

2) Here's the main problem: SYSMark 2001 ran a certain set of tasks but in the move to SYSMark 2002, a good deal of the tasks that AMD's Athlon was faster at were removed and replaced with tasks that the Pentium 4 was faster at. Both sets of tasks are perfectly valid tests of CPU performance (it's not like BAPCo just stuck in random tasks that don't do anything) but the point that must be made is that the changes were made seemingly without any user-level research to back them up. If there was some research that said "this is how most people use their systems" that caused BAPCo to change their methodology then this wouldn't have been a problem, but without that backing for their decision then it just seems as if BAPCo optimized the benchmark for the Pentium 4.

3) AMD's secondary complaint is that the benchmarks now use much larger datasets (e.g. Excel). This is more of a minor complaint since it penalizes the Athlon XP for having a smaller cache than Intel's Northwood. AMD would not have made the same complaint had their Hammer already been out since the larger datasets would mean that Hammer's on-die memory controller would give it the advantage.

4) I've been working with AMD on analyzing this information, it's very simple to obtain but requires a bit of effort to analyze. Even AMD today sent me an email saying that they had to order some special software in order to fully understand what's going on in the benchmark. It's too early to make any complete conclusions but what can be said is that SYSMark 2002 can no longer be used as a sole measurement of application performance.

It's pretty sad that it has come to this, but what I can envision happening (at least on AnandTech) is a larger set of office application benchmarks just as Modus and Rand have suggested in this thread. I would like to put together our own tests but it is definitely not an easy task; in light of these discoveries I will have to put much more thought into doing just that however.

The good news is that now that AMD is a part of BAPCo, SYSMark 2003 should become a much better and more balanced benchmark. Before, the only real input from a major CPU vendor was coming from Intel (I was always afraid that SYSMark 2003 would be released and it would show an incredibly unrealistic gain with HyperThreading enabled) but now with AMD involved things will hopefully become more balanced. According to AMD, BAPCo is infinitely more responsive to their needs now that they're a part of the organization and they should be having a formal meeting to discuss this issue very soon (if they haven't already).

I'll keep you posted on what's going on as soon as I get the info I need from AMD/BAPCo.

Take care,
Anand

 

[res1bhmg]

So who's right?

Anand for uncovering an Intel bias in SYSMark

or

Dr. Tom saying that there was no hard evidence of such a bias, and that AMD was pimping this story to hardware sites to get some free PR.

 

[Nemesis77]

Originally posted by: res1bhmg
So who's right?

Anand for uncovering an Intel bias in SYSMark

or

Dr. Tom saying that there was no hard evidence of such a bias, and that AMD was pimping this story to hardware sites to get some free PR.

Having to choose between Dr. Tom and Anand, I choose Anand. Every time.

 

[res1bhmg]

Having to choose between Dr. Tom and Anand, I choose Anand. Every time.

Thanks for the fanboy response, but did you actually read both articles and compare their merits, or does your decision have to do with the fact that you have 4900+ posts on the Anandtech Forums?

 

[Nemesis77]

Originally posted by: res1bhmg

Having to choose between Dr. Tom and Anand, I choose Anand. Every time.

Thanks for the fanboy response, but did you actually read both articles and compare their merits, or does your decision have to do with the fact that you have 4900+ posts on the Anandtech Forums?

I base my decision on the fact that Anand seems like a nice and credible guy, whereas Dr. Tom is a egomaniac who suffers from superiority-complex. And I did read the text posted by SuperCyrix, and it does seem that SYSmark is biased against Athlon.

 

[Desslok]

Herr Tom has no technical merit. He just uses other people knowledge to look smart. As I said before, after he plagerized works that Van Smith wrote under his own name I lost all respect for him.

 

[Spike]

I personally have nothing against Tom, I am not an avid reader of his site, but what I have read seems well put together and mostly objective. I do think that the sysmark benchmarks were slightly biased towards intel, but if you were someone who buys something based on one benchmark (or even any number of benchmarks) then I think you are mistaken. remeber, YMMV so benchmarks are never good ways to make decisions, they just help narrow the field by giving ideas of performance.

Does intel have the fastest processors right now? IMO, the answer is yes, though I do and will continue to run AMD due to the better price/preformance ratio. I think this whole thing has gotten way more attention then it deserves.

-spike

 

[Buz2b]

Thanks for the fanboy response, but did you actually read both articles and compare their merits, or does your decision have to do with the fact that you have 4900+ posts on the Anandtech Forums?

And your reasons for your decision are based on what??
There are many Intel users, just as there are many AMD users on this site. Just because someone chooses to side with one opinion does not necessarily mean it is because he hangs out here long enough to have 4900+ posts. In fact, (s)he may even be an Intel user; there is no mention anywhere in the post or the profile of their "system orientation". (Say, is that a new "politically correct" saying??)
Before you condemn, why not ask some questions. Kinda like the way it's supposed to be, "innocent until proven guilty." :Q

BTW, I read the articles also. I have a strong suspicion that there is a money trail somewhere. Hmmm, let's see, between AMD and Intel, who has both the influence and money advantage?? Of course, that is an opinion, nothing else.

 

[res1bhmg]

And your reasons for your decision are based on what??

What decision?

BTW, Kyle from Hard|OCP has also been critical of sites reporting this bias in SYSMark. So Dr. Tom isn't the only voice in the community.

 

[keyeye]

My reading is that Kyle was critical of Van reporting what AMD sent him without independently verifying it. NOT that AMD was wrong.
This is a valid point, though I must disagree with Kyle on the vigor on which he attacks Van. Van clearly quotes his sources, so I do not find it a piece of terrible or misleading journalism. Heck, everytime I see a product release on ZDnet, it's full of direct from PR material, appropriately cited. IMHO, Kyle and Tom have not added anything to the real questions of (1) bias in Sysmark 2002 (verified by Anand), (2) whether this bias was deliberate or stupid. It's just all senseless bickering.

BTW, there is no need to put in offensive and baseless remarks

"or does your decision have to do with the fact that you have 4900+ posts on the Anandtech Forums?"

Unless you want to be just like Tom....

 

[SuperCyrix]

I first saw this on Tom's and had no idea what's going on because all he said was that it was inconclusive without posting anymore detail.
Next I went to Van's, but obviously there's the strong AMD bias factor there so couldn't really get anything out of that.
Next I saw the AMD pdf file from Van's. It was pretty convincing, but me not being anything close to technical about this still have litte clue as what's going on.

Finally snooping around Anand forum I found that post.
Since there seems to be a little of bitterness towards each other in the other major hardware site that really clouds things up, I really was hoping that Anand would do a full detail article about this.

 

[Deeko]

Thanks for the fanboy response,

duh this is anandtech. who do you think we prefer?