Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the use of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as shown on this page
http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13
http://www.hardocp.com/article.html?art=NTI0LDU=
http://www.tomshardware.com/cpu/20030923/athlon_64-22.html#opengl_benchmarks
Let me say the article is great, no complaints there. I know it takes alot of work to produce these articles.
Now, I see two reasons for using a game as a cpu benchmark:
1) It presents a fair (emphasis on the word 'fair') comparison of the competing cpu architectures and scaling issues.
2) The game itself is of current interest to the community.
In your article you already concede 2). Quake3 itself is not relevant as a game to anybody. Quake3-derived games are another matter, and are still popular and certainly relevant. More on these later.
I believe there is strong evidence that Quake3 does not provide a fair benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well) AMD cpus vs Intel cpus. The reason being (and let me emphasize that I don't know this as an verified fact, I'm going on what a couple of programmers involved with helping AMD produce optimized game code have told me) that the Quake3 cpu recognition code does not recognize the AthlonXP as an SSE-capable cpu. Not only that, but the 3DNow code in Quake3 is apparently non-functional for this cpu.
The politics and history behind this are interesting, but probably boil down to the AthlonXP being released well after Quake3, and Carmack being rightly uninterested in patching an old game.
If this is true, you are benchmarking two equally SSE-capable cpus against each other, using a game engine which enables SSE for the Intel cpu and *disables* SSE for the AMD cpu (apparently there's no simple way to force SSE recognition either), for no valid reason, other than the game is too old to know about the AMD cpu's capabilities. What would be even worse is if this same recognition problem carries over to the Athlon64 (I have no word on this) and to newer Quake3-based games.
Again, assuming this is true, it removes any rationale for using a 3-year old game that: a) few people play, b) which gives ridiculously high scores, and which c) unfairly handicaps AMD cpus; as a benchmark to be used specifically in comparing AMD cpus vs their Intel competitors in articles such as this one.
So. Here are the recommendations I, as an interested Hardocp/Anand/Toms reader (and admitted AMD fan) am making to you and your site:
1) Investigate this matter further, and write an article discussing it. And in particular discuss the relevance of this cpu issue to current Quake3-based games. Assuming there is in fact an Intel bias to Quake3-based benchmarking I think people would be very interested to learn about it. Apparently the SSE issue does indeed carry over to later games.
2) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark, and especially discontinue it's use when comparing AMD vs Intel cpus. The game will never be patched to fix this issue, and using 3rd party fixes noone cares about is more or less pointless too. I'm referring to the dlls on this page:
http://speedycpu.dyndns.org/opt/
This guy is one of the programmers I referred to earlier, and he tells me the dlls do not enable SSE where it really matters anyway. The other was a student working at AMD writing assembly 3DNow code. The best solution is simply to retire this benchmark, just as Q1 and Q2 were retired.
rms
http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13
http://www.hardocp.com/article.html?art=NTI0LDU=
http://www.tomshardware.com/cpu/20030923/athlon_64-22.html#opengl_benchmarks
Let me say the article is great, no complaints there. I know it takes alot of work to produce these articles.
Now, I see two reasons for using a game as a cpu benchmark:
1) It presents a fair (emphasis on the word 'fair') comparison of the competing cpu architectures and scaling issues.
2) The game itself is of current interest to the community.
In your article you already concede 2). Quake3 itself is not relevant as a game to anybody. Quake3-derived games are another matter, and are still popular and certainly relevant. More on these later.
I believe there is strong evidence that Quake3 does not provide a fair benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well) AMD cpus vs Intel cpus. The reason being (and let me emphasize that I don't know this as an verified fact, I'm going on what a couple of programmers involved with helping AMD produce optimized game code have told me) that the Quake3 cpu recognition code does not recognize the AthlonXP as an SSE-capable cpu. Not only that, but the 3DNow code in Quake3 is apparently non-functional for this cpu.
The politics and history behind this are interesting, but probably boil down to the AthlonXP being released well after Quake3, and Carmack being rightly uninterested in patching an old game.
If this is true, you are benchmarking two equally SSE-capable cpus against each other, using a game engine which enables SSE for the Intel cpu and *disables* SSE for the AMD cpu (apparently there's no simple way to force SSE recognition either), for no valid reason, other than the game is too old to know about the AMD cpu's capabilities. What would be even worse is if this same recognition problem carries over to the Athlon64 (I have no word on this) and to newer Quake3-based games.
Again, assuming this is true, it removes any rationale for using a 3-year old game that: a) few people play, b) which gives ridiculously high scores, and which c) unfairly handicaps AMD cpus; as a benchmark to be used specifically in comparing AMD cpus vs their Intel competitors in articles such as this one.
So. Here are the recommendations I, as an interested Hardocp/Anand/Toms reader (and admitted AMD fan) am making to you and your site:
1) Investigate this matter further, and write an article discussing it. And in particular discuss the relevance of this cpu issue to current Quake3-based games. Assuming there is in fact an Intel bias to Quake3-based benchmarking I think people would be very interested to learn about it. Apparently the SSE issue does indeed carry over to later games.
2) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark, and especially discontinue it's use when comparing AMD vs Intel cpus. The game will never be patched to fix this issue, and using 3rd party fixes noone cares about is more or less pointless too. I'm referring to the dlls on this page:
http://speedycpu.dyndns.org/opt/
This guy is one of the programmers I referred to earlier, and he tells me the dlls do not enable SSE where it really matters anyway. The other was a student working at AMD writing assembly 3DNow code. The best solution is simply to retire this benchmark, just as Q1 and Q2 were retired.
rms
Facebook
Twitter