Best article yet on Aberdeen's debunking of AMD's PR rating system...Now ZDNET steps in

[NFS4]

Aberdeen Report AMD Analysis

I just read this article this morning; pretty good read. Some key points I picked up on:


<< Nevertheless, Advanced Micro Devices (AMD) last year deliberately took a step down a slippery slope of bad science when it named its Athlon XP line of microprocessors with clock-speed gigahertz ratings equivalent to Intel?s competing Pentium 4 (P4) based on a set of application benchmarks audited by Arthur Anderson and fully described in AMD vs. Intel?s comparisons at AMD?s Web site.

This is an obvious conflict in logic, given the information presented in the paragraph before. First, Aberdeen agrees with AMD?s position on performance metrics (namely, that clock speed alone is not sufficient), but then immediately slams the company for using gigahertz equivalency ratings that incorporate information other than clock speed. In the following paragraph, Aberdeen again refers to AMD?s equivalency ratings as ?bad science?, and predicts the company will pay a price in terms of market share throughout 2002, as customer?s become increasingly confused. >>


It's almost as though AMD can't win. One one hand, Aberdeen doesn't like GHz ratings, but then slams AMD for their method of representing processor performance.



<< There?s an honestly hilarious answer to why Aberdeen?s report is so logically inept?almost every single problem the company claims exists in AMD?s Model Rating system exists to the same or a greater degree, in the MHz rating system used by Intel. While Aberdeen goes to great lengths to attack GHz as a performance rating system, it never, ever, turns around and applies its own methodology to Intel specifically. Why not? Might have something to do with the fact that Intel paid for the report. >>


This just made me laugh



<< ?The key flaw is that the equivalency rating is a snapshot in a moment in time?making the GHz equivalency subject to increasing variance over time.?

This is technically true, but again, it applies to Intel just as much as AMD. Whenever a website or magazine publishes a set of benchmarks, they are benchmarks run using a specific set of software?and the results published therein are a valid ?snapshot? or performance. Two computers both running at 1.5 GHz might well have vastly different performance ratings based upon their software configurations at the time the ?snapshot? was taken. This is particularly true of the P4. Early in its lifetime, the P4 was, in fact, barely faster than the P3, particularly in its 1.3 and 1.4 GHz versions. Now, thanks to software optimization, P4 performance has increased, but that doesn?t invalidate the truth of the earlier comparison, or make its ?snapshot? any less valid. >>




<< The biggest joke of all is how the report rather forcibly attempts to co-opt support for itself from the technical community. I quote: ?Close reading of the PC trade press and PC performance websites show an increasing skepticism, and it would not be suprising to see the press turn surly.? >>


I agree. At first, the online reviewing community was strongly against the PR rating system. But it seems as though most, if not all of them have embraced it now as they see how the system stacks up.



<< The Aberdeen report honestly isn?t worth the paper it?s printed on. Its internal logic is so inconsistent it ends up sounding almost schizophrenic, it omits any consideration of the problems of benchmarking any machine or the difficulty of proper benchmark development as potential impacts on the development of any processor rating system, completely omits the fact that its criteria apply to other chip manufacturers as well (often in equally-damning terms) and displays a total ignorance of AMD?s own plans regarding their Model Rating system or the strides the system takes towards TPI. >>

 

[dkozloski]

Aberdeen is about as crappy and dishonest as any computer parts seller I have ever ordered from. They have no stock and lie about it, tech support is incomprehensible, and shipping is done by Cheech and Chong. I wouldn't believe them if they told me the sun was sure to come up tomorrow morning.

 

[RSMemphis]

Pentium IV's barely faster than Pentium III's in the beginning?

They got their little shiny a$$es whooped by their "inferior" brethren.
Now, finally after about two years of Pentium IV's, they regained the CPU throne - we'll see for how long.
The problem was that the Pentium III did not scale, so they needed to increase their pipeline.

I don't particularly care what Aberdeen says. It's true that the Pentium IV has ruled the GHz game all along.
To some, that may matter. To me, it does not.

 

[Rob9874]

I think the flaw is trying to create a performance rating that coincides with processor speed in MHz. If AMD feels that MHz/GHz are not an accurate way of measuring CPUs, then why do they devise a rating that sounds like a GHz speed? I agree that there are other factors to consider (bus speed, L2 cache, etc.). I think different companies are going to have to come up with different ways to differentiate their product. We don't only consider horsepower when comparing cars.

 

[AGodspeed]

Lol. I completely agree NFS4. In fact, VH made a couple of points I hadn't even thought of.

Very well done IMHO (just a couple minor issues with some of VH's points, but nothing that takes away from their conclusion).

 

[sandorski]

Rob: How else could they possibly make a system that makes any sense, that doesn't use a mhz type naming system?

 

[Rob9874]

I don't know. Call me Socrates, all questions and no answers. But if they claim that MHz is not an accurate performance measure, then why state, "We're as fast as a 2100MHz CPU"? Does that mean they're as fast as their own 2.1GHz? No, because their 2.1GHz chip would be as fast as a 2.6GHz. So are they as fast as Intel's CPU? If so, which one? Northwood or not? 100MHz bus or 133? Running optimized software or not? It seems very subjective and arbitrary.

And why is their PR rating growing faster than their MHz speed? Shouldn't it be linear? Are their CPUs getting exponentially better? More questions....

 

[Raven76]

There is a way they could make a PR rating that would measure final CPU power and not just one part of the equation (like MHz). However, it would require a certain amount of industry/consumer support.

First a non-biased third party organization such as Aberdeen (j/k!) could develope and maintain a suite of benchmarks. It could be called the "System for CPU Industry Standard Testing " or SCIST. Basically this would consist of the same stuff you see in any online CPU review (Media encoding, Quake3, Content Creation, etc.). Then Intel, AMD, and Cyrix could send their latest CPU's off for testing by this third party organization. There would be a formula to sum the total of all the benchmark scores and the final # would be the PR number for that CPU. Eventually, maybe people would stop paying attention to MHz and just say, "I've got a 2650 SCIST's AMD CPU"..."That's nothing! My CPU is overclocked to 3400 SCIST's."

Of course the systems would have to be identical except for motherboard/CPU. Also the benchmark suite would have to be updated over time, and current CPU's would have to be tested again. The formula used to sum up the benchmark totals could be ajusted to compensate for the change in benchmarking software.

The suite would even be designed to be as platform independent as possible so that Macintoch/Sun/MIPS/whatever systems could be rated as well. That would complicate things much further though.

This is basically what people in the know do all the time anyway. I don't buy a CPU based on a PR# or Ghz. I buy it based on how it performs in reviews compared to other CPUs. This system would just be for the same people that PR ratings are aimed for.

 

[NFS4]

AMD-Intel benchmark war escalates

 

[vash]

This entire Mhz battle and naming system is really for the birds. We know that benchmarks from XXX website will show one better than another and when we look at the sponsorships and suppliers of the parts that are being benched, we know who wins these benches.

The problem with the Mhz battle is the different architecture of the CPUs that are being compared. What we really need is something that we can bench ALL cpus against (x86, x86-64, risc, powerpc, etc). With a benchmark that could encompass EVERY CPU, then we can accurately determine how fast each new processor is versus going off their naming convention.

Will this EVER happen? Maybe. Will AMD and Intel every agree on going to a standard to bench their chips? I doubt it (especially on the Intel side; AMD is probably very willing to compare their chips against a set of "standards" vs Intel).

vash

 

[SexyK]

hey we all know Athlon cores kill the P4 core in IPC right?

Well AMD claims that the PR isn't compared to a P4, but rather how fast a tbird would have to be in order to perform at the same level as the XP.

That's sort of funny because the PRs tend to align with Intel's P4 chips, which would mean that the P4 and the Tbird Athlon have the same IPC :Q

Who would have thought the P4 could outperform the Thunderbird clock for clock?

Whoops, i forgot, the PR system is completely arbitrary and self-serving, but lets all still crap on Intel anyway!

Kramer

 

[dullard]

<< First a non-biased third party organization could develop and maintain a suite of benchmarks. It could be called the "System for CPU Industry Standard Testing " or SCIST. Basically this would consist of the same stuff you see in any online CPU review (Media encoding, Quake3, Content Creation, etc.). Eventually, maybe people would stop paying attention to MHz and just say, "I've got a 2650 SCIST's AMD CPU"..."That's nothing! My CPU is overclocked to 3400 SCIST's." >>




<< What we really need is something that we can bench ALL cpus against (x86, x86-64, risc, powerpc, etc). With a benchmark that could encompass EVERY CPU, then we can accurately determine how fast each new processor is versus going off their naming convention >>


These third party organizations exist using a compilation of commonly used programs. However no one on these forums likes them. You see comments such as "no one uses the programs in that list" or "that is synthetic" or "that benchmark is meaningless" or "use a different compiler and the results change slightly". For example the SPEC bencmarks are hated on these forums.

The first on the SPEC list is for graphics users (programs include 3D Studio Max, Pro/Engineer, Solid Edge, Unigraphics, Maya, etc.) If you wanted to do intense graphics work and wanted the best processor for you, just look at the results and pick it.

SPECint is a commonly used number. In includes: Chess, 3D ray tracing, Gzip compression, CAD, compilation time, combinatorial optimization, Grammer Check in word processors, etc. I think most of us frequently use those types of programs. But no one here cares about SPECint scores.

They really need a SPECgame category for home users. Include all the popluar games from the last 12 months.