Did Apple cheat (with the G5 benches)? Hannibal from Ars weighs in.

[Eug]

Hannibal Stokes from Ars weighs in:

I think the whole thing has been blown way out of proportion. When I was first watching the keynote live, I thought that Apple had stacked the deck, and I still entertain the possibility that they did so deliberately. But who cares? It's a keynote, and it's their biggest hardware introduction since the original G4, so they're going to make it look really good at all costs--there's just too much at stake for them. I mean, I hate to sound like an Apple apologist, but any deck-stacking that was done was extremely tame and forgivable in the notoriously underhanded world of competitive benchmarking

-snip-

The bottom line is this: no, Apple and IBM didn't "cheat" in any sense of the word that I would acknowledge, but they did set up a fair contest that they knew they could win. The hardware will be out soon enough, though, and then we can all test it ourselves under whatever conditions we like.

There's much more in the article, particularly about the use of GCC. Check it out.

 

[Rand]

I wouldnt call it cheating per-say, merely setting up an intentionally favourable position, quite similar is some ways to the fact that AMD, and Intel's press kits come with a different recommended LightWave benchmark scene.
Naturally they'll choose the scene that shows their processor in the best light, and different scenes can have wildly different display characteristics.

 

[andreasl]

Please show me any other company that either conducts or hires someone else to conduct tests of competing MPUs in SPEC CPU2000 instead of taking those numbers from the official SPEC site.

I recently saw a HP document that compared Madison performance in SPEC CPU2000 and other benchmarks to other processor architectures (even including Opteron). Do you think HP compiled the benchmark for Opteron, Power4, US3 etc themselves? NO, they pulled the numbers from the official site.

The only 'fair' way of comparing SPEC CPU2000 performance across platforms is to use the best compiler for each platform. If GCC 3.3 is the best compiler for the Mac then I really pity that platform... At least IBM has their own compiler they will use if they ever submit scores for the PPC970. And they surely won't try to artificially cripple the competing platforms (other than perhaps comparing with SGI instead of HP for example but using official numbers).

 

[Eug]

Please show me any other company that either conducts or hires someone else to conduct tests of competing MPUs in SPEC CPU2000 instead of taking those numbers from the official SPEC site

I could also ask you to show any company that compares their performance with other machines and then publishes the methods for the test using a 3rd party agency.

Apple's tests did favour themselves (of course), but it's about as open as you could have have gotten.

 

[andreasl]

Originally posted by: Eug

Please show me any other company that either conducts or hires someone else to conduct tests of competing MPUs in SPEC CPU2000 instead of taking those numbers from the official SPEC site

I could also ask you to show any company that compares their performance with other machines and then publishes the methods for the test using a 3rd party agency.

Apple's tests did favour themselves (of course), but it's about as open as you could have have gotten.

I never disputed that. And exactly what are you trying to say? Usually a company will cite a SPEC CPU2000 score before it appears on the SPEC site. The scores are put up for peer review (by their competitors on the committe!) before they are published, thus the delay.

Never have I seen a company running SPEC CPU2000 on a competing platform before. Can you imagine the uproar if Intel did this with an AMD system? This forum would explode in a ball of fire.

 

[Eug]

Can you imagine the uproar if Intel did this with an AMD system? This forum would explode in a ball of fire.

As did happen here. Jobs is sitting back enjoying all the free press.

 

[Cadaver]

Regardless of the minutia of the benchmarks, the G5 is clearly a fast machine. Unlike the G4s, Apple now has a CPU that can play in the same league as a high-MHz P4 or Athlon. That Mathematica demo has me drooling.

The G5 is a huge step in terms of system architecture for Apple. Clearly brings them into the state of the art, as opposed to their warmed over 133MHz buses they've been using since 1999 (or whatever).

If I had the money for (yet another) new machine, I'd probably buy one. For now though, I'm trying to consolidate everything on to one machine where possible. I'm down to an nice PC (see "rigs") for daily use and a 1GHz PowerBook G4 for on-the-road use (only recently have there been PC laptops that come close to the size/weight/feature list of the PowerBooks).

 

[sharkeeper]

So I guess we're gonna start seeing commercials with snails crawling with P4's strapped to their backs?

I remember the 1998 ads for the G3 vs. P2...

With all that fancy processing power you would think they could get creative and use a turtle complete with a ZIF socket embedded in his shell, P4 processor and running cooler--perhaps a thermaltake smart fan with blinking LED! While the turtle is crawling. That would be worth seeing!

-DAK-

 

[LeeTJ]

Originally posted by: Cadaver
Regardless of the minutia of the benchmarks, the G5 is clearly a fast machine. Unlike the G4s, Apple now has a CPU that can play in the same league as a high-MHz P4 or Athlon. That Mathematica demo has me drooling.

The G5 is a huge step in terms of system architecture for Apple. Clearly brings them into the state of the art, as opposed to their warmed over 133MHz buses they've been using since 1999 (or whatever).

If I had the money for (yet another) new machine, I'd probably buy one. For now though, I'm trying to consolidate everything on to one machine where possible. I'm down to an nice PC (see "rigs") for daily use and a 1GHz PowerBook G4 for on-the-road use (only recently have there been PC laptops that come close to the size/weight/feature list of the PowerBooks).

is this indicative of how far the G series has come? or more indicative of how little progress there has been with the x86 cpus lately.

 

[RaynorWolfcastle]

Originally posted by: LeeTJ

Originally posted by: Cadaver
Regardless of the minutia of the benchmarks, the G5 is clearly a fast machine. Unlike the G4s, Apple now has a CPU that can play in the same league as a high-MHz P4 or Athlon. That Mathematica demo has me drooling.

The G5 is a huge step in terms of system architecture for Apple. Clearly brings them into the state of the art, as opposed to their warmed over 133MHz buses they've been using since 1999 (or whatever).

If I had the money for (yet another) new machine, I'd probably buy one. For now though, I'm trying to consolidate everything on to one machine where possible. I'm down to an nice PC (see "rigs") for daily use and a 1GHz PowerBook G4 for on-the-road use (only recently have there been PC laptops that come close to the size/weight/feature list of the PowerBooks).

is this indicative of how far the G series has come? or more indicative of how little progress there has been with the x86 cpus lately.

It is indicative of how far the G series has come, the G4 had progressed very little since its introduction. The G5 finally catches up to x86 CPUs in terms of techology. The G5 looks to be a great CPU but IMHO, the whole idea of a 1 GHz FSB is pure marketing though. Judging by the P4C overclocking results it is *extremely* rare to find an i865/i875 motherboard that can't do 1 GHz FSB. That is Intel very likely could have released a 1 GHz FSB P4 but since there is no 500 MHz DDR RAM what would be the point? (rhetorical question)

The G5 is an impressive CPU but when it starts shipping in August the Athlon64 and Prescott will be just around the corner. Seeing how the G5 looks to be competitive with the current crop of x86 CPUs it'll be interesting to see how it performs against next-gen x86 CPUs. Time will tell...