Truth or Myth?: Is SYSmark a Reliable Benchmark?

[Abwx]

From what I get they are the only top-20 semicom enterprise that is being sued for misleading information provided to their own investors.

lol..

http://www.vanshardware.com/reviews/2002/08/020822_AthlonXP2600/SYSmark%202002%20Analysis%20Presentation%20FINAL.pdf

Intel has agreed to settle a class action lawsuit that claims the company “manipulated” benchmark scores in the early 2000s to make its new Pentium 4 chip seem faster than AMD’s Athlon.

And also seems that Povray was a strategic bench, for infos :

. (Note: POV-Ray 3.5 was not compiled with an Intel compiler. POV-Ray 3.6.0 was.).

And of course some sysmarking :

http://www.extremetech.com/computin...ners-over-amd-athlon-benchmarking-shenanigans

 

[TheELF]

lol..

http://www.vanshardware.com/reviews/2002/08/020822_AthlonXP2600/SYSmark%202002%20Analysis%20Presentation%20FINAL.pdf




And also seems that Povray was a strategic bench, for infos :

Spoiler

And of course some sysmarking :

Spoiler

http://www.extremetech.com/computin...ners-over-amd-athlon-benchmarking-shenanigans

From the link

The funny thing is, I did do that — but the programmer friend who helped me with Intel’s compiler could never reproduce the results in POV-Ray 3.6.0, despite compiling six different executables with different optimization levels in an attempt to do so.

Fast forward almost a decade. A few months ago, I decided to play with a Perl script that can strip the “Cripple AMD” functions out of executables compiled by Intel compilers. I tested it on the copy of POV-Ray 3.6.0 I’ve kept on hand ever since. Please note that I tested using modern hardware and under Windows 7, not a 2004-era system.

Couldn't reproduce the lower performance back in the day but could now (well in 2014) can't possibly be an artifact of this
http://www.anandtech.com/show/5448/the-bulldozer-scheduling-patch-tested ,especially on a "modern" module cpu that throttles like crazy, must be evil intel corps fault.

An oh man them huge differences in sysmark...

Also from the same article

The principle reason no one makes a big deal about these gaps anymore is because the difference between Intel and AMD has simply grown too wide. An 8-12% systemic improvement for Intel may make AMD look worse than it otherwise would, but AMD’s performance in Sysmark 2012 can lag Intel by as much as 50% — and that’s not something that compiler patches can fix.

 

[The Stilt]

The (extremely) old versions of Intel compilers are completely another story. They infact disable instructions based on the CPUID VendorID. Euler3D Stars (from CaseLabs) is a good example of this. Once you kill the dispatcher, AMD CPUs will see > 30% boost in performance.

Nowdays such thing isn´t possible with Intel compiler (in case an evil developer would like to use it). If you use Intel specific optimizations (Qx options), you´ll face a screen which says that you are missing required instructions in case you are trying to run it on AMD CPU. Regardless which Qx option the code was compiled and if the instructions are actually missing or not. Regardless, the dispatcher is extremely easy to remove afterwards and it´s effects can be fully tested under VMWare if the modifications to the code are not possible.

I think AMD is pretty comfortable with Intel compilers, since they compile their display drivers and ACML library with Intel compilers. Surely they wouldn´t do that if it produced code which is crippled on their CPUs :sneaky:

 

[Abwx]

From the link

Couldn't reproduce the lower performance back in the day but could now (well in 2014) can't possibly be an artifact of this
http://www.anandtech.com/show/5448/the-bulldozer-scheduling-patch-tested ,especially on a "modern" module cpu that throttles like crazy, must be evil intel corps fault.

An oh man them huge differences in sysmark...

Also from the same article

Well, the article doesnt say that there s no built in advantage for Intel but still, there s some benches that say it all, we know that Kabini has 20% better IPC than BayTrails in Integer and roughly 30% for FP, so let s see the scores according to Sysmark :

http://www.anandtech.com/show/8067/...athlon-53505150-and-sempron-38502650-tested/3

You can check the other benches at said review, as for Sysmark being reliable i would say that yes, it s very reliable for whom need some marketing tool to promote Intel CPUs, otherwise it s useless for whom want actualy reliable numbers.

 

[AtenRa]

Like Cinebench, SYSmark should only be compared between the same company products. That is Intel vs Intel and AMD vs AMD.

 

[ShintaiDK]

Like Cinebench, SYSmark should only be compared between the same company products. That is Intel vs Intel and AMD vs AMD.

Whats next, games should only be compared Intel vs Intel and AMD vs AMD?

 

[Abwx]

Like Cinebench, SYSmark should only be compared between the same company products. That is Intel vs Intel and AMD vs AMD.

That s the case for Cinebench since AMD use it to compare their own CPUs parts, but i doubt that Sysmark could have the same value for such comparisons.

Not sure that it s relevant even for Intel/Intel comparisons since it doesnt use GPU acceleration when available for a sub test, hence new GPUs with better Open CL perfs are artificialy removed from the equation.

 

[mrmt]

Whats next, games should only be compared Intel vs Intel and AMD vs AMD?

I think we should go for the next step and develop an AMD-doctored benchmark. That way resellers from all over the world will be able to easily fool their customers into buying slower AMD processors and AMD marketing department will have a benchmark win to show.

 

[TheELF]

I think we should go for the next step and develop an AMD-doctored benchmark. That way resellers from all over the world will be able to easily fool their customers into buying slower AMD processors and AMD marketing department will have a benchmark win to show.

Already exists, in every in-build game benchmark ever,just ask atenra.
Other then that, AMD's turbo mode takes care of any bench that runs separate single and multicore tests.
Oh yeah and don't forget all the 2012 benchmarks that don't even know what a modern core is,like fritz- ,x264- and 7zip-bench.

 

[Sweepr]

Already exists, in every in-build game benchmark ever,just ask atenra.
Other then that, AMD's turbo mode takes care of any bench that runs separate single and multicore tests.
Oh yeah and don't forget all the 2012 benchmarks that don't even know what a modern core is,like fritz- ,x264- and 7zip-bench.

Can't believe there's still websites (Hardware.fr) that use this crap to measure CPU efficiency.

 

[mysticjbyrd]

Yes, now ask AMD whether they would release the source code or at least allow audit on the source code of their benchmarks scripts.

It's funny how such shady company like AMD thinks it has the moral authority to question >>anyone<< else on the market. From what I get they are the only top-20 semicom enterprise that is being sued for misleading information provided to their own investors.

This is not a valid argument. Even if, you could prove AMD was satan himself, it doesn't absolve intel of potential wrongdoing. Their hand was already caught in the cookie jar before.

Perf/$ and perf/watt is not the same,also we are talking about those in relation to what the target group wants.
With AMD you get really bad perf/watt in everything that is considered everyday workload,you get very good perf/$ if you only do rendering or something else purely multicored.

For most people browsing and gaming is all they do and that's why AMD has such a low market share.

I wasn't even the one making that claim. Perf/$ and Perf/watt are basically the same thing if you can accurately calculate the added energy cost of less efficient cpus, and add that into the initial cost.

For the average consumer, the efficiency is probably close to irrelevant.

I was talking about AMDs past offerings, and not so much today.

Sure release the source code for a benchmark that is supposed to give numbers to government organizations and then start hunting down every modified bench each tom dick and harry releases that decides to skew it in any direction.

There is plenty of open source software floating around, and it doesn't devolve into software anarchy! There is a ruling body that can manage it as it changes.

 

[Burpo]

.. "doesn't absolve intel of potential wrongdoing"?

That's funny..
Maybe I shouldn't be absolved of potentially stealing a car.. lol

 

[Lorne]

has anyone changed the vendor ID on an Intel and run SysMark to see if it makes any difference?

 

[BarkingGhostar]

Considering some buyers, (myself included) only occasionally use benchmarks to compare products within the same manufacturer, the entire story behind Intel optimizing a test to favor their products over the competition is a moot point.

I have yet to hear in person a single buyer of a CPU stating their buying decision was based on the results of comparing AMD to Intel using XYZ benchmarking. Now, if I worked a retail floor, acted as a department purchaser, etc., maybe, but as a smuck on Anandtech ... nadda.

I buy Intel because I feel comfortable buying Intel. I don't go bragging about it, or whining about the other guys. How many of you buy processors for personal use using your own money with such frequency that you need to eek out so much performance for the wallet that you will frog back and forth between manufacturers?

 

[Ma_Deuce]

Whats next, games should only be compared Intel vs Intel and AMD vs AMD?

The CPU with a better benchmark of CoD is probably going to give you a better experience with actual use of the product, ie playing CoD.

The CPU with the better benchmark of sysmark might only give you a better experience running sysmark but not actual use of the product (unless you just run benchmarks...)

Isn't that what benchmarking is supposed to be about, end use?

 

[ShintaiDK]

I think AMD is pretty comfortable with Intel compilers, since they compile their display drivers and ACML library with Intel compilers. Surely they wouldn´t do that if it produced code which is crippled on their CPUs :sneaky:

But, but, but...RTG favours Intel CPUs!

 

[mrmt]

This is not a valid argument. Even if, you could prove AMD was satan himself, it doesn't absolve intel of potential wrongdoing. Their hand was already caught in the cookie jar before.

I'm not absolving Intel of anything, but look at AMD is doing. It is not pointing out a potential wrongdoing, it is accusing Intel of wrongdoing. They didn't provide the evidence of wrongdoing, and they certainly do not have enough credibility to be taken seriously.

 

[The Stilt]

But, but, but...RTG favours Intel CPUs!

This makes no sense to me, since the real world performance difference is just 7% and not (>)50% (as SYSMark indicates) :hmm:

 

[ShintaiDK]

This makes no sense to me, since the real world performance difference is just 7% and not (>)50% (as SYSMark indicates) :hmm:

So you saying Radeon Technology Group is undermining the AMD CPU division to boost Intel sales. Interesting! :ninja:

 

[TheELF]

This makes no sense to me, since the real world performance difference is just 7% and not (>)50% (as SYSMark indicates) :hmm:

Look at the video, one more time, intel scores 987 and AMD scores 659, that's not the exaggerated 50% that the AMD "experts" claim and it's certainly not over 50% it's the 35-40% that we know that intel is faster in single threaded workloads,with comparable CPUs.
And kid yourself not,do some work on office apps and tell us if your CPU ever hits 100% or if only one core hits 100% ,it will only run multicored on certain database operations.

 

[MrTeal]

Look at the video, one more time, intel scores 987 and AMD scores 659, that's not the exaggerated 50% that the AMD "experts" claim and it's certainly not over 50% it's the 35-40% that we know that intel is faster in single threaded workloads,with comparable CPUs.
And kid yourself not,do some work on office apps and tell us if your CPU ever hits 100% or if only one core hits 100% ,it will only run multicored on certain database operations.

... Did you actually try to do the math? Sure, 49.77% isn't 50%, but it's a bit much to call that an exaggeration.

 

[The Stilt]

Since you missed the intended sarcasm: AMD bashes SYSMark for producing "50%" better scores for "equivalent" Intel systems, which according to AMD do not represent the real world performance differences at all.

Meanwhile RTG is using Intel CPUs for their demo systems, because their own CPUs would bottle neck their high-end GPUs. Interestingly Haswell / Broadwell / Skylake happen to be > 50% faster (IPC) in FP workloads on average, while the difference can be as high as > 90% (FP workloads w/ modern instructions).

Please check the calibrations of your sarcasmdar :biggrin:

 

[TheELF]

... Did you actually try to do the math? Sure, 49.77% isn't 50%, but it's a bit much to call that an exaggeration.

How do you come up with 49.77% ?
Intel scores 987 and AMD scores 659, 50% of 987 is 493 and not 659.

 

[MrTeal]

How do you come up with 49.77% ?
Intel scores 987 and AMD scores 659, 50% of 987 is 493 and not 659.

If AMD is 50% of Intel, then Intel is 200% faster than AMD. If Intel is 50% faster than AMD, than Intel's score is 1.5 larger than AMD. AMD in that case is 67% of the speed of Intel.

 

[TheELF]

If AMD is 50% of Intel, then Intel is 200% faster than AMD. If Intel is 50% faster than AMD, than Intel's score is 1.5 larger than AMD. AMD in that case is 67% of the speed of Intel.

You are making it more complicated than it is but yes.
Baseline of sysmark is 1000,in the video the intel is 98.7% of that and the amd is 65.9% of that,that's a ~33% difference and not the claimed 50% .
It's in line of the single speed performance advantage that intel has.