Truth or Myth?: Is SYSmark a Reliable Benchmark?

[Azuma Hazuki]

So tired of the "free-market" worshipers and their idol, the Invisible Hand. It's a combination of false simplicity and unearned, complacent smugness that makes me want to take a baseball bat to them. I can't tell if it's lack of imagination, inability to think critically, or actual sociopathy, but it's making me nuts to watch.

Economics has got to be the only "science" which is allowed to ignore the laws of thermodynamics, and the only "social studies" program which is allowed to assume rational actors. They always reminded me of the physics teacher who retires to a farm, and begins with "Imagine a perfectly spherical cow, in vacuum..."


Threadcrapping is not allowed here. This is a technical forum.
Markfw900

 

[Arachnotronic]

So tired of the "free-market" worshipers and their idol, the Invisible Hand. It's a combination of false simplicity and unearned, complacent smugness that makes me want to take a baseball bat to them. I can't tell if it's lack of imagination, inability to think critically, or actual sociopathy, but it's making me nuts to watch.

Lol, what's driving this angst?

 

[Burpo]

Femme fatale?

Ok, back on topic..

What is BAPCo?
BAPCo stands for Business Applications Performance Corporation and its current members include: Acer, ARCIntuition, ChinaByte, CNET, Compal, Dell, Hewlett-Packard, Hitachi, Intel, LC Future Center, Lenovo, Microsoft, Western Digital, Wistron, Samsung, Sony, Toshiba, Zol and others. BAPCo is a non-profit consortium with a charter to develop and distribute a set of objective performance benchmarks for personal computers based on popular software applications and operating systems. Applications used in SYSmark 2012 were selected based on market research and include Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others.

AMD dropped out, and does not wish to participate.. Doesn't make the benchmark less valid.

http://www.anandtech.com/show/4464/amd-resigns-from-bapco-over-sysmark12-concerns

"Advanced Micro Devices (AMD) was, until recently, a long standing member of BAPCo. We welcomed AMD’s full participation in the two year development cycle of SYSmark 2012, AMD’s leadership role in creating the development process that BAPCo uses today and in providing expert resources for developing the workload contents. Each member in BAPCo gets one vote on any proposals made by member companies. AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.

BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.

BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications."

 

[coercitiv]

That's exactly the same to what I said.

No it's not. You can argue it used little OpenCL accelerated functions, sure, but your concluson is way off.

CPU grunt will always be more important for the overall performance.

In the case of Adobe Photoshop, GPU performance of a certain level is the most important factor. Again, modern Intel iGPUs handle that just fine, offering quite a nice experience, but just because they do doesn't mean the CPU side is more important to begin with. After you pass a certain performance threshold for both CPU&GPU, one will pull ahead with the requirements (CPU quite often), but depending on your most common workload increasing CPU clocks/cores may or may not increase performance. This is not something I made up reading benchmark results on the web, I used PS for a living for more than 10 years now.

If you want to experience Photoshop in true "CPU mode", disable GPU acceleration in the Performance menu. Even on a modern i7 quad core the experience will be atrocious, display quality will degrade in order to keep in UI responsive, but even scaling or cropping will be embarrassingly slow.

Again, if SYSmark runs it's media creation tests with GPU acceleration enabled, then it's all fine because it gives relevant results. But if it does not, the benchmark fails it's purpose (give customers a measure of how a certain system will perform with a set of productivity tools)

Here's a few scores for systems equiped with i7 4790K. Although it doesn't really prove anything, it should raise some question marks. You can check for more results here.

 

[TheELF]

No it's not. You can argue it used little OpenCL accelerated functions, sure, but your concluson is way off.

In the case of Adobe Photoshop, GPU performance of a certain level is the most important factor. Again, modern Intel iGPUs handle that just fine, offering quite a nice experience, but just because they do doesn't mean the CPU side is more important to begin with.

You do realise that this whole thread is only about GPGPU importance,right?
I already mentioned that you will need a GPU for display,the argument from AMD is that the benches should only (or mainly) test GPGPU stuff...
Since you use PS for a living for more than 10 years now,can you take a guess on a percentage of how much of your daily workload is GPGPU related?
Is it even more than the 5% benefit that sysmark claims with a good GPU?

 

[coercitiv]

You do realise that this whole thread is only about GPGPU importance,right?
I already mentioned that you will need a GPU for display,the argument from AMD is that the benches should only (or mainly) test GPGPU stuff...
Since you use PS for a living for more than 10 years now,can you take a guess on a percentage of how much of your daily workload is GPGPU related?
Is it even more than the 5% benefit that sysmark claims with a good GPU?

GPGPU importance? Sure, let's see benches.

Take a look at this Adobe Photoshop CS6 benchmark.

From our testing, we can now confidently state that even though the NVIDIA 600-series and Intel HD 4000 graphics are not on Adobe's compatibility list, GPU acceleration in Photoshop CS6 works great on those cards. So, contrary to the compatibility list, there is no reason to use an older generation NVIDIA GTX 580 since the current generation NVIDIA cards performs as well or better.

From a budget standpoint, the NVIDIA Geforce GTX 650 1GB and AMD Radeon HD 7750 both did great for their price points, performing just a few seconds slower than the fastest cards we tested. If you have a bit more to spend, but cannot afford a NVIDIA GTX 680 2GB (which was the top performer), we recommend the NVIDIA Geforce GTX 660 2GB as it's performance was almost identical to the NVIDIA Geforce GTX 660 Ti and GTX 670, yet is much cheaper.

Or this Premiere bench.

Ready for a shock?

 

[ShintaiDK]

You could argue that for their current cpus, but not in the past. It's also not true of their gpu market.

Because you decided so? AMDs GPU market share collapsed when its like the CPU part couldn't compete in any way in performance/watt.

 

[TheELF]

GPGPU importance? Sure, let's see benches.

Yes look at them and especially look at the bottom for the individual tests,3 versions of blur and two other filters,that's the lot of what you can accelerate,sure if you only use these 5 things in PS than you will get an awesome boost.

 

[TheELF]

Ready for a shock?

Are you? They only test exporting to mpeg2 and x264, sure it's a big part of working with premiere but it also is the big part that is always running in the background/overnight or whenever,makes a big difference if you have a lot of transcoding to do but doesn't really affect how you perceive the speed of your system while you are editing.

 

[coercitiv]

Are you? They only test exporting to mpeg2 and x264, sure it's a big part of working with premiere but it also is the big part that is always running in the background/overnight or whenever,makes a big difference if you have a lot of transcoding to do but doesn't really affect how you perceive the speed of your system while you are editing.

Wait, so when I was talking about the impact of GPU on the perceived speed of the system while editing in Photoshop (antialiasing, zoom, pan, crop, rotate, brush control, color conversion) you were more concerned with how GPGPU helps with the end result of a project, yet now in Premiere we shift the emphasis from rendering back to editing?

For these media creation tools both CPU & GPU performance matter, with certain workloads favoring one or the other, and seeing SYSmark Media Creation scores for the same CPU putting the equal sign between Intel HD 4600 and GeForce 650 or 750Ti is a little disconcerting.

 

[TheELF]

Wait, so when I was talking about the impact of GPU on the perceived speed of the system while editing in Photoshop (antialiasing, zoom, pan, crop, rotate, brush control, color conversion) you were more concerned with how GPGPU helps with the end result of a project, yet now in Premiere we shift the emphasis from rendering back to editing?

For these media creation tools both CPU & GPU performance matter, with certain workloads favoring one or the other, and seeing SYSmark Media Creation scores for the same CPU putting the equal sign between Intel HD 4600 and GeForce 650 or 750Ti is a little disconcerting.

As you yourself said even the most basic intel IGPU will accelerate this stuff enough to not make any difference.
And now that I think about it,have you ever checked out quick sync? Even on a celleron?
https://www.youtube.com/watch?v=zsTI-625Na0

Sysmark is supposed to show how a system will feel to the user and not how much gain you could get from gpgpu in certain specialized scenarios.
That's the bottom line and all the talking around in circles won't change that.
So is SYSmark a Reliable Benchmark? Even though it doesn't use any(or very little) GPGPU?
Yes it is!Because users as well don't use any(or very little) GPGPU.

 

[coercitiv]

As you yourself said even the most basic intel IGPU will accelerate this stuff enough to not make any difference.

You seem to purposely ignore that Photoshop can work in two modes: software or GPU accelerated.

When GPU acceleration is turned off, not even the HD 4600 helps anymore. Hence my question: is SYSmark using PS or Premiere in software mode for benching?

And now that I think about it,have you ever checked out quick sync? Even on a celleron?
https://www.youtube.com/watch?v=zsTI-625Na0

Quick sync for professional video output?

Sysmark is supposed to show how a system will feel to the user and not how much gain you could get from gpgpu in certain specialized scenarios.
That's the bottom line and all the talking around in circles won't change that.

Ok, let's cut to the chase.

Would you recommend Adobe Premiere users to buy the fastest Intel CPU to date and just use the Intel iGPU for their work? (maximum budget for CPU, where all the real gains are)

 

[TheELF]

You seem to purposely ignore that Photoshop can work in two modes: software or GPU accelerated.

When GPU acceleration is turned off, not even the HD 4600 helps anymore. Hence my question: is SYSmark using PS or Premiere in software mode for benching?

This thread and AMD's video is about GPGPU and how much it can accelerate your daily work,not about how bad your working experience would be without any graphics output at all.

Quick sync for professional video output?

Anything other then software x264 for professional video output?
See?Either you have acceleration with reduced quality or you don't have (much) acceleration.

Ok, let's cut to the chase.

Would you recommend Adobe Premiere users to buy the fastest Intel CPU to date and just use the Intel iGPU for their work? (maximum budget for CPU, where all the real gains are)

Would you recommend them to get a really expensive GPU if it would mean they would have to get a much slower CPU?

 

[coercitiv]

This thread and AMD's video is about GPGPU and how much it can accelerate your daily work,not about how bad your working experience would be without any graphics output at all.

No, this thread is about SYSmark being an accurate benchmark for it's declared use case. You keep taking the GPU acceleration for granted, being on iGPU, but the same software can work in CPU mode only, with dramatically skewed results performance wise.

AMD can cry all they want, if SYSmark uses GPU acceleration in the software bundles that support it, then this discussion is futile. But if it does not, then the benchmark fails to reflect true user experience.

Anyway, I've dragged this discussion astray for long enough, let's just agree to disagree on the matter and move on.

 

[caswow]

this thread just shows the hatred against amd of some people. even sysmark is now a valid benchmark because amd doesnt like it D:

 

[TheELF]

No, this thread is about SYSmark being an accurate benchmark for it's declared use case. You keep taking the GPU acceleration for granted, being on iGPU, but the same software can work in CPU mode only, with dramatically skewed results performance wise.

Now you make sense but looking at the second part of AMDs video they make it pretty clear that they are talking about GPGPU,or lack thereof, they even show a screen of pcmark showing that they test with GPGPU (openCL1.1) ,so I thought that was a given,they want to convince people that their graphics cores should be considered normal computing cores,usable for anything,when they are nothing of the kind.

 

[mrmt]

[/B]Would you recommend Adobe Premiere users to buy the fastest Intel CPU to date and just use the Intel iGPU for their work? (maximum budget for CPU, where all the real gains are)

Sysmark is a general benchmark, if you will only use Premiere and other content creation tools, it's better to benchmark these tools and disregard the others included in Sysmark. Sysmark is not for you.

 

[mrmt]

Now you make sense but looking at the second part of AMDs video they make it pretty clear that they are talking about GPGPU,or lack thereof, they even show a screen of pcmark showing that they test with GPGPU (openCL1.1) ,so I thought that was a given,they want to convince people that their graphics cores should be considered normal computing cores,usable for anything,when they are nothing of the kind.

Four years after "the future is fusion" AMD did not provide tools for people to develop applications for their wonderchips.

 

[AtenRa]

Four years after "the future is fusion" AMD did not provide tools for people to develop applications for their wonderchips.

It seems you are interested in OpenCL, the following link will be very helpful.

http://developer.amd.com/tools-and-sdks/opencl-zone/opencl-resources/

And here for Tools & SDKs for more than OpenCL.

http://developer.amd.com/tools-and-sdks/

 

[DrMrLordX]

My first thought when seeing this story was that AMD was preparing the public for poor SYSmark scores on Summit Ridge.

Now, bear in mind that they have thus far only chosen to complain about SYSmark. Many benchmarks have shown AMD CPUs in a poor light over the years (particularly the last 5 years). SYSmark is probably the least-forgiving of all widely-circulated benchmarks when running on AMD CPUs.

The biggest problem I see with SYSmark is that it's a batch of scripts/macros performing some tasks which are not necessarily provided in detail to the public, using shrink-wrapped versions of the listed applications. When review sites (such as Anandtech) are quite capable of formulating their own individual benchmarks on a per-application basis AND when said sites are willing to provide the readers with details as to what functions comprise said benchmarks, SYSmark loses all meaning.

Take Photoshop as an example: AT (and others) can do Photoshop benchmarks with and without GPGPU acceleration, either by disabling GPGPU for certain tests or by simply choosing workloads not accelerated by GPUs. Then they can leave it up to us readers as to which elements of the test are most relevant to our usage patterns, present or future.

The appeal of benchmarks like SYSmark is for know-nothing manager-types who want a single office performance number that they can examine in about five seconds time (or less) to make important purchasing decisions for their entire department. Insert pointy-haired boss where necessary.

 

[mysticjbyrd]

Because you decided so? AMDs GPU market share collapsed when its like the CPU part couldn't compete in any way in performance/watt.

You are the one that incorrectly made the assumption that the only thing that matters is price/performance. If that were the case, then when AMDs CPUs were superior, they would have acquired most of the market share. This did not happen.

AMDs GPU division has been competitive for a long time, thus they should own about half of the marketshare. Again, going by your assumption that $/perf is the only criteria.

Ideally, I agree with you, which is why I said they would be in a much better place in a fair market. $/perf should be the only factor ever considered, but that's not how the real world works under the good circumstances, never mind this crony capitalistic society we live in.

Perhaps SYSmark is yet another one of those factors?

Femme fatale?

Ok, back on topic..

What is BAPCo?
BAPCo stands for Business Applications Performance Corporation and its current members include: Acer, ARCIntuition, ChinaByte, CNET, Compal, Dell, Hewlett-Packard, Hitachi, Intel, LC Future Center, Lenovo, Microsoft, Western Digital, Wistron, Samsung, Sony, Toshiba, Zol and others. BAPCo is a non-profit consortium with a charter to develop and distribute a set of objective performance benchmarks for personal computers based on popular software applications and operating systems. Applications used in SYSmark 2012 were selected based on market research and include Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others.

AMD dropped out, and does not wish to participate.. Doesn't make the benchmark less valid.

http://www.anandtech.com/show/4464/amd-resigns-from-bapco-over-sysmark12-concerns

"Advanced Micro Devices (AMD) was, until recently, a long standing member of BAPCo. We welcomed AMD’s full participation in the two year development cycle of SYSmark 2012, AMD’s leadership role in creating the development process that BAPCo uses today and in providing expert resources for developing the workload contents. Each member in BAPCo gets one vote on any proposals made by member companies. AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.

BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.

BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications."

A non-profit organization is about one of the least credible entities in existence.

So, why isn't the code open source? Wouldn't that make more sense from a non-profit organization looking out for everyone? If you wholeheartedly believe whatever a corporation tells you, then you are naive. Personally, I don't buy it... Something stinks.

Unless you know what is really being measured and the method is open to scrutiny then the results are meaningless.

A black box benchmark covertly distributed as independent by the company producing one of the competing products being tested, fails all requirements of being independent and open to examination and scrutiny.

 

[TheELF]

You are the one that incorrectly made the assumption that the only thing that matters is price/performance. If that were the case, then when AMDs CPUs were superior, they would have acquired most of the market share. This did not happen.

Perf/$ and perf/watt is not the same,also we are talking about those in relation to what the target group wants.
With AMD you get really bad perf/watt in everything that is considered everyday workload,you get very good perf/$ if you only do rendering or something else purely multicored.

For most people browsing and gaming is all they do and that's why AMD has such a low market share.

A non-profit organization is about one of the least credible entities in existence.

So, why isn't the code open source? Wouldn't that make more sense from a non-profit organization looking out for everyone?

Sure release the source code for a benchmark that is supposed to give numbers to government organizations and then start hunting down every modified bench each tom dick and harry releases that decides to skew it in any direction.

 

[mrmt]

Sure release the source code for a benchmark that is supposed to give numbers to government organizations and then start hunting down every modified bench each tom dick and harry releases that decides to skew it in any direction.

Actually Sysmark code can be audited by governmental entities upon request.

 

[TheELF]

Actually Sysmark code can be audited by governmental entities upon request.

Makes sense,they wouldn't release a modified version

 

[mrmt]

Makes sense,they wouldn't release a modified version

Yes, now ask AMD whether they would release the source code or at least allow audit on the source code of their benchmarks scripts.

It's funny how such shady company like AMD thinks it has the moral authority to question >>anyone<< else on the market. From what I get they are the only top-20 semicom enterprise that is being sued for misleading information provided to their own investors.