Truth or Myth?: Is SYSmark a Reliable Benchmark?

[mysticjbyrd]

Truth or Myth?: Is SYSmark a Reliable Benchmark?


Spoiler:
"Not representative of real world performance."

"Only optimized for intel."

 

[jhu]

I could have told you that

 

[Good_fella]

Typical AMD. First they not sending Fury Nano review samples because of mythical biased review sites, now blaming benchmarks.

I wonder how AMD Defense Force will spin this? If Intel/Nvidia would do that AMDDF would bash them.

 

[TheELF]

2:13
"In the case of the sysmark suit, when probed, we identified that there is an excessive high amount of CPU tasking being done"

Well I'd never...who ever thought of benchmarking a CPU by using an excessive high amount of CPU tasking? That's just crazy,who ever used their CPU to do CPU stuff anyway?

 

[ViRGE]

2:13 Well I'd never...who ever thought of benchmarking a CPU by using an excessive high amount of CPU tasking? That's just crazy,who ever used their CPU to do CPU stuff anyway?

That is pretty much the crux of their argument, that there's more to system performance than CPU performance. That said BD is so far behind the curve right now that it's hard to take AMD's concerns seriously. They're losing on the current test, so of course they want to change the test.

 

[guskline]

Virge, I couldn't agree more. I have been a loyal AMD follower since the AMD 386DX40 days so I go way back. AMD in the top end lost it's way. They better hope Zen gets them back in the hunt. Crying foul about a benchmark at this stage is silly. Build faster cpus.

 

[Arachnotronic]

Build faster cpus.

Doing so is really, really hard.

 

[MrTeal]

Two ways to look at it, really. If you're looking at a CPU benchmark, as TheELF said it's pretty ludicrous to criticize it for excessive CPU usage. If you're looking at choosing a CPU for a discrete build or are trying to pare down a list of laptops using a discrete CPU, it might be your primary metric for that part of the build.

If you're an average customer just buying a laptop for general use though, Sysmark really isn't a great tool to use as a primary determinant of performance. As a single score to determine platform performance, something that also stresses the GPU, the storage subsystem, battery life, etc would be more representative of how they would experience the system in general use.

The construction cores might be a disaster on the CPU front, but most people don't stress the CPU enough other than in gaming for that really to make a big difference. If the Intel laptop (using the numbers in the video) is 50% faster in Sysmark than an AMD laptop but 6-7% faster in PCMark, at the same price you'd still go with Intel. If the Intel laptop costs 40% more than the AMD one, the benchmark metric you use can play a large part in how you define the platform value.

 

[coercitiv]

If you're looking at a CPU benchmark, as TheELF said it's pretty ludicrous to criticize it for excessive CPU usage.

But is SYSmark a CPU benchmark?

SYSmark is an application-based benchmark that reflects usage patterns of business users in the areas of office productivity, media creation, and data/financial analysis. SYSmark 2014 ver 1.5 features the latest and most popular applications from each of their respective fields.

 

[MrTeal]

But is SYSmark a CPU benchmark?

Not really sure what system components SYSmark really tests. I really shouldn't have used Sysmark in the second paragraph and used CPU benchmark instead, my comments were more intended to be general. AFAIK it's not a pure CPU/memory bound benchmark like say Prime95 would be. Obviously in their marketing AMD is going to want to push their advantages. Hence why they're always pushing PCMark accelerated. That in itself can create some issues, as most people probably won't be using a lot of OpenCL accelerated stuff.

 

[railven]

Wonder if this is a premptive move to get Sysmark removed before Zen comes out and possibly flops on it too.

Tin foil hatting, why would AMD care this far into Bulldozer to even try this angle?

 

[mysticjbyrd]

That is pretty much the crux of their argument, that there's more to system performance than CPU performance. That said BD is so far behind the curve right now that it's hard to take AMD's concerns seriously. They're losing on the current test, so of course they want to change the test.

Sounded like the crux of their argument were the things I quoted. If you can't refute those claims, then you are just howling in the wind.

 

[mrmt]

That is pretty much the crux of their argument, that there's more to system performance than CPU performance. That said BD is so far behind the curve right now that it's hard to take AMD's concerns seriously. They're losing on the current test, so of course they want to change the test.

I would be very interested in hear AMD's propositions for a new benchmarks of the same baseline scenarios.

 

[ShintaiDK]

Wonder if this is a premptive move to get Sysmark removed before Zen comes out and possibly flops on it too.

Tin foil hatting, why would AMD care this far into Bulldozer to even try this angle?

It doesn't inspire confidence when they have to make YouTube slander on a CPU heavy benchmark ahead of Zen.

 

[Burpo]

Exactly what I saw when I watched it.. Sad that they're "prepping" the world for a let down. I had hope until I saw that.

"On certain cherry picked benchmarks, we're only behind 6 or 7%!" :\

Who runs AMD Marketing? A moron?

 

[Vortex6700]

This is not a good marketing move, but are we going to pretend Intel bias in benchmarks hasn't happened before?

Seriously, the only part of intel that hasnt broken anticompetition laws is the janitorial staff.

Oh wait, 386 clean rooms. nevermind.

 

[Deders]

Looks to me like Sysmark might favour single threaded performance.

 

[Lil'John]

Sounded like the crux of their argument were the things I quoted. If you can't refute those claims, then you are just howling in the wind.

The converse of the bolded also holds true: "If you don't prove your claims, then you are just howling in the wind."

I didn't go watch the video but from the OP what proof was offered for either of these claims:
"Not representative of real world performance."
"Only optimized for intel."

From reading the comments here, it sounds like the video is an AMD PR video trying to mitigate bad performance of BD on Sysmark... and if this is indeed the claim, it would raise concerns to me about the potential performance of ZEN.

 

[bystander36]

Let's be honest here. Any synthetic benchmark is not going to mirror real life performance. Synthetics test what is possible, and try to mimic real world tasks, but if you have a task in mind, just compare benchmarks for those programs.

 

[ViRGE]

From reading the comments here, it sounds like the video is an AMD PR video trying to mitigate bad performance of BD on Sysmark... and if this is indeed the claim, it would raise concerns to me about the potential performance of ZEN.

One thing also worth keeping in mind is that AMD resigned from the BAPCo board almost 5 years ago. This is a new video, but AMD's complaints (and overall desire to get SysMark written off) haven't really changed since the beginning. Every year or two make noise to remind everyone about their concerns with SysMark and that they don't participate in its development.

 

[PPB]

Sysmark is garbage and will ever be. Just like cinebench is when they show you 5-7% ipc gain from ivy to haswell and then real world renderwrs showed 20% for example. They fail to accurately depict the real world user case they pretend to represent. They are thus bad benchmarks altogether.

I see apologists already picked this as a way to slam Zen. That argument is silly because they bench tool they push in the video relies more on igp performance. Tell me again which Zen product is gonna ship first?

For the anti AMD brigade, there is another company you are very fond of that also left BAPCO with similar claims as AMD. I think they kind of make gpus, who would that be?....

 

[MrTeal]

Let's be honest here. Any synthetic benchmark is not going to mirror real life performance. Synthetics test what is possible, and try to mimic real world tasks, but if you have a task in mind, just compare benchmarks for those programs.

Agreed, although even benchmarks on applications you use can be misleading if your usage is different. If you primarily use a laptop for Excel, should you really rely on a scripted Excel benchmark? For most people, a good keyboard, monitor, SSD and good battery life would be much more important for Excel use than an Excel benchmark, since 95% of what they're doing is simple data entry and arithmetic. Similar thing with Photoshop; a pro doing batch processing might care about raw PS performance. A home user trying to crop their ex out of a vacation picture to post on Facebook, maybe not as much.

 

[sontin]

For the anti AMD brigade, there is another company you are very fond of that also left BAPCO with similar claims as AMD. I think they kind of make gpus, who would that be?....

nVidia left because the benchmark wasnt relevant for them anymore after they left the chipset buisness.

AMD on the other hand is selling CPUs without an iGPU. So they are complaining about a CPU benchmark because their CPUs are much more worse than Intel's...

It is like saying that these 5 games are not reliable benchmarks because their GPUs perform worse than nVidia's...

It is a dumb marketing move to blame the software.

 

[mysticjbyrd]

The converse of the bolded also holds true: "If you don't prove your claims, then you are just howling in the wind."

I didn't go watch the video but from the OP what proof was offered for either of these claims:
"Not representative of real world performance."
"Only optimized for intel."

From reading the comments here, it sounds like the video is an AMD PR video trying to mitigate bad performance of BD on Sysmark... and if this is indeed the claim, it would raise concerns to me about the potential performance of ZEN.

I absolutely agree, but AMD did offer up evidence to prove their claims. Though, since we can't see it in detail, you can claim it's rather weak. Still though, I don't see any counter arguments here.

Is SYSmark indicative of real world performance?
Is SYSmark only optimized for Intel?

 

[Deders]

I don't want to be too down on AMD but if their CPU logic is anything like the logic they used in that video, it's no wonder they are scoring poorly.