Truth or Myth?: Is SYSmark a Reliable Benchmark?

[AtenRa]

Office Productivity:

Adobe® Acrobat Pro XI
Google® Chrome® 32.0
Microsoft® Excel® 2013
Microsoft® OneNote® 2013
Microsoft® Outlook® 2013
Microsoft® PowerPoint® 2013
Microsoft® Word® 2013
WinZip Pro 17.5

Media Creation:

Adobe® Photoshop® CS6 Extended
Adobe® Premiere® Pro CS6
Trimble® SketchUp™ Pro 2013

Data / Financial Analysis

Microsoft® Excel® 2013
WinZip Pro 17.5

These do not represent real world workloads? :hmm:

Those DOES represent real world applications, BUT

Adobe Photoshop also supports GPU acceleration

Adobe® Premiere Pro CS6 also supports GPU acceleration
http://blogs.adobe.com/premierepro/2013/06/adobe-premiere-pro-cc-and-gpu-support.html
http://blogs.adobe.com/premierepro/2011/02/cuda-mercury-playback-engine-and-adobe-premiere-pro.html

Trimble® SketchUp™ Pro 2013 also uses OpenGL on GPUs, using the CPU to make the job of the GPU is not the most efficient method

https://help.sketchup.com/en/article/114278

3D applications, such as SketchUp, require abundant system resources. Aside from having a fast CPU and large amounts of RAM, your video card and video card drivers must be 100% OpenGL compliant. What is OpenGL?
OpenGL is the industry-standard, used in numerous software applications and games, to draw 3D geometry. Most Microsoft Windows and Apple OS X operating systems come with a software-based OpenGL driver. However, these drivers rely heavily on the CPU to perform the rendering calculations of OpenGL (a task that is not done efficiently by most CPUs).
Many video card manufacturers have also built cards that support the OpenGL standard. These cards perform the rendering calculations using a specialized chip called the Graphics Processing Unit or GPU (instead of relying on the CPU). These chips significantly enhance OpenGL performance upward of 3000 percent. This performance enhancement is known as Hardware Acceleration.

Also, WinZip fom version 16.5 onward has OpenCL acceleration
http://www.winzip.com/whatsnew165.htm


So, those are real world applications but if you only use the CPU to measure the performance is not a real world usage, especially in a Laptop

 

[USER8000]

Good to see you find impartial forum posters.

/sarcasm

I dunno what some hatred of a forum poster has got to do with it:

https://www.crunchbase.com/person/shervin-kheradpir#/entity

That's the first Google hit for him.

VP, Sales & Marketing Group & GM, Platform Evaluation & Competitive Assessmen

Kheradpir joined Intel in 1989 as a technical marketing engineer shortly after completing his education.

If you Google his name and BAPCo,you get the following hit:

http://www.businesswire.com/news/home/20060925006184/en/BAPCo-Announces-EECoMark-TM----Joint-Venture

Shervin Kheradpir, President of BAPCo

So,VP, Sales & Marketing Group & GM, Platform Evaluation & Competitive Assessment at Intel being a president of BAPCo,at least in 2006.

Is that normal for these sort of benchmark companies. Maybe it is?? I dunno,seems rather weird.

But still I don't see any point in anybody really caring about a benchmark where only Intel and not AMD,Nvidia and ARM are present.

Having 75% of all the companies providing CPUs and GPUs to Windows PC makes it pointless IMHO.

Plus does anyone really use something like Sysmark or 3DMark to buy parts??

Seems so 90s!!

 

[AtenRa]

Personally i believe the truth is somewhere in between. Intel has the faster CPU and AMD has the faster GPU. More and More applications are using GPU acceleration. Intel wants to promote its CPU strong performance and AMD wants to promote its GPU strong performance.
So the performance is highly effected on the personal application and usage of each individual.

Also today lots of companies are going Cloud only (local or Internet). That means that CPU and iGPU advantage on laptops or office desktops goes to hell for those corporate sales.

 

[USER8000]

I think all these benchmark suites are dumb anyway - whenever has 3DMark ever been a good indication of actual performance?? Great for benchmarking threads though.

Its just better,if you really want to care about CPU or GPU performance to look at the multitude of tests on review sites out there and choose whatever suits what you are doing better.

If one CPU or GPU is better at a price-point,it will be pretty easy to spot.

Edit to post.

Even if some of these benchmarks are proven to have Intel involvment,AMD either need to push to get involved in the same way or simply try and get market exposure in products,either through better products and/or working with OEMs and software devs harder. The whole problem is they are fighting companies with more resources,so they sadly needly to work twice as hard to get marketshare back.

 

[mrmt]

Is that normal for these sort of benchmark companies. Maybe it is?? I dunno,seems rather weird.

if we are talking about a consortium then we are talking about the companies members of the consortium providing resources, personnel included, for it to reach it objectives, so the fact that we are seeing an Intel benchmark guy working on BAPCO is nothing short of what would be expected in a consortium.

AMD should have been providing appointments to key positions until they resigned, especially because they had to at least analyze BAPCO's proposals for their benchmarks tools.

 

[TheELF]

Those DOES represent real world applications, BUT

Adobe Photoshop also supports GPU acceleration

Adobe® Premiere Pro CS6 also supports GPU acceleration
http://blogs.adobe.com/premierepro/2013/06/adobe-premiere-pro-cc-and-gpu-support.html
http://blogs.adobe.com/premierepro/2011/02/cuda-mercury-playback-engine-and-adobe-premiere-pro.html

Trimble® SketchUp™ Pro 2013 also uses OpenGL on GPUs, using the CPU to make the job of the GPU is not the most efficient method

https://help.sketchup.com/en/article/114278
Also, WinZip fom version 16.5 onward has OpenCL acceleration
http://www.winzip.com/whatsnew165.htm


So, those are real world applications but if you only use the CPU to measure the performance is not a real world usage, especially in a Laptop

These programs do use CL but it is in a very small percentage,for example photoshop,from your link:

Blur Gallery - Field Blur, Iris Blur, Tilt-Shift, Path Blur, Spin Blur (OpenCL accelerated)
Smart Sharpen (Noise Reduction – OpenCL accelerated)
Oil Paint (OpenCL accelerated)

Only 3 filters that use the GPU for CL, everything else is just good old displaying graphics and nothing more...
No matter how much the companies will try, there are only limited uses for CL/cuda,CPU grunt will always be more important for the overall performance.

 

[USER8000]

if we are talking about a consortium then we are talking about the companies members of the consortium providing resources, personnel included, for it to reach it objectives, so the fact that we are seeing an Intel benchmark guy working on BAPCO is nothing short of what would be expected in a consortium.

AMD should have been providing appointments to key positions until they resigned, especially because they had to at least analyze BAPCO's proposals for their benchmarks tools.

I looking around on Google and saw this article from The Register in 2000:

http://www.theregister.co.uk/2000/07/24/intel_and_bapco_just_good/

When we asked Intel last week what they were doing to alleviate their self-professed 'difficulty' with benchmarks not showing how well Rambus really performed against PC-133 SDRAM, they told us: "We're working with the major benchmarking organisations to develop more realistic test methodologies which will show the kind of performance of which Rambus is capable."


But it wasn't until Van Smith, Technical Editor of Tom's Hardware Guide, suggested we checked out the other Register, the one which lists domain names, that we discovered the grisly truth: that BAPCo, the benchmarking outfit that all hardware review sites rely on, is working closer with Intel than is seemly.


In fact, 'incestuous' might be the word we're looking for here.


A search for BAPCo returns the following address: Business Applications Performance Corporation (BAPCo) (BAPCO-DOM)
2200 Mission College Blvd MS
SC12-608 (Sc12 = building number, 608 = maildrop)
Santa Clara, CA 95052
Whilst a search for Chipzilla tells us they're to be found at:
Intel Corporation (INTEL-DOM)
2200 Mission College Blvd,
P.O. Box 58119
Santa Clara, CA 95052-8119
Just fancy that! ®

Then,if you at look at some of the press releases around the same time:


http://www.futuremark.com/pressreleases/20409

John Peterson BAPCo 2200 Mission College Blvd., SC12-608, Santa Clara, CA 95052 408-988-7654

He works for Intel too:

https://www.linkedin.com/in/john-peterson-390b39a8

Why is/was BAPCo situated at a Intel site??

So,the VP, Sales & Marketing Group & GM, Platform Evaluation & Competitive Assessment was the president of BAPCo,some other bloke in the early 2000s who worked for Intel was a representative of the company and BAPCo offices were located at Intel headquarters.

If that is the case no wonder AMD,Nvidia and VIA all left. Intel is only chip make left at BAPCo.

Just seems all rather weird. Sure,it still does not mean Intel changed the benchmark to "suit" them(hehe!),but still...!

Emm,if that is normal for "independent" benchmarking companies I think I am going to avoid all these "test suites".

We might find Passmark HQ is located in the AMD or Nvidia HQ(not saying they are) for all we know!!

Think I would rather read review sites and get an overview of what products to buy.

Edit to Post.

After remembering all the mucking around with 3DMark by vendors,I can't see why anyone would even want to take half these benchmarks suites even at face value. Only great for pointless benchmark threads on forums.

Enough reviews out there to get a good overview of what is better or worse.

 

[The Stilt]

Funny how suddenly the very same applications become relevant, real world applications when GPGPU is used :sneaky:

https://www.youtube.com/watch?v=4CwwQq1ddQ4

 

[ShintaiDK]

Funny how suddenly the very same applications become relevant, real world applications when GPGPU is used :sneaky:

https://www.youtube.com/watch?v=4CwwQq1ddQ4

LOL, nice catch

 

[USER8000]

Funny how suddenly the very same applications become relevant, real world applications when GPGPU is used :sneaky:

https://www.youtube.com/watch?v=4CwwQq1ddQ4

True,but the difference at least we know they are biased AMD ones showing their products in the brightess glow of Polaris..... not one from an independent benchmarking trading company who might/might not be biased.

This is the problem - we can't say how biased/unbiased all these test suites like Sysmark,Passmark,3DMark,etc are.

Its why were need to get an average of reviews out there too see what is better. We can even see that with independent reviews the same software can be tested,ie,games for example,and we can see different relative performance for CPUs and graphics cards. A bit like Crysis3 - test an AMD CPU in welcome to the jungle it looks far better than some other parts of the game,etc.

 

[Burpo]

Comparing a 35 watt AMD FX8800P to a 15 watt Intel i5? Ya think it's biased?

 

[AtenRa]

Comparing a 35 watt AMD FX8800P to a 15 watt Intel i5? Ya think it's biased?

It says 15W in the video not 35W for the FX8800P

 

[The Stilt]

Comparing a 35 watt AMD FX8800P to a 15 watt Intel i5? Ya think it's biased?

FX-8800P is a 15W TDP APU, however it is allowed to boost to 25W.

 

[Burpo]

AMD FX-8800P
Description: Socket: FP4, Clockspeed: 2.1 GHz, Turbo Speed: 3.4 GHz, No of Cores: 2 (2 logical cores per physical), Max TDP: 35 W
Other names: AMD FX-8800P Radeon R7, 12 Compute Cores 4C+8G

 

[The Stilt]

TDP 15W, cTDP 12 - 35W. C stands for configurable.

 

[Burpo]

In this case the c stands for cheat/biased. It's a more powerful cpu than the 15 watt TDP intel 5200u. Apples to oranges..

 

[positivedoppler]

I looking around on Google and saw this article from The Register in 2000:

http://www.theregister.co.uk/2000/07/24/intel_and_bapco_just_good/
Then,if you at look at some of the press releases around the same time:


http://www.futuremark.com/pressreleases/20409


He works for Intel too:

https://www.linkedin.com/in/john-peterson-390b39a8

Why is/was BAPCo situated at a Intel site??

So,the VP, Sales & Marketing Group & GM, Platform Evaluation & Competitive Assessment was the president of BAPCo,some other bloke in the early 2000s who worked for Intel was a representative of the company and BAPCo offices were located at Intel headquarters.

If that is the case no wonder AMD,Nvidia and VIA all left. Intel is only chip make left at BAPCo.

Just seems all rather weird. Sure,it still does not mean Intel changed the benchmark to "suit" them(hehe!),but still...!

Emm,if that is normal for "independent" benchmarking companies I think I am going to avoid all these "test suites".

We might find Passmark HQ is located in the AMD or Nvidia HQ(not saying they are) for all we know!!

Think I would rather read review sites and get an overview of what products to buy.

Edit to Post.

After remembering all the mucking around with 3DMark by vendors,I can't see why anyone would even want to take half these benchmarks suites even at face value. Only great for pointless benchmark threads on forums.

Enough reviews out there to get a good overview of what is better or worse.

So Intel pretty much bought off BAPco from under the table? It's like if the FDA has an office in the heart of Monsanto's corporate office

 

[coercitiv]

In this case the c stands for cheat/biased.

Are you also accusing Intel of cheating? Their low voltage CPU have had cTDP ever since Ivy Bridge.

Funny how suddenly the very same applications become relevant, real world applications when GPGPU is used.

Question: is SYSmark using GPU acceleration where possible? If it is then AMD is full of it, if it's not, then SYSmark is an irrelevant benchmark.

These programs do use CL but it is in a very small percentage,for example photoshop,from your link:

Small percentage... the entire UI benefits from GPU acceleration. Intel GPUs handle it just fine, but make no mistake: Photoshop without GPU acceleration is a really sad experience.

GPU-enhanced features added in Photoshop CS6

• Adaptive Wide Angle Filter (compatible video card required)
• Liquify (accelerated with compatible video card with 512 MB of VRAM)
• Oil Paint (compatible video card required)
• Warp and Puppet Warp (accelerated with compatible video card)
• Field Blur, Iris Blur, and Tilt/Shift (accelerated with compatible video card supporting OpenCL)
• Lighting Effects Gallery (compatible video card required with 512 MB
  of VRAM)
• New 3D enhancements (3D features in Photoshop require a compatible video card with 512 MB of VRAM):
  • Draggable Shadows
  • Ground plane reflections
  • Roughness
  • On-canvas user interface controls
  • Ground plane
  • Light widgets on edge of canvas
  • IBL (image-based light) controller

GPU features added in previous versions Photoshop

• Scrubby Zoom. See Zoom continuously.
• Heads Up Display (HUD) color picker. See Choose a color while painting.
• Color sampling ring. See Choose colors with the Eyedropper tool.
• Brush dynamic resize and hardness control.
• Bristle Brush tip previews. See Bristle tip shape options.
• Rule of thirds crop grid overlay. See Crop images.
• Zoom enhancements. Smooth display at all zoom levels and temporary zoom. See Zoom continuously and Temporarily zoom an image.
• Animated transitions for one-stop zoom. Press Ctrl+Plus Sign (Windows) or Command+Plus Sign to zoom, and the image animates slightly between zoom levels. The zoom can be subtle.
• Flick-panning. Choose Edit > Preferences (Windows) or Photoshop > Preferences (Mac OS). In the General panel, select Enable Flick Panning. Then, select the Hand tool and click-flick the image, like a flick gesture on an iPhone. The image glides smoothly to the new position.
• Rotate the canvas. See Use the Rotate View tool.
• View nonsquare pixel images. See Adjust pixel aspect ratio.
• Pixel grid. A pixel grid appears when zooming in more than 500% on an image. See Hide the pixel grid.
• Adobe Color Engine (ACE). Color conversions are faster because the GPU handles the processing instead of the CPU.
• Draw Brush tip cursors. Choose Edit > Preferences (Windows) or Photoshop > Preferences (Mac OS). In the Cursors panel, choose a Brush Preview color. Then, when you interactively adjust the size or hardness of the Brush tool, the preview color displays the change in real time.

 

[TheELF]

Small percentage... the entire UI benefits from GPU acceleration. Intel GPUs handle it just fine, but make no mistake: Photoshop without GPU acceleration is a really sad experience.

That's exactly the same to what I said.
Face it, cl/cuda is just as useless/usefull as "multithreaded" (the ones that can get 100% usage on all available cores) software is ,it can only work on very limited graphics and database scenarios.

These programs do use CL but it is in a very small percentage,for example photoshop,from your link:

Only 3 filters that use the GPU for CL, everything else is just good old displaying graphics and nothing more...
No matter how much the companies will try, there are only limited uses for CL/cuda,CPU grunt will always be more important for the overall performance.

 

[Phynaz]

Question: is SYSmark using GPU acceleration where possible?

Something I'd like to know also.

 

[Phynaz]

What's even funnier is using their own custom developed test scripts AMD is still slower than Intel, while using more power.

"Yeah, we're just not as bad as our competition says we are."

Maybe, but your marketing team is as bad as ever.

 

[AtenRa]

Something I'd like to know also.

nope,
And there is too much CPU Core scaling in office workloads that you dont see in real world. Does Office 2013 scale to 6-cores ???

https://bapco.com/wp-content/uploads/2015/09/SYSmark2014Whitepaper_1.0.pdf

So, SYSmark 2014 is purely a CPU benchmark and not representative of real system performance in todays applications.

 

[Phynaz]

Stop it please, you don't know if they do or not. I don't want your supposition, I want proof.

 

[Burpo]

And as a CPU benchmark, it was wise of AMD to drop out..

 

[AtenRa]

Stop it please, you don't know if they do or not. I don't want your supposition, I want proof.

Did you see second picture i just posted ??? the one with GPU sensitivity ??? R9 290X is 4.8% faster than HD4350 in Media Creation, really ???

Did you just forget what Applications SYSmark uses for the Media Creation scenario ???

Media Creation:

Adobe® Photoshop® CS6 Extended
Adobe® Premiere® Pro CS6
Trimble® SketchUp™ Pro 2013

All three use GPU acceleration but R9 290X is only 4.8% faster than HD4350