Looking for multiplatform CPU benchmarking programs

[dondon44]

Hello!

I was trying to find some good multiplatform CPU benchmarking programs. I want to compare some CPUs of widely different architectures.

What are your recommendations?

Obligatory conditions:
- works on Windows (x86 64-bit)
- works on Linuxes (x86 32&64, ARMv6, ARMv7, ARMv8 A32&A64)
- can compare scores between different platforms & architectures (scores are comparable)
- is free of charge

Bonuses:
- is completely free (as in speech) & open source
- can be used via command line (CLI user interface) (Edit-added)
- has a single-threaded score, or multiple single threaded tests
- produces sensible & useful results

Thanks!

 

[Thala]

From my experience Geekbench is quite reliable - make sure you look at the integer and floating point scores independently. Regarding integer performance 7-zip has nice benchmark mode where you can test single and multicore performance - make sure you compile 7-zip without SSE optimizations in order to not skew the results. Likewise PovRay is open source and you can test floating point performance both single and multicore. Again i suggest to disable AVX in order to not skew the results when compiling.

I did compile both 7-zip and PovRay for Windows ARM64 - if anyone is interested in the binaries drop a note here.

 

[richaron]

Pretty sure Handbrake runs on some/most/all ARM systems (as well as the rest). Plus I've seen reports of Blender doing the same. Both pretty common for use in CPU benchmarking.

 

[Rancor]

You might find something interesting among this list. Maybe compiling something natively https://wiki.archlinux.org/index.php/Stress_testing

 

[Thala]

Pretty sure Handbrake runs on some/most/all ARM systems (as well as the rest). Plus I've seen reports of Blender doing the same. Both pretty common for use in CPU benchmarking.

I am pretty sure that neither Handbrake nor Blender is available in the combination OS: Windows Target Architecture: ARM.

 

[dogen1]

if i had to pick one, probably geekbench?

 

[richaron]

I am pretty sure that neither Handbrake nor Blender is available in the combination OS: Windows Target Architecture: ARM.

Both are open source so I assume they can be compiled for the appropriate architecture.

If I'm wrong I'm happy to hear an explination.

 

[Nothingness]

From my experience Geekbench is quite reliable - make sure you look at the integer and floating point scores independently. Regarding integer performance 7-zip has nice benchmark mode where you can test single and multicore performance - make sure you compile 7-zip without SSE optimizations in order to not skew the results. Likewise PovRay is open source and you can test floating point performance both single and multicore. Again i suggest to disable AVX in order to not skew the results when compiling.

Did you disable NEON? Would you disable SVE if you had an ARM CPU supporting it because Intel doesn't have variable length vectors?

I disagree that instruction set features should not be used. If an ISA has good instruction sets, it should be used to its full potential.

 

[Thala]

Both are open source so I assume they can be compiled for the appropriate architecture.

If I'm wrong I'm happy to hear an explination.

I am just saying that both currently are not existing as binary for benchmarking - so it most likely does not help the ask of the original poster in his search for multi-platform benchmarking programs.

Regarding your second question and taking Blender as example.
1) It uses cmake in order to generate the according visual studio solutions - cmake currently cannot generate arm64 solutions for visual studio
2) Even if cmake could generate the solutions, the cmake files do the wrong assumtion, that if the compiler compiles 64 bits it must be x64 - so the blender makefiles need to be updated
As example you find lines like:

if(CMAKE_CL_64)
set(PLATFORM_LINKFLAGS "/MACHINE:X64 ${PLATFORM_LINKFLAGS}")
else()
set(PLATFORM_LINKFLAGS "/MACHINE:IX86 /LARGEADDRESSAWARE ${PLATFORM_LINKFLAGS}")
endif()

in platform_win32_cmake in the latest Blender source repository.

3) blender only provides the 3rd party (open source) libraries as binaries and not as source - currently only x86 and x64 versions - you need to compile the libs yourself and do some guesswork related to required version and configuration.

 

[Thala]

Did you disable NEON? Would you disable SVE if you had an ARM CPU supporting it because Intel doesn't have variable length vectors?
I disagree that instruction set features should not be used. If an ISA has good instruction sets, it should be used to its full potential.

For code i use as benchmark, i make sure that i use vector extensions on all architectures or on neither architecture. For both 7-zip and PovRay there was no usable NEON code-path available - so i disabled SSE such that the used source code is 100% identical.

 

[Nothingness]

For code i use as benchmark, i make sure that i use vector extensions on all architectures or on neither architecture. For both 7-zip and PovRay there was no usable NEON code-path available - so i disabled SSE such that the used source code is 100% identical.

If there's no room to use NEON, I doubt vectorized SSE would be applicable.

 

[dondon44]

I am pretty sure that neither Handbrake nor Blender is available in the combination OS: Windows Target Architecture: ARM.

Sorry for the misunderstanding, but such a target is not my requirement (I have just clarified this in the original post). For Windows, the only supported architectures need to be x86 32&64. I don't mind compiling the benchmark.

I can see that there are lot of proposals for Handbrake, 7-zip and PovRay. Those are OK in the sense of testing performance in specific applications. A bit cumbersome to do since each has to be compiled and run separately. Also, while those benchmarks are OK, I would prefer something slightly more general.

Also, CLI interface is highly preferred (added in original post).

Geekbench mostly fits the description, except that it is not open-source at all. Can someone comment on how well does it compare different architectures, can the results be sensibly compared?

I disagree that instruction set features should not be used. If an ISA has good instruction sets, it should be used to its full potential.

I partially agree. For example, Linpack, I would enable AVX as Linpack is usually compiled for a specific CPU. For 7-zip, I would most likely use the same settings as the official binary uses. So, it all depends on what is being benchmarked and why.

Thank you all for the replies.

 

[dondon44]

Wait, Geekbench does not have a CLI user inteface. Well, that's a meh. And not open source.

At least, is it any good in comparing different architectures?

Edit:
This state of things is still unsatisfactory to me. I can't find any suitable benchmarks, and the only option I have is to run multiple CLI applications one-by-one, and use their individual scores as a benchmark. This is very cumbersome and biased towards selected applications and their specific routines.

I was hoping there would be many free general-purpose benchmarks available, but it seems that I was wrong (given my understanding of what is 'general purpose').

 

[Thala]

If there's no room to use NEON, I doubt vectorized SSE would be applicable.

I meant this literally - no available code path for ARM64 i could use for compilation - not that there is no room for ARM64 optimizations.

 

[Thala]

Sorry for the misunderstanding, but such a target is not my requirement (I have just clarified this in the original post). For Windows, the only supported architectures need to be x86 32&64. I don't mind compiling the benchmark.

I wonder why you would drop this requirement. In particular since Windows allows you to load all 8 cores in a big.LITTLE configuration - and the latest ARMs do have the thermal headroom to do so in a passively cooled tablet configuration without throttling. This means with an <5W ARM Windows machine you are typically in the multicore performance region of the 15W actively cooled Intel machines and beyond.

Geekbench mostly fits the description, except that it is not open-source at all. Can someone comment on how well does it compare different architectures, can the results be sensibly compared?

As usual, since this is closed source, the results are compiler dependent - and Linux versions using a different compiler than Windows. Other than this i consider the results very much in line with every other code i ran when comparing ARM vs x86.

 

[dondon44]

Sorry for the misunderstanding, but such a target is not my requirement (I have just clarified this in the original post). For Windows, the only supported architectures need to be x86 32&64. I don't mind compiling the benchmark.

I wonder why you would drop this requirement. In particular since Windows allows you to load all 8 cores in a big.LITTLE configuration - and the latest ARMs do have the thermal headroom to do so in a passively cooled tablet configuration without throttling. This means with an <5W ARM Windows machine you are typically in the multicore performance region of the 15W actively cooled Intel machines and beyond.

Well, I never installed Windows on ARM, and I don't have any devices with such combination around me.

Also, I assume that the difference Windows vs. Linux should be minor for the type of benchmark I would like to do. The compiler would have some effect, ISA extensions perhaps even more, but the greatest factor is the CPU itself- the thing being tested.

So instead Windows on ARM, the result of Linux on ARM is 'close enough' for me. Would be surprised to see a significant difference.

Edit - Windows on ARM??? Is anyone using that at all? It appears to be an insignificant platform to me. I think I'm just not going to bother myself with this combination.

 

[Thala]

Well, I never installed Windows on ARM, and I don't have any devices with such combination around me.

Also, I assume that the difference Windows vs. Linux should be minor for the type of benchmark I would like to do. The compiler would have some effect, ISA extensions perhaps even more, but the greatest factor is the CPU itself- the thing being tested.

So instead Windows on ARM, the result of Linux on ARM is 'close enough' for me. Would be surprised to see a significant difference.

Edit - Windows on ARM??? Is anyone using that at all? It appears to be an insignificant platform to me. I think I'm just not going to bother myself with this combination.

Regarding Linux on ARM, i mostly used Android devices in the past as baseline here, and the CPU governor - which determines dynamic thread distribution has always been hit and miss for me. I many cases, when creating 8 threads it was just not using all 8 cores (1 thread per core). On Windows at least that has been very consistent as far, if i have 8 non-blocking threads all 8 cores of the DynamIQ big.LITTLE cluster will be loaded 100%.
Regarding Windows on ARM, this has been a very good experience for me. The device is passively cooled, never gets hot, has very long battery life and extremely long active standby time - and has multithreaded performance easily matching and exceeding 15W actively cooled Intel devices - considering it is only Cortex A73/A53.

 

[dondon44]

Regarding Linux on ARM, i mostly used Android devices in the past as baseline here, and the CPU governor - which determines dynamic thread distribution has always been hit and miss for me.

I guess I'm much more interested in single-threaded than in multithreaded results.

If a specific OS is not loading all the cores, well - that is an interesting fact to know, but it is mostly an OS software issue which will likely be fixed in the future.

 

[Nothingness]

I meant this literally - no available code path for ARM64 i could use for compilation - not that there is no room for ARM64 optimizations.

Oh I see: you mean intrinsics or assembly language dedicated routines. In that case I certainly agree with you.

 

[Thala]

Oh I see: you mean intrinsics or assembly language dedicated routines. In that case I certainly agree with you.

Yes indeed, i was talking about hand optimized assembly/intrinsics.

 

[dondon44]

Regarding Linux on ARM, i mostly used Android devices in the past as baseline here, and the CPU governor - which determines dynamic thread distribution has always been hit and miss for me.

Perhaps the problem is that you used Android, and it is likely that Android's scheduler is optimized for power-saving instead of performance.

I was thinking of rooting the ARM devices and running the benchmarks on something much closer to the mainline kernel, in fact, my plan was to use Debian. I think it is unlikely that I am going to run into the same multi-core throttling issue you encountered on Android.

 

[Rancor]

I stumbled across nbench which runs on Linux, Windows in combination with cygwin, and there is supposed to be a Mac port. NBench is single-threaded. It looks like an old benchmark judging by the baseline and results table. I saw someone use it to benchmark an ARM9 BMC (Baseband Management Controller) built into a modern Xeon workstation mobo.