Highly Technical reason for why benchmarks are never the same

[Shalmanese]

Why do benchmarks vary up to 1-2% between tests? Is it because of invisible CPU scheduling that affects CPU load or something else. I understand why stuff like HD benchmarks vary since the object is mechanical in nature but for game benchmarks where all parts are solid state, shouldn't the numbers be absolutely constant every time?

 

[Nothinman]

They can't be absolutely constant because the OS isn't absolutely constant. The OS has to manage on avg ~50 tasks most of which are sleeping but any of which can be woken up at any time, and memory allocation is never exactly the same so the OS may have to take a few extra cycles to allocate a big enough contiguous block of memory for the game one run where the last it didn't.

 

[corkyg]

No . . . technically speaking, nothing is ever the same. Things can be similar with similar settings, but there will always be some variations in every iteration.

 

[JameyF]

Although I probably don't belong here , I'll give it a try with a simple answer....chaos. There are too many variables that the benchmark program won't be written to oversee. 1-2% difference is usually meaningless. If you need something more accurate, you'll have to search for another program that does manage some/more of these variables, or do what most do and run the test more than once (the more the better) and average the results.

 

[Shalmanese]

But the way I think of it, there is nothing analog in the circuit that can introduce errors, everything is based on the clock. The CPU sends a signal and the memory resoponds exactly 4 clocks later, not 3.85, not 4.21.

Everything should be deterministic. I can understand the OS using up resources but shouldnt this show up on a CPU load chart. Whenever I am in the desktop, the CPU load is consistently 0% which means there is at most a 0.5% variation in CPU load from the OS.

Also, does the position of the data within the memory effect the access times?

Would it be possible to create an OS with no fluctuations in CPU load, ie. everything is running all the time.

 

[Sohcan]

<< Everything should be deterministic >>

But computers aren't deterministic (though for simple performance analysis simulations, treating them as so makes it easier ), even if they were completely solid state. There's just too many variables and events that can take place: cache misses, TLB misses, page faults, IO interupts, exceptions, CPU scheduling, DRAM refreshes, OS system calls, dynamic memory allocation, garbage collection....the list goes on and on. Each of these events can occur at different frequencies, and many have their own variables (a page fault could be dealt with any number of page replacement algorithms, the new page may replace a clean or dirty frame, the replaced page may be of the same or different process of the new page).



<< The CPU sends a signal and the memory resoponds exactly 4 clocks later, not 3.85, not 4.21. >>

There's no guarantee that a memory access always takes the same amount of time....a memory access could hit any one of the caches, the main memory, or a paged frame. There are also TLB hits/misses involved for virtual address translations. Even if a memory access does hit the main memory, it doesn't necessarily take the same amount of time....there could be a column/page hit (not to be confused with virtual memory/paging), a page miss, or a precharge event...for most high-performance DRAM, this takes 2, 4, or 6 cycles, respectively. Plus every few milliseconds the DRAM cells have to be refreshed, which takes time. There's also bus arbitration to consider...there may be multiple outstanding memory accesses which may have to compete for the FSB, along with I/O. And if the memory bus is asynchronous with the FSB, there may be an extra cycle while the memory controller waits for the next rising FSB clock edge.

 

[C2]

I was going to say what he said, but he said it first (ya right)

 

[CTho9305]

the hard drive platters aren't necessarily at the same point rotationally every time you run the benchmark, so swapping times would vary for that matter, the heads could be at a different location causing a longer seek.