AT Posts Revised benchmarks Ryzen/Intel, much close to other sites

[William Gaatjes]

I think the whole idea for HPET is good but when reading about the implementation, blegh. It is as if they did not have transistors left and used the simplest solution.

But it is strange that HPET causes such a performance reduction when forced.
Although i am still reading about it.
I think anandtech did great, they found interesting results, went digging when notified something was off and found the culprit.
A lot of people chimed in and now we have another great story that reads as an detective novel.
Anandtech did nothing wrong, i would say this is another example what attracted me to anandtech articles.


edit:
Aha, a bad design decision in the chipset from Intel.

 

[wahdangun]

I think the whole idea for HPET is good but when reading about the implementation, blegh. It is as if they did not have transistors left and used the simplest solution.

But it is strange that HPET causes such a performance reduction when forced.
Although i am still reading about it.
I think anandtech did great, they found interesting results, went digging when notified something was off and found the culprit.
A lot of people chimed in and now we have another great story that reads as an detective novel.
Anandtech did nothing wrong, i would say this is another example what attracted me to anandtech articles.


edit:
Aha, a bad design decision in the chipset from Intel.

Yeah this need to be tested more, maybe testing a program that use HPET while, not forcing it.

It will be interesting to know if there are any performance impact.

And since HPET quite crucial for VM (my VM will constantly time drifting if I'm not enable HPET) the any related VM benchmark will be invalid, since it's not representative.

 

[PhonakV30]

on other hand both Results are correct.one is for HPET Off and one is for HPET On.

 

[William Gaatjes]

Yeah this need to be tested more, maybe testing a program that use HPET while, not forcing it.

It will be interesting to know if there are any performance impact.

And since HPET quite crucial for VM (my VM will constantly time drifting if I'm not enable HPET) the any related VM benchmark will be invalid, since it's not representative.

Well, the overclockers site linked by AT Ryan & Ian has a timer bench program that you can use to test it. They go indepth about the problem.

https://www.overclockers.at/articles/the-hpet-bug-what-it-is-and-what-it-isnt

Anandtech recently released an article that pointed out problems with their CPU reviews due to an enabled High Precision Event Timer in Windows. Some Intel processors suffered from decreased performance in games and other benchmarks. Since then a lot of misconceptions are going around. People are calling out Intel as cheaters when actually the opposite is going on. We take this opportunity to have another look at the HPET bug and finally announce TimerBench, our Windows timer benchmark to the public. It provides proof behind these infamous HPET problems and helps you to test the impact of your timer configuration on your system performance.


Let's rip the bandaid off: Yes, there IS an HPET bug and no, Intel is NOT cheating. The contrary is the case, Intel suffers from this bug and we are pretty sure that Anandtech is not the only review site that has published wrong results because of these issues. Evidently there are lots of people out there having low framerates on their setups and no explanation for it.

To understand this properly we need to go back serveral years when HPET was already diagnosed to hurt gaming performance in certain situations. Forum threads about this topic popped up here an there, lots of misinformation and make-believe happened back then. Yet the root of some of the performance problems was the same as it is today, but in a different way. HPET was always a costly way to retrieve an incremental timestamp counter, especially when CPU cycles were on a tight budget. A lot of things that are now implemented in hardware had to be done in software in the early days. Not to forget that CPUs had a single core or games were simply not ready to use multiple threads in an efficient way. So the usage of HPET took away precious calculation power of already CPU bound games therefor hurting 3D performance.

Since then CPUs became more powerful and serveral dedicated CPU cores are the defacto standard nowadays. In addition graphics APIs finally adopted multithreading as well, all in favor of reducing the impact of CPU power in games. Today the bottleneck of 3D performance has shifted from CPU to GPU, that's especially true for high resolutions, AA, post processing and the likes. That's the reason why CPU reviews are benched in FullHD without AA, because otherwise you would barely see a difference.

In the process of this shift the graphics API makers, engine and game developers started to use lots of timestamp queries to measure performance, provide framerate-independent functionality like animations and other movement and so on. And why not, when there is no impact at all.

This is very it gets tricky. When Skylake X and Kaby Lake X was released (too early) in summer 2017 we had the pleasure to review the i7-7740X and i9-7900X. After a few days we came across the same thing that happened to Anandtech recently: the numbers for game benchmarks on lower resolutions didn't add up at all. It took some effort to finally pinpoint the low framerates to an enabled HPET timer. The following video shows our findings and should paint a clear picture of the impact of this issue:

The first encounter of the HPET bug came with Skylake X


We named it the "X299 HPET bug" as the anomaly only occured on CPUs using the X299 chipset back then. Other CPUs were not affected at the time. We contacted Intel and they didn't even bother to comment on this. When approaching an Intel engineer at a press workshop, they even knew about our bug report but denied us to show further proof. Anyway, soon after Coffee Lake S came along it became clear that all new Intel platforms are affected by the bug. We were pretty sure now that this will blow up into Intel's face at some point in the future.

So what is happening here? As we mentioned earlier timers are used heavily these days. The goto timer function for high precision in Windows is called QueryPerformanceCounter(), a WIN32 API function to access the most accurate timer available in the system. QPC is used by almost everybody, although it's known for its problems and inconsistency (but that's another story). The big problem with QPC and its abstraction layer is that the developers don't know what timer will actually be used on the customer's system. Or to put it in a more truthful way: they don't care about it. In normal conditions QPC will use TSC, a very fast timestamp counter inside the CPU. But when HPET is forcefully enabled by the user or an application, QPC will prefer it to TSC. Although the query for an HPET timestamp takes longer, it's more accurate as well. There is nothing wrong with that and there hasn't been for years. Then X299 showed up and the query for an HPET timestamp suddenly takes 7 times longer! The number of possible HPET timer calls per seconds went from 1.4 million on Broadwell-E to merely 200.000 calls on Skylake X. Let's remember that this is a high-end platform also used for scientific purposes where accuracy and performance are both very relevant.

In summary the problem is a very slow timer implementation of the High Precision Event Timer on modern platforms, that is used without care by the developers. Badly affected are Skylake X and Kaby Lake X. Impacts can also be shown on Threadripper, Coffee Lake and in some degree on Ryzen as well. It could be discussed if a slow functionality is a bug, but honestly let's just call it the "HPET bug".

While the reduced theoretical numbers of HPET timer calls are quite self explantory, the impact of the slow HPET can not be directly applied on game performance. It heavily depends on the usage of timer functions in the game/engine and the combination of resolution, details and graphics card in place. So to trigger the bug you normally run your games on something like FullHD, maybe an older, less GPU heavy game as well, and power it with an oversized graphics card. In effect the HPET bug will show on screen with a decreased average framerate and an additional stuttering every now and then. Especially the last part is were the bug really kicks. Due to horribly high frametimes it looks like the game freezes for a few milliseconds. With X299 this stuttering happens on the Windows UI as well. It starts in the final stages of booting with some mild flickering of the loading icon and can be seen in action when dragging windows around or a window/control gets invalidated and is refreshed. Not always but once you see it, it can not be unseen. Bottom line is your expensive system will give an inadequate experience once HPET is enabled.

Because the HPET bug can be difficult to spot, we have implemented a timer benchmark for windows that sheds some light on your timer configuration and its performance. It's called TimerBench and mainly focuses on QPC because it's the defacto standard in Windows. There is a synthetic test to show the maximum number of possible timer calls and a game test to analyze the impact of your configured timer in 3D applications. It uses Unreal Engine 4 and DirectX 11, a famous combination for games.

I tried it but have not tested it with HPET enforced.

 

[frozentundra123456]

on other hand both Results are correct.one is for HPET Off and one is for HPET On.

Yea, I am sure Intel users are going to turn on HPET and reduce performance so they can match AT's "correct"
results. And even worse, they only published one set of the "correct" results. I also believe HPET of off by default. Now if they had published *both* sets of results, that would have been an informative article. As it was, the results were misleading at best. To use an exaggerated car analogy, if one tested a corvette with S rated tires, the top speed would be 120 mph. That result is certainly "correct", but hardly the only or optimal one.

 

[wahdangun]

Well, the overclockers site linked by AT Ryan & Ian has a timer bench program that you can use to test it. They go indepth about the problem.

https://www.overclockers.at/articles/the-hpet-bug-what-it-is-and-what-it-isnt



I tried it but have not tested it with HPET enforced.

But I don't have coffelake cpu or Skylake sp, just haswell and kabylake xeon. And in my testing that cpu is not affected or have minimal impact, so this anandtech testing really opened my eyes, and I'm quite surprised tbh.

 

[wahdangun]

Yea, I am sure Intel users are going to turn on HPET and reduce performance so they can match AT's "correct"
results. And even worse, they only published one set of the "correct" results. I also believe HPET of off by default. Now if they had published *both* sets of results, that would have been an informative article. As it was, the results were misleading at best. To use an exaggerated car analogy, if one tested a corvette with S rated tires, the top speed would be 120 mph. That result is certainly "correct", but hardly the only or optimal one.

But it was still valid use case, you really want to force HPET if you running VM or else your VM will be time drifting.

And programming that's require high accuracy time is a bitch if you don't use HPET.

 

[TheELF]

But it is strange that HPET causes such a performance reduction when forced.

It doesn't the performance stays the same ,HPET is a timer if it tells you that 70 sec have passed instead of 60 sec you will do your division with the time it took to complete the benchmark and get different results but the performance is the same.

 

[William Gaatjes]

It doesn't the performance stays the same ,HPET is a timer if it tells you that 70 sec have passed instead of 60 sec you will do your division with the time it took to complete the benchmark and get different results but the performance is the same.

This is not true :
Because over use the timer causes stuttering, not just merely a calculation drop in frames.
From the overclock site :

So what is happening here? As we mentioned earlier timers are used heavily these days. The goto timer function for high precision in Windows is called QueryPerformanceCounter(), a WIN32 API function to access the most accurate timer available in the system. QPC is used by almost everybody, although it's known for its problems and inconsistency (but that's another story). The big problem with QPC and its abstraction layer is that the developers don't know what timer will actually be used on the customer's system. Or to put it in a more truthful way: they don't care about it. In normal conditions QPC will use TSC, a very fast timestamp counter inside the CPU. But when HPET is forcefully enabled by the user or an application, QPC will prefer it to TSC. Although the query for an HPET timestamp takes longer, it's more accurate as well. There is nothing wrong with that and there hasn't been for years. Then X299 showed up and the query for an HPET timestamp suddenly takes 7 times longer! The number of possible HPET timer calls per seconds went from 1.4 million on Broadwell-E to merely 200.000 calls on Skylake X. Let's remember that this is a high-end platform also used for scientific purposes where accuracy and performance are both very relevant.

In summary the problem is a very slow timer implementation of the High Precision Event Timer on modern platforms, that is used without care by the developers. Badly affected are Skylake X and Kaby Lake X. Impacts can also be shown on Threadripper, Coffee Lake and in some degree on Ryzen as well. It could be discussed if a slow functionality is a bug, but honestly let's just call it the "HPET bug".

While the reduced theoretical numbers of HPET timer calls are quite self explantory,Edit the impact of the slow HPET can not be directly applied on game performance. It heavily depends on the usage of timer functions in the game/engine and the combination of resolution, details and graphics card in place. So to trigger the bug you normally run your games on something like FullHD, maybe an older, less GPU heavy game as well, and power it with an oversized graphics card. In effect the HPET bug will show on screen with a decreased average framerate and an additional stuttering every now and then. Especially the last part is were the bug really kicks. Due to horribly high frametimes it looks like the game freezes for a few milliseconds. With X299 this stuttering happens on the Windows UI as well. It starts in the final stages of booting with some mild flickering of the loading icon and can be seen in action when dragging windows around or a window/control gets invalidated and is refreshed. Not always but once you see it, it can not be unseen. Bottom line is your expensive system will give an inadequate experience once HPET is enabled.

Now using your pc as described is perhaps only relevant for competitive gaming where people play at lowest resolutions and settings.
But i am interested what happens with multiplayer games that fully saturate the cpu.
And i personally am more worried about stuttering than a few percent fps reduction.
The biggest issue is that when the pc system is used for professional use or scientific use that suddenly this shows up, that is just bad.

 

[TheELF]

This is not true :
Because over use the timer causes stuttering, not just merely a calculation drop in frames.
From the overclock site :


Now using your pc as described is perhaps only relevant for competitive gaming where people play at lowest resolutions and settings.
But i am interested what happens with multiplayer games that fully saturate the cpu.
And i personally am more worried about stuttering than a few percent fps reduction.
The biggest issue is that when the pc system is used for professional use or scientific use that suddenly this shows up, that is just bad.

And all of this has what to do with the benchmarks anandtech did?
All that's important for the benchmarks is that a miss measurement of time means a miss measurement of performance.

The fact that console ports have zero means of synchronization and cause all kind of mayhem is a completely different thing.

 

[William Gaatjes]

And all of this has what to do with the benchmarks anandtech did?
All that's important for the benchmarks is that a miss measurement of time means a miss measurement of performance.

The fact that console ports have zero means of synchronization and cause all kind of mayhem is a completely different thing.

Sigh, have you not read the article or something ? The whole reason the AT benchmarks were off and the I7 8700 results were lower was because HPET was forced on causing this bug to appear and resulting in lower framerates than an I7 8700 should have.

I do not know where you get console ports from. I think you are confusing things.

 

[wahdangun]

It doesn't the performance stays the same ,HPET is a timer if it tells you that 70 sec have passed instead of 60 sec you will do your division with the time it took to complete the benchmark and get different results but the performance is the same.

Actually that can happen when HPET is off or not forced, especially when the cpu is stressed or overclock.

That's the decision why Ian forced HPET, because he have background in overclocking, and sometimes the time will drifting if the system push to the limit or overclocked and that will screwed the results.


And what happened in here was just bug in intel that caused performance hit when HPET enabled, and no when you enable HPET the time will not or never will drifting. It's just plain performance hit.

 

[TheELF]

Sigh, have you not read the article or something ? The whole reason the AT benchmarks were off and the I7 8700 results were lower was because HPET was forced on causing this bug to appear and resulting in lower framerates than an I7 8700 should have.

I do not know where you get console ports from. I think you are confusing things.

Yes the article talks about stuttering,the AT benchmarks never mention stutter it's plain graphs with FPS/seconds to complete,I said that the timer skews the results because the time you divide the results with changes and you started with this article that's about stutter...

 

[William Gaatjes]

Yes the article talks about stuttering,the AT benchmarks never mention stutter it's plain graphs with FPS/seconds to complete,I said that the timer skews the results because the time you divide the results with changes and you started with this article that's about stutter...

I come with the article because AT linked that article because they are not the only one who experienced this and overclockers.at went investigating in the past and found the issue.
The timer is accurate. It is the accessing of the timer that is the problem. It takes 7 times longer.
So if you query the timer a lot through the win32 api function QueryPerformanceCounter(), you will notice that you cannot query it as often as you expect.
The way i understand it is that the software calls the function but has to wait for the result to come back, so if you query it often enough, you have a cpu core constantly waiting for the result to return but it takes 7 times longer. If you do that often enough, you might experience stuttering.

If you want to test it with your system, you can download timer bench and see what happens with timer

https://www.overclockers.at/articles/the-hpet-bug-what-it-is-and-what-it-isnt

For background infromation about the windows function :
https://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx

The whole thing is that as long as windows does not need the HPET and is not forced to use the HPET only, you will never encounter this.
But when the HPET is used for everything by the os because windows is forced to use the HPET timer only then the bug becomes noticable on systems afflicted. And that is where all the different results came from.

 

[dahorns]

Yes the article talks about stuttering,the AT benchmarks never mention stutter it's plain graphs with FPS/seconds to complete,I said that the timer skews the results because the time you divide the results with changes and you started with this article that's about stutter...

Seriously, read the Anandtech article, the overclockers article, and really anything from this thread. This isnt about a measurement error. The timer does decrease performance due to increased overhead and drastically increased latency. Games use the timing information to figure out when to do things, and if it takes forever and a day to query the HPET timer you are obvously going to have a performance hit.

 

[Schmide]

This isnt about a measurement error.

Well it could be. If you are using Euler's method for integrating, the slope of the function you are approximating will vary based on, not only then number of points used, but also the discrete steps or resolution of the values.

 

[dahorns]

Well it could be. If you are using Euler's method for integrating, the slope of the function you are approximating will vary based on, not only then number of points used, but also the discrete steps or resolution of the values.

Correction, not only about a measurement error.

 

[Schmide]

As bad as 7 times sounds, it's basically the cost of an atomic.