CPU Hyperthreading and Gaming: Real-World Benchmarks

[Termie]

UPDATE: For more info on this topic, see a follow-up article I wrote here: http://www.techbuyersguru.com/CPUgaming.php

---------------------------------------------

There has been much discussion recently about whether hyperthreading has an effect on games, and therefore whether it's worth paying extra for it in gaming rigs. I had previously done a relatively straightforward test using the Batman: Arkham City built-in benchmark, which showed that 2 Lynnfield cores @3.4/with HT could nearly equal 4 Lynnfield cores @3.0/without HT, and 4 cores plus HT was slightly superior. My results can be found here: http://forums.anandtech.com/showthread.php?t=2255844. This finding was in line with professional reviews which often show the i3-2100 matching older quad-core processors, and tends to prove that HT is a significant asset for dual cores processors (and hence it's worth paying for).

Well, given how many people ask about the benefits of HT on Sandy Bridge and Ivy Bridge quads, I decided to perform a more in-depth analysis with a number of games, spanning several years and genres. This time I'm using a much higher end gaming rig (i7-3770k@4.4, GTX670@1215). My results are below, with four single-player games ordered by age, followed by one multiplayer game (BF3):

To summarize the graph, we can conclude that:
(1) older game engines like Dragon Age and Fallout may not be able to take advantage of more than 4 cores, and appear to respond negatively to hyperthreading.
(2) newer game engines like Deus Ex and Max Payne appear to be able to take advantage of hyperthreading, and respond quite positively, especially in regard to minimums.
(3) Battlefield 3, which many suspect to be CPU-limited in many scenarios, shows very little benefit from hyperthreading. Others running SLI rigs have found that hyperthreading helps, but it appears my rig is essentially GPU-limited.

I'd also propose one theory as to why HT may not be producing significant benefits in all of these tests: a 3770k at 4.4GHz is probably more than sufficient to move the bottleneck to the GPU, even without HT enabled. To test that theory, I compiled the following data on my HTPC (i7-860@3.2, GTX460@850). It shows the very significant benefit of hyperthreading with an older processor, even where the GPU is a limiting factor (i.e., it's running at close to 99% the entire time):

It's fairly clear that on an i7-860@3.2, HT makes a big difference not just in minimums but also averages, and this is most evident in Deus Ex and GTA:IV. It's my opinion, then, that HT is one way to prolong the useful lifespan of a processor - as games advance (and inevitably use more cores), all things being equal HT will give a processor an edge.

Because I tested Deus Ex on both systems and there appeared to be a consistent benefit of HT in the game, I have conducted additional in-depth analysis of the game with 2 cores, 4 cores, with HT and without HT. The dual-core was the i7-860@3.65, which approximates some of the fastest i3 dual-cores on the market today, and the quad-core was again the i7-860@3.2. The number of active cores and the HT state on any i7 can be set within the BIOS.

I ran two identical runs through Hengsha's streets immediately after being dropped off by the transport. Because there was absolutely no variation between any of the sets of two runs, I did not repeat any run for a third time. My results in my opinion are conclusive as to the benefits of HT:

For more information on my methodology, see the next post.

 

[Termie]

Testing on the i7-3770k

I tested each game by actually playing a level, not by running a built-in benchmark. I collected between three and four 60-second FRAPS runs for each game, with the exception of BF3, for which I collected seven runs to account for significant in-game variability. For each game, I attempted to follow the exact same path over the various 60-second runs with HT on and off, although given that I had to restart the computer to conduct the various tests, in some cases I could not precisely duplicate the steps/actions I took previously.

For each game, I used whatever settings I usually play at, always at 1920x1200 resolution. For Dragon Age, Fallout, and Deux Ex, it was maximum, for Max Payne and Battlefield 3, it was maximum but without MSAA.

My data are shown below:

Testing on the i7-860

For the data presented on the i7-860, other than the in-depth Deus Ex analysis, I took a much simpler approach - I did a single four-minute run through a certain map/level, with and without HT. I followed as close to the exact same path each time. Graphics quality settings were at maximum for Deus Ex and Just Cause 2, while GTA:IV had some settings reduced to fit within the VRAM limit of the GTX460-768. These tests were conducted at 1920x1080.

Here is the data for the in-depth analysis of Deus Ex (with the number of active cores and the HT state set within the BIOS), along with a GPU usage screenshot with the i7-860@3.2 with HT:

 

[SPBHM]

good post,
I think the trouble is that most games will use up to 4 threads, and with HT you have 8...

I think the Core i3 shows a lot more how well HT can work, since it only have 4 threads with HT;

 

[Phynaz]

Nice chart!

 

[grimpr]

Hyperthreading is essential on an Intel dualcore, dont miss it for less money spend on an Pentium.

 

[cusideabelincoln]

Hyperthreading would most likely shine with multitasking. Such as streaming your game to Twitch.tv, which would mean the processor needs to compress/encode the video in real time.

 

[tweakboy]

Games don't support HT or am I missing something

 

[Idontcare]

This is very interesting, thanks for taking the time to post it up for everyone's benefit! :thumbsup:

One thing that I have wondered regarding the benefits of HT (or CMT in the case of AMD) is in terms of "other stuff the computer might be doing in the background".

For me personally that would be my antivirus program and antispyware, both of which purport to be performing "realtime background scanning".

Now no two people will have identically configured gaming setups, and most gaming setups that are prepared for benchmarking purposes are stripped/devoid of all these background personal programs. But in real-life they are there, doing their updates like Adobe flash or windows updates, in the background while we are attempting to do other things (like gaming) with our computer in the foreground.

So I'm wondering if in cases where from a purely "gaming centric" perspective we might be led to conclude "older game engines...appear to respond negatively to hyperthreading" but if we load up that computer with the kinds of background processing tasks that we do end up loading up then do those older engines actually benefit from the presence of HT and CMT cores because they aren't competing for CPU cycles with Norton Antivirus or Windows Update service?

 

[grimpr]

One real core goes for serving dpcs,interrupt requests and all those drivers running in the background, you can find more with the help of Latencymon, another serving the background tasks like av's so i guess the more cores the better for the overall responsiveness of real world multitasking usage.

 

[Yuriman]

Games don't support HT or am I missing something

Hyperthreading isn't something you "support". To software, a quadcore CPU with hyperthreading looks like it has 8 cores.

Hyperthreading takes unused execution resources and makes them into a virtual core.

 

[TJCS]

Thanks for the info OP!

Nice to see comparison of actual gaming performance!

 

[Termie]

This is very interesting, thanks for taking the time to post it up for everyone's benefit! :thumbsup:

One thing that I have wondered regarding the benefits of HT (or CMT in the case of AMD) is in terms of "other stuff the computer might be doing in the background".

For me personally that would be my antivirus program and antispyware, both of which purport to be performing "realtime background scanning".

Now no two people will have identically configured gaming setups, and most gaming setups that are prepared for benchmarking purposes are stripped/devoid of all these background personal programs. But in real-life they are there, doing their updates like Adobe flash or windows updates, in the background while we are attempting to do other things (like gaming) with our computer in the foreground.

So I'm wondering if in cases where from a purely "gaming centric" perspective we might be led to conclude "older game engines...appear to respond negatively to hyperthreading" but if we load up that computer with the kinds of background processing tasks that we do end up loading up then do those older engines actually benefit from the presence of HT and CMT cores because they aren't competing for CPU cycles with Norton Antivirus or Windows Update service?

Thanks for the info OP!

Nice to see comparison of actual gaming performance!

Thanks guys. Take a look at the new data I've added from my i7-860@3.2. With this CPU, HT makes a much bigger difference, even with a midrange GPU.

 

[SPBHM]

Games don't support HT or am I missing something

in some cases there is no need for more threads, that's why there is no difference,

quick test I just did here:


big difference,

edit:
tested on another 2 games, the difference was not as big as in F1 2012 (the Codemasters racing games, F1 and Dirt series all seem to love HT); but even with a result as close as this on Witcher, with HT it felt slightly smoother (less stutter)

but the framerate was not great, and the GPU usage was far from the limit; even with 2c/4t

 

[Jovec]

You need to look at core loading with the "HT on" tests. Games that perform worse with HT are probably loading both the physical and shared virtual core rather than loading all of the physical cores first. Or better yet, disable two cores and run 2c/4t HT on versus 4c/4t HT off.

This really has to do with the maturation of multi-threaded rendering. Older engines keep the rendering to a single core for simplicity, and are thus effected negatively by the shared virtual core being loaded too. New engines are able to more effectively deal with the rendering being dispersed across threads, and so are less effected by a given core's type or load. Optimization comes into play too, as better optimized game engines should see less impact from HT than poorly optimized ones.

 

[tweakboy]

Good stuff. Very good thread.

Yes HT is treated like cores but the advantage is 30 to 40 percent avg speed boost in HT apps.

Having 8 threads will be nice for DAW and Video Rendering. But if the software really isn't designed to take full advantage then that 30 to 40 percent can become 15 to 25 percent ,, estimate... gl

 

[toyota]

results are not making sense for GTA 4, Just Cause 2 and Deus Ex. for instance its a fact that GTA 4 only uses 4 cores. and for Just Cause 2, I get the exact same performance using just 2 cores of my cpu as I do for 4 cores even at just 1024x768 so there is no way HT could help a quad core at all. same for Deus Ex because at just 1024x768 only 2 of my cores gave almost same framerates so how could going beyond 4 cores help when its clear that even 4 oced cores are not needed? pretty much any other of the few HT articles show HT will do almost nothing 99.9% of time on quads and if anything can hurt performance by a tiny bit.

 

[Termie]

results are not making sense for GTA 4, Just Cause 2 and Deus Ex. for instance its a fact that GTA 4 only uses 4 cores. and for Just Cause 2, I get the exact same performance using just 2 cores of my cpu as I do for 4 cores even at just 1024x768 so there is no way HT could help a quad core at all. same for Deus Ex because at just 1024x768 only 2 of my cores gave almost same framerates so how could going beyond 4 cores help when its clear that even 4 oced cores are not needed? pretty much any other of the few HT articles show HT will do almost nothing 99.9% of time on quads and if anything can hurt performance by a tiny bit.

Toyota - feel free to post FRAPS data showing your findings. My conclusions above are based on my testing over many games on two different systems. With this much data, I just don't see how you could conclude that HT does not have an effect. If it were one or two games, you could chalk it up to an error in methodology. On 6 games and 2 systems, I think that's harder, especially given that 2 of the games showed the negative HT scaling that many have talked about. Conversely, with Deus Ex:HR, which I ran on both systems, the improvement in minimums was very clear, as was the improvement in averages on the slower system. Perhaps your findings are different because you're conducting a different test - 2 fast physical cores versus 4 fast physical cores, with a slower GPU. Without the option to use HT, you just can't duplicate my methodology.

Also, I'm not interested in data at 1024x768. The theory that CPU-bottlenecks can be identified at that resolution is an old one, and, frankly, I wanted to conduct a different, more relevant test. That's why I tested every game at settings I actually play at.

You point out that the very few articles out there that explore HT have concluded that HT doesn't help at all. I am aware of that, and I don't think the available data points are sufficient. That's why I've conducted this testing, and I invite others to try to duplicate or disprove my findings. Already, user SBPHM above has shown the great effect of HT on a dual-core in F1 2012 and Crysis 2. If HT is so effective on a dual-core, doesn't it stand to reason that it could be effective on a quad-core as well? Just stating as a fact that this or that game "only uses 4 cores" is not a particularly useful data point, because it's hard to track down a source for that statement, and also may not necessarily rule out the possibility that HT allows four cores to be used more effectively.

 

[know of fence]

Maybe varying results like this are a good opportunity to brush up on or learn about uncertainty analysis, mean deviation and variance/standard deviation.
Just looking at the numbers the measured difference appears to be well within error range, the result is, that there is no measurable HT effect. The impact of HT had to be huge to make a difference with that kind of variance.

This suspicion is reinforced by the fact that, when you use more samples in BF3 test, the results move closer still: there is almost no difference between the two means.

 

[Termie]

Maybe varying results like this are a good opportunity to brush up on or learn about uncertainty analysis, mean deviation and variance/standard deviation.
Just looking at the numbers the measured difference appears to be well within error range, the result is, that there is no measurable HT effect. The impact of HT had to be huge to make a difference with that kind of variance.

This suspicion is reinforced by the fact that, when you use more samples in BF3 test, the results move closer still: there is almost no difference between the two means.

A key tenet of statistical analysis is the higher the sample size, the more certain you can be that an observed pattern in the data is occurring outside the bounds of statistical error.

Rather than a long discussion of Statistics 101, however, I'd be interested in other users who are willing to try the same or similar tests in order to increase the sample size. And when's the last time you saw a regression analysis in a video card review, anyway? Is that really your standard for a meaningful benchmark?

 

[know of fence]

The Fraps average already contains >6000 frames measured, still it is wildly inconsistent. Likely because it's hard to eliminate other random variables, like (in ascending order) CPU and GPU frequency scaling, mouse movement, lack of a scripted sequence.

Maybe you could find a single player sequence, and also set a single value for the frequencies, lock sights on a single object and not move the mouse. This may help to narrow down the range of results you are getting. AT for the longest time has relied on built in benchmarks or the Metro 2033 rail cart sequence(?). It would be interesting to see if this improves things, and if you see sigma shrink, that would be an unmistakable sign of improved testing, no regression analysis necessary.

 

[Termie]

The Fraps average already contains >6000 frames measured, still it is wildly inconsistent. Likely because it's hard to eliminate other random variables, like (in ascending order) CPU and GPU frequency scaling, mouse movement, lack of a scripted sequence.

Maybe you could find a single player sequence, and also set a single value for the frequencies, lock sights on a single object and not move the mouse. This may help to narrow down the range of results you are getting. AT for the longest time has relied on built in benchmarks or the Metro 2033 rail cart sequence(?). It would be interesting to see if this improves things, and if you see sigma shrink, that would be an unmistakable sign of improved testing, no regression analysis necessary.

I've used that method (stand in one place and measure fps) to test GPU overclocks. But here it might not work because it would likely eliminate any load on the CPU. I agree there is a lot of variability in my numbers, but that's the nature of real-world benchmarks. The same thing can be seen in any of the video card benchmarks published by HardOCP.

Just to be clear, the multiple runs I conducted for each of the tests on the 3770k were not all identical. I charted four runs in each game and did all four runs with HT on and then HT off. Thus they were not stressing the system identically across runs and a different outcome should be expected. Only in battlefield was there no set of runs I did due to the nature of the game.

I used a less time-intensive approach on the i7-860, for which I simply ran similar four-minute runs at each setting. If I had unlimited time, I would play an entire level of a game at each setting, but this risks losing control of the results, for instance if my character dies or I'm forced to take a different path by the AI.

 

[toyota]

again its a FACT that GTA 4 only uses four threads. I know from testing Deus Ex that going beyond 3 cores did zero and even 2 cores at 4.4 gave nearly the same performance as 4 cores at 4.4. and Toms even said Deus Ex does not go beyond four threads in their review. and I have spent hours testing Just cause 2 and know darn well that it does basically does NOT scale beyond 2 cores at 4.4 even at just 1024x768. your results are screwy.

 

[wand3r3r]

Interesting topic, thanks for the testing thus far. That takes quite a bit of effort. I agree that some more testing could be useful, but at least you have ran several runs to try reduce the margin of error. It's interesting

In the second post I assume the last row of each test is the average. It's the only row with a decimal so I assume that was where you averaged the rows above. (could be noted in that row)

The 860 seems to be pretty maxed out and HT actually makes quite a difference. For the i7 3770k perhaps the GPU is more the bottleneck so the threads aren't fully being utilized? It would be interesting to try this same test with a 4 way SLI or Crossfire to see the increase / decrease due to hyperthreading.

One easy way to test if a game can use more then 4 threads/cores would be to use a 6 core and try 4 vs. 6 without hyperthreading and see if there is a difference.

The results on the i3 sure have a dramatic difference due to hyperthreading.

I really hope the next generation of consoles will have multithreading/cores so hopefully games will start to utilize more cores.

 

[Termie]

Interesting topic, thanks for the testing thus far. That takes quite a bit of effort. I agree that some more testing could be useful, but at least you have ran several runs to try reduce the margin of error. It's interesting

In the second post I assume the last row of each test is the average. It's the only row with a decimal so I assume that was where you averaged the rows above. (could be noted in that row)

The 860 seems to be pretty maxed out and HT actually makes quite a difference. For the i7 3770k perhaps the GPU is more the bottleneck so the threads aren't fully being utilized?

...

I really hope the next generation of consoles will have multithreading/cores so hopefully games will start to utilize more cores.

Yeah, sorry, the last row in my datasets is the average. Since I made it just as my own worksheet, I forgot to label it.

I'm thinking about retesting the games that appeared to have the largest gains from hyperthreading. Being able to repeat the findings would tend to show it wasn't simply testing error. Max Payne is relatively easy to test because it's very linear, so I'll start with that.

And yes, the 860 seems to need the extra help more than the 3770k. Pretty amazing given the old 460 in that system.

 

[Phynaz]

again its a FACT that GTA 4 only uses four threads. I know from testing Deus Ex that going beyond 3 cores did zero and even 2 cores at 4.4 gave nearly the same performance as 4 cores at 4.4. and Toms even said Deus Ex does not go beyond four threads in their review. and I have spent hours testing Just cause 2 and know darn well that it does basically does NOT scale beyond 2 cores at 4.4 even at just 1024x768. your results are screwy.

How about drivers?