Just thought I'd make a poll here. If I have a processor, and test it in a scenario where I know the processor is not the limiting factor (e.g. benchmark processors based on USB 2.0 performance, or network performance, or video card limited performance) have I provided any meaningful insight in to the performance of that processor, or its capabilities in comparison to another processor?
Are CPU tests with non-CPU bottlenecks good indicators of CPU performance?
- Thread starter Ferzerp
- Start date
Generally no, but there's hardly any software which is completely unaffected by the CPU performance. All of the examples you're giving require the CPU to do some work in between requests to other hardware. Some of it can be done in a parallel thread, and some can't.
So while it's not the best test of the CPU's capabilities, it can still give a valuable indication of the importance of picking one CPU over another in real-world scenarios. Also note that many 'synthetic' benchmarks which do focus entirely on the CPU are often not entirely representative of real-world performance.
So any way you look at it, a benchmark is just an measure of how a device perform under the exact circumstances created by that benchmark. Nothing more, nothing less. You always have to be careful when trying to extrapolate those results to a different use case.
So while it's not the best test of the CPU's capabilities, it can still give a valuable indication of the importance of picking one CPU over another in real-world scenarios. Also note that many 'synthetic' benchmarks which do focus entirely on the CPU are often not entirely representative of real-world performance.
So any way you look at it, a benchmark is just an measure of how a device perform under the exact circumstances created by that benchmark. Nothing more, nothing less. You always have to be careful when trying to extrapolate those results to a different use case.
No. It blows my mind that on every review that does contains CPU performance gaming benchmarks, a small angry mob forms wanting the game tested at such and such settings.
Unless there is a giant platform bottleneck, benchmarking Crysis at 2650x1600 max settings doesn't actually show us anymore useful information than 1024x726. I'll go ahead and bring the car hyperboles into full swing on Post #3.
No one buys a Lamborghini Aventador because it can follow a semi truck up a steep hill. No one is going to buy a Ducati Panigale so they can follow the speed limit at all times. Remove the bottleneck so we can see what the performance is like when it matters. Lets compare two graphs from the recent Ivy preview:
Imagine if we were only allowed to see the second one. A lot of people would get the idea that a Llano is basically close enough to an Ivy for it to not matter. Yes, they are both overkill for DIRT 3 for most people, but in the future that Ivy is going to continue to provide enough CPU power for games to come.
I know of at least one person on this forum who bought a PII x3 around the same time as I bought a Nehalem based on the fact that they were basically throwing up the same performance on Fallout 3 with a 8800 GT. As the years past, he upgraded to a x4 and again to a x6. I'm no expert, but I'm pretty sure Nehalem is still faster on CPU heavy games.
Unless there is a giant platform bottleneck, benchmarking Crysis at 2650x1600 max settings doesn't actually show us anymore useful information than 1024x726. I'll go ahead and bring the car hyperboles into full swing on Post #3.
No one buys a Lamborghini Aventador because it can follow a semi truck up a steep hill. No one is going to buy a Ducati Panigale so they can follow the speed limit at all times. Remove the bottleneck so we can see what the performance is like when it matters. Lets compare two graphs from the recent Ivy preview:
Imagine if we were only allowed to see the second one. A lot of people would get the idea that a Llano is basically close enough to an Ivy for it to not matter. Yes, they are both overkill for DIRT 3 for most people, but in the future that Ivy is going to continue to provide enough CPU power for games to come.
I know of at least one person on this forum who bought a PII x3 around the same time as I bought a Nehalem based on the fact that they were basically throwing up the same performance on Fallout 3 with a 8800 GT. As the years past, he upgraded to a x4 and again to a x6. I'm no expert, but I'm pretty sure Nehalem is still faster on CPU heavy games.
Last edited:
Are CPU tests with non-CPU bottlenecks good indicators of CPU performance?
Of course not, but I think I know why you made the thread.
The thing is, you are asking the wrong question.
If the common usage of the computer is going to be limited by a non-CPU part, regardless of CPU, then is there any real benefit to buying a more expensive "faster" CPU?
I argue that the answer is "no", while it seems that you are trying to imply that the answer is yes.
Also, the point of a benchmark isn't only to show the difference, it's also to show the lack of differences. If a $350 3770 and a $90 FX-4100 perform within 5% of each other at some particular task, it might be useful information to know, don't you think?
Also, there is the fact that these hypothetical tests you talk about are not 100% consistent and never-changing. A game that is GPU bound with a cheap or bargain priced GPU might be CPU limited with a 7970 crossfire setup. This is useful information, and in order to know exactly where the "GPU limited" part starts, you need to do some GPU limited tests as well as some that show the benchmark scaling with CPU performance.
Final answer: no they are not good measure of CPU performance, but for a complete and accurate review they are absolutely required.
Only problem is that "a Llano is basically close enough to an Ivy for it to not matter" is completely true and accurate in this case. There is nothing wrong with people getting that idea, because it is the truth.
If you want to complain, complain about the game being tested, but the graphic settings in the first test are just completely unrealistic and stupid. Unless you have some superhuman capability (and some as yet unreleased monitor technology) it's impossible to perceive the 180 fps you could hypothetically get by playing on the fastest CPU.
And then you talk about future games, but if you watched the historical progress of gaming at all I think you would find the future newer games require more GPU resources, not less, and will just be more GPU limited than current games, making the CPU even less relevant.
Of course not, but I think I know why you made the thread.
The thing is, you are asking the wrong question.
If the common usage of the computer is going to be limited by a non-CPU part, regardless of CPU, then is there any real benefit to buying a more expensive "faster" CPU?
I argue that the answer is "no", while it seems that you are trying to imply that the answer is yes.
Also, the point of a benchmark isn't only to show the difference, it's also to show the lack of differences. If a $350 3770 and a $90 FX-4100 perform within 5% of each other at some particular task, it might be useful information to know, don't you think?
Also, there is the fact that these hypothetical tests you talk about are not 100% consistent and never-changing. A game that is GPU bound with a cheap or bargain priced GPU might be CPU limited with a 7970 crossfire setup. This is useful information, and in order to know exactly where the "GPU limited" part starts, you need to do some GPU limited tests as well as some that show the benchmark scaling with CPU performance.
Final answer: no they are not good measure of CPU performance, but for a complete and accurate review they are absolutely required.
Only problem is that "a Llano is basically close enough to an Ivy for it to not matter" is completely true and accurate in this case. There is nothing wrong with people getting that idea, because it is the truth.
If you want to complain, complain about the game being tested, but the graphic settings in the first test are just completely unrealistic and stupid. Unless you have some superhuman capability (and some as yet unreleased monitor technology) it's impossible to perceive the 180 fps you could hypothetically get by playing on the fastest CPU.
And then you talk about future games, but if you watched the historical progress of gaming at all I think you would find the future newer games require more GPU resources, not less, and will just be more GPU limited than current games, making the CPU even less relevant.
Last edited:
No.
Testing a CPU is a difficult thing. Most sites do it wrong, Anandtech is not optimal either. For example they test with low quality without knowing if this would lessen CPU load, too.
Rules for a good CPU test
Testing a CPU is a difficult thing. Most sites do it wrong, Anandtech is not optimal either. For example they test with low quality without knowing if this would lessen CPU load, too.
Rules for a good CPU test
- Use your own savegames. No timesdemos or automated benchmarks
- Use a scenario where there is actually something going on. Example Far Cry 2: Not the "ranch" benchmark but the "action" benchmark.
- No low quality settings (unless you have the time to test each and every one of them for CPU/GPU performance impact)
- Make sure the results are reproducible
- Use either low resolution or the fastest card in SLI/CF
- If SLI/CF in 3. make sure they scale perfectly
- Don't make the test scene too long. 15 seconds max, as you don't know if you're going into GPU bound territory along the way. Also reproducibility is better with shorter benchmarks
Last edited:
It depends on what part of the CPU you are testing--if you are assuming a CPU black box and want to test raw ALU or FPU performance, then the answer is yes, a non CPU-limited benchmark may not be the correct benchmark for what you are testing.
On the other hand, if you are testing elements of the CPU outside of the computational core of the CPU (such as memory bandwidth/on chip memory controller performance, or perhaps thread/context switching performance, or power management features/power consumption, or real PCIe lane speed/bandwidth) to get an overall understanding of CPU "performance" (i.e. characterization) then you may choose a non-CPU limited benchmark which fully stresses the areas of the CPU you are interested in characterizing.
For example: I want to test overall TDP of my processor (~how much power does the chip consume at load) with Hyper-threading on vs off. I could choose any benchmark that stresses the CPU some non-trivial amount and still be able to compare the delta between running it with Hyper-threading vs without. It may not be completely floored on the CPU, but completion time isn't the only thing we're looking at here. We're also measuring the amount of power consumed.
I'd say it depends on what you mean by "good indicators" of CPU performance.
Yes. It might not be a good measure of the cpu itself, but considering the system as a whole it's a valid test. If I was upgrading a CPU in a gaming machine, I'd like to know whether or not I'd see a noticeable difference at 1920x1200 high settings, as opposed to the 1024x768 low settings that I never use.
Puppies04
Diamond Member
- Apr 25, 2011
- 5,909
- 17
- 76
But if you only want to play at the settings shown in the graph it doesn't matter..... of course digging a little deeper will reveal that when the llano gets sent to the giant HTPC in the sky the 3770k will still be gaming happlily but not everyone has this forethought and will happily repost the same graph over and over to prove the 3770k owner got ripped off.
The question was "have I provided any meaningful insight in to the performance of that processor, or its capabilities in comparison to another processor", but your stance is "yes" but also "It might not be a good measure of the cpu itself"
How does this work exactly?
It works by considering how a processor performs a given task in a given system. Typical benchmarks isolate the cpu to show how much one is better than the the other in a hypothetical situation, and don't necessarily translate into any meaningful real-world performance improvement.
That's why the good review sites will test at low res and extremely high res with powerful multi-GPU setups. If every chip appears to be roughly the same in gaming then you know you're doing it wrong :biggrin:
They are (generally) equal at GPU bottle-necked 1080p resolutions, but that quickly changes if you get a more powerful GPU or decide to do multi-GPU setups. Though most people won't spend that much money on GPU horsepower, having a more powerful CPU that can drive future GPUs will allow you to skip over a complete platform overhaul and just spend money where you need to. This doesn't necessarily imply that everyone should buy SB-Es, but one should definitely do their homework when it comes to buying CPUs with the intent of gaming.
Example of poor benchmarks
x1600 with a single GPU, Tom's? Really?
How it should be done.
Last edited:
So you feel that the first two images that pelov posted are giving meaningful comparisons of the processors on that list?
I am asking about processors. Not whole systems. I am interested in processors here.
Yes it does. It tells me that for Crysys 2, the latest and greatest Core i7 is no better than a Phenom X6.
Really? So no matter what video cards we may ever use, i7-2600k = i7-920 = 1100T = 8150?
That sounds like what you're trying to tell me, and if you believe that the graphs are meaningful *for cpu performance*, please explain how you can reconcile that with any other answer if that isn't what you are telling me.
That sounds like what you're trying to tell me, and if you believe that the graphs are meaningful *for cpu performance*, please explain how you can reconcile that with any other answer if that isn't what you are telling me.
That's why you need to consider the whole system, and not just cpu. If you have a tri-sli setup, then look at the tri-sli graphs. For people like myself using a single gpu, how a cpu performs in tri-sli is not important. What is important is knowing whether a more expensive cpu will give me a tangible improvement using my video card and my gaming resolution.
Oh for god's sake, no!. Just after I got done posting what I did you had to go and misinterpret. So here we go again...
http://www.guru3d.com/article/crysis-2-dx11-vga-and-cpu-performance-benchmarks/5
http://www.guru3d.com/article/crysis-2-dx11-vga-and-cpu-performance-benchmarks/6
Notice there's a difference?
Then there's this.
DX10
DX11
DX9 is even worse for BD due to the lack of threading.
None of this you would really see by doing your testing at GPU-capped scenarios. They would, for the most part, look roughly the same with Bulldozer doing mostly worse than most Intel chips and sometimes worse than Deneb/Thubans. In some cases it might slip ahead at those same resolutions than even a 2500K. Does it mean that it's a better chip for that particular game? No it doesn't. The point is you need both the low and the highest res in order to eliminate the GPU as the culprit of any funky results you'll be seeing, as this is a cpu benchmark not a GPU one. Tom's, for all their good articles (their recent one about Direct Compute and openCL is worth a look) has also reminded us time and time again why they've slipped for most readers. If you see Crysis 2 showing no difference in FPS between across many CPUs then you need to either lower the resolution or add another more powerful GPU and preferably both. This takes more time and access to more hardware, but instead of putting in the extra work to show that there is in fact a difference, they upped the resolution to see if that would change anything without touching the GPUs used. And, to no one's surprise, the scores stayed equal at x1600. No sh.. :thumbsdown:
You appear unwilling to answer the questions I am asking.
Are you aware that no cpu will ever get more than 41-42fps in that exact test? The best processor that ever is made with a chipset supporting PCIe x16 lanes (so that the same video card may be used) will score the exact same. The CPU could be 100 times faster in all metrics that were cpu bottlenecked, but it will still get 41-42 fps here.
If we change the labels on those graphs to data throughput with a certain USB drive, do you still feel the graph is a meaningful cpu test? If not, what makes a gpu bottlenecked test a valid comparison, but a USB drive bottlenecked comparison is not?
Are you aware that no cpu will ever get more than 41-42fps in that exact test? The best processor that ever is made with a chipset supporting PCIe x16 lanes (so that the same video card may be used) will score the exact same. The CPU could be 100 times faster in all metrics that were cpu bottlenecked, but it will still get 41-42 fps here.
If we change the labels on those graphs to data throughput with a certain USB drive, do you still feel the graph is a meaningful cpu test? If not, what makes a gpu bottlenecked test a valid comparison, but a USB drive bottlenecked comparison is not?
Last edited:
You're asking a loaded question, that's the problem. And the answer remains the same - if my priority was USB throughput, and I saw that a $1000 CPU is giving me the same results as a $100 cpu, then I would arrive at the same conclusion - that I don't need a faster CPU.
Are you aware that no cpu will ever get more than 41-42fps in that exact test? The best processor that ever is made with a chipset supporting PCIe x16 lanes (so that the same video card may be used) will score the exact same. The CPU could be 100 times faster in all metrics that were cpu bottlenecked, but it will still get 41-42 fps here.
i7-2600k = i7-920 = 1100T = 8150? The graph says it is true. Is it true?
Yes, I am reposting questions because I feel you are avoiding answering them.
i7-2600k = i7-920 = 1100T = 8150? The graph says it is true. Is it true?
Yes, I am reposting questions because I feel you are avoiding answering them.
Oh for god's sake, no!. Just after I got done posting what I did you had to go and misinterpret. So here we go again...
http://www.guru3d.com/article/crysis-2-dx11-vga-and-cpu-performance-benchmarks/5
http://www.guru3d.com/article/crysis-2-dx11-vga-and-cpu-performance-benchmarks/6
Notice there's a difference?
Then there's this.
DX10
DX11
DX9 is even worse for BD due to the lack of threading.
None of this you would really see by doing your testing at GPU-capped scenarios. They would, for the most part, look roughly the same with Bulldozer doing mostly worse than most Intel chips and sometimes worse than Deneb/Thubans. In some cases it might slip ahead at those same resolutions than even a 2500K. Does it mean that it's a better chip for that particular game? No it doesn't. The point is you need both the low and the highest res in order to eliminate the GPU as the culprit of any funky results you'll be seeing, as this is a cpu benchmark not a GPU one. Tom's, for all their good articles (their recent one about Direct Compute and openCL is worth a look) has also reminded us time and time again why they've slipped for most readers. If you see Crysis 2 showing no difference in FPS between across many CPUs then you need to either lower the resolution or add another more powerful GPU and preferably both. This takes more time and access to more hardware, but instead of putting in the extra work to show that there is in fact a difference, they upped the resolution to see if that would change anything without touching the GPUs used. And, to no one's surprise, the scores stayed equal at x1600. No sh.. :thumbsdown:
So tell me, are you going to add another gpu or lower the resolution if you pop in a shiny new cpu and see no difference? In other words, what relevance does this method have to real life applications? All the effort to show how one cpu is better than the other are irrelevant unless they are directly applicable to the user. For a user with a single gpu, it's important to know whether or not a more expensive cpu will benefit HIM, not some theoretical tri-SLI setup which he has no plans of buying.
Seems to me like you're interested in discussing the theoretical applications of cpu performance, as opposed to the practical applications.
You realize, if your position is well reasoned and consistent, you won't need to avoid answering any direct questions regarding it.
If you fear that it isn't though, I can see why you would be loathe to answer.
If you fear that it isn't though, I can see why you would be loathe to answer.
First, don't jump to conclusions so fast. It wouldn't be the first time a test was thought to be gpu-limited, and then a brand spanking new cpu blew right past the limit.
Second, I already answered your question - for a consumer buying cpu's with his own money, it's important to know if he will see no improvement in his usage scenario. Even if the test is completely gpu-bound, that piece of information is important when making purchasing decisions.
Because unless your CPU is very old (to the point where nobody puts it on modern benchmarks) or you've got a new-ish CPU and play at 1080p they are all roughly the same (minus Blizzard titles which tend to heavily favor Intel's architecture).
When 10 CPUs are all equal at 1080p gaming does that make them equal gaming processors? Take a look at those benchmarks again and tell me with a straight face that the 2500K is just as good as a Bulldozer :\
Also, you're under the assumption that multi-GPU gaming is somehow not a "real life scenario." Or that people don't and won't ever play on anything more than a single 1080p monitor. Clearly they will if both nVidia and AMD have offered multi-GPU and multi-monitor gaming. Furthermore, why restrict your options? want to get a large IPS monitor? Sorry, bud, but you'll never get 60FPS with that Bulldozer CPU despite all of those worthless benchmarks showing you that there was no difference at 1080p.
This isn't to say that people should buy the best possible gaming CPU. We're way past that era where the CPU was a big factor. Nowadays a GPU matters far more than it used to and you see that at any 1080p single-GPU benchmark. If you're going to play at 1080p then just get a well rounded system that will cost you less money and offer great performance. If you're looking for something that will last you longer and more room to stretch your legs without dumping your platform then the benchmarks above provide a perfect example of why "equal at ..." isn't equal.
No, my questions are :
i7-2600k = i7-920 = 1100T = 8150? The graph says it is true. Is it true? (This is a true or false question)
Are you aware that no cpu will ever get more than 41-42fps in that exact test? The best processor that ever is made with a chipset supporting PCIe x16 lanes (so that the same video card may be used) will score the exact same. The CPU could be 100 times faster in all metrics that were cpu bottlenecked, but it will still get 41-42 fps here.
If you dispute the validity of the question, please cite an example. Otherwise, let us clarify here. Is it your position that a cpu with the same video card can score higher than 41-42?
i7-2600k = i7-920 = 1100T = 8150? The graph says it is true. Is it true? (This is a true or false question)
Are you aware that no cpu will ever get more than 41-42fps in that exact test? The best processor that ever is made with a chipset supporting PCIe x16 lanes (so that the same video card may be used) will score the exact same. The CPU could be 100 times faster in all metrics that were cpu bottlenecked, but it will still get 41-42 fps here.
If you dispute the validity of the question, please cite an example. Otherwise, let us clarify here. Is it your position that a cpu with the same video card can score higher than 41-42?
TRENDING THREADS
-
Discussion Zen 5 Speculation (EPYC Turin and Strix Point/Granite Ridge - Ryzen 9000)- Started by DisEnchantment
- Replies: 24K
-
Discussion Intel Meteor, Arrow, Lunar & Panther Lakes Discussion Threads
- Started by Tigerick
- Replies: 19K
-
-
-
Facebook
Twitter