Efficiency investigations - misleading and wrong?

[boxleitnerb]

Basically every review out there is badly failing when it comes to analyzing efficiency. Here in VC&G, TPUs results are often consulted, and everytime I see that, I'm baffled how such a flawed methodology can be referenced over and over again.

What I'm talking about is mixing one (!) power consumption measurement with a whole plethora of performance numbers (19 games/benchmarks in their latest 7870 LE review). How is that not wrong?

Most sites have a specific game where they test power consumption. And of course, that may favor AMD or Nvidia. It may by chance actually be a good fit for a wider range of typical gaming scenarios, but it is more likely that it is not.

I think it would be better to either do this thing right or don't do these single measurements and perf/W ratings at all, because they are wrong and misleading. Even if you were to make a perf/W rating for the specific game you used to measure power consumption, that would still be only valid for this one game - hardly ideal.

What I like is the way 3DCenter goes about it. They make a performance index based on multiple reviews. Then they make an average power consumption value based on multiple reviews where only the card was measured. Usually that includes at least 6 games/benchmarks (Anno 2070, BF3, Crysis 2, BFBC2, Tom Clancy HawX, Canyon Flight 3DMark06).
Finally, they correlate both values. It's not ideal, but 6 data points are better than only one.
http://translate.google.de/translat...tung-des-grafikkarten-stromverbrauchs&act=url

What is your take on this?

 

[Ketchup]

Maybe because power can't be optimized like a driver can.

 

[Dark Shroud]

Well AMD has PowerTune which can be adjusted by the user. Nvidia has some variation that auto throttles when their 600 series cards hit 70C.

 

[Lonyo]

It is very true, usually sites pick the hardest game on a card and use that to show power (e.g. AT).
Applying that to work out general efficiency is pretty stupid, much like looking at absolute frame rates and dividing to get an average would be stupid (e.g. take Crysis and HAWX2 since those are at opposite ends, one card gets 30/200, another 25/300, that makes an average of 115 vs 162.5. Which card would you rather have?).

That's not to say it would mean B is suddenly more efficient than A, but it is pretty pointless to take outliers and then use that to establish overall power efficiency by applying it to entirely unrelated games with entirely different performance characteristics.

 

[Termie]

HardOCP sometimes provides measurements in multiple games, as in this HD7970 review: http://hardocp.com/article/2012/10/30/xfx_double_d_hd_7970_ghz_edition_video_card_review/10

Notice that the "more efficient" GTX680 (according to many reviews) actually uses more power in just about every game except the most demanding game, BF3 MP.

 

[Ketchup]

HardOCP sometimes provides measurements in multiple games, as in this HD7970 review: http://hardocp.com/article/2012/10/30/xfx_double_d_hd_7970_ghz_edition_video_card_review/10

Notice that the "more efficient" GTX680 (according to many reviews) actually uses more power in just about every game except the most demanding game, BF3 MP.

So in order to read this graph as it appears, wouldn't you have to believe that CPU utilization numbers are the same for each card used?

 

[Termie]

So in order to read this graph as it appears, wouldn't you have to believe that CPU utilization numbers are the same for each card used?

Well, I think you can assume as a baseline argument that CPU utilization should be the same for any given game at a given level of performance, and if it's not, then that is a legitimate distinction to make between cards. For instance, if one card requires a higher CPU overhead than the other, just looking at GPU power usage isn't accurate.

In actuality, a higher-performance card almost always leads to higher system power draw compared to a lower-performance card drawing the same watts, due to higher CPU utilization, even though it is more efficient per frame rendered.

Ultimately, which power numbers are relevant depends on what you're trying to gauge - the efficiency of the card or the power use of the system while gaming.

 

[Ketchup]

Well, I think you can assume as a baseline argument that CPU utilization should be the same for any given game at a given level of performance, and if it's not, then that is a legitimate distinction to make between cards. For instance, if one card requires a higher CPU overhead than the other, just looking at GPU power usage isn't accurate.

In actuality, a higher-performance card almost always leads to higher system power draw compared to a lower-performance card drawing the same watts, due to higher CPU utilization, even though it is more efficient per frame rendered.

Ultimately, which power numbers are relevant depends on what you're trying to gauge - the efficiency of the card or the power use of the system while gaming.

It seems like gaming while worrying about power usage contradict each other, to a point. 1 baseline number is plenty for me. Using a couple more watts for a better card for my favorite game is not a problem IMO.

OP, are you trying to say that Anandtech is somehow using this number to lead to a 'this manufacturer is better' deal?

 

[boxleitnerb]

I wasn't specifically referring to Anandtech, rather TPU because they provide a skewed perf/W rating. If a site like AT just puts a single measurement out there like AT and most sites do, it's not optimal but not as bad as a skewed rating either.

Especially the Crysis 2 measurement from TPU is misleading because that is a game where many AMD/NV pairs are close together (or even reversed) while in other games they are not.

For example (from the link in the OP):
GTX680 vs HD7970: 166W vs. 163W. Actually 3W more for the 680.
But on average it is 169W vs 189W, so 20W less for the 680.

GTX670 vs HD7950: 144W vs 126W. 18W more for the 670
Average: 143W vs 147W, 4W less for the 670.

This goes on and on. Crysis 2 is not representative. In almost every other of these reviews (hardware.fr, heise, PCGH, ht4u) there are significantly different results. What TPU is measuring is almost always an outlier and not the norm.