Spin Off: AT's Testing Methods & Uber Mode

[Teizo]

As others have pointed out, just do default vs default out of the box for the comparative review, and if a certain card has a feature than enables more performance, take a few games and run it by itself in that particular mode with the caveat it is not default and why this part is listed separately, or perhaps even in a separate review, then list the trade offs of noise/power/heat.

For instance, in 'Quiet Mode' the 290X delivers performance on par or better than the 780...and that is what can be expected straight out of the box. Then, if someone is willing to put up with the extra noise/power demands...they can enable 'Uber Mode' which will give performance on par with the Titan and 780 OC models.

I do believe no full review should leave any feature of a card untested....but it is also important that cards are tested on a level playing field to offer the best comparative results . There is a way to do both I think.

Also, as far as opinion vs objective 'reviews'. Well, reviewers have opinions. They are supposed to, and their opinions are important. Otherwise AMD or Nvidia would have no use for them and would just put the scores up on their website and let that be it. So, people getting on Ryan for supposedly not being objective in his review per their opinion I think are being unfair. One of the reasons you are here reading his review (among a few others) is because you value his (their) 'opinion' and feel as though his (their) opinion is reputable. As thus, that also means he is not obliged to always agree with you, or you him. Otherwise he would just be a puppet, so to speak...which is no good.

 

[Subyman]

It is a philosophical debate if you don't test them like the consumer would use them. You can argue about the "purest" policy or the most "fair", but at the end of the day the review is for the benefit of the end user. For that reason, test each card at its maximum advertised potential and then discuss any negatives associated with running the card at its advertised maximum speed.

If the maximum advertised potential is different from the out-of-the-box setting, then test both and write an analysis with both settings in mind. Again, an "out-of-the-box" only review policy is very easy to strictly adhere to and very, very consistent, but if it doesn't align with how the actual product will be used by the consumer, then it is all academic.

 

[Wall Street]

Anand,

Thank you for your input. Any thoughts about changing how you display and describe the data to best illustrate the tradeoffs? For example, a scatterplot could be used to show performance vs. noise (don't know if sone or dB should be used here) with another used to show performance vs. price.

I love the thoroughness of Anandtech's reviews. However, sometimes I feel like the stodgy old FPS bar chart presentation is a little behind the times. If you look at the HardOCP frame rate histograms, the PCPer frame percentile charts or the Techreport price vs. performance scatterplots you can interesting new ways to show these tradeoffs. In each of these cases had presented data in a new way that lets the consumer decide what is important. I don't think anyone has made a really good visualization of the noise vs. price vs. performance tradeoff.

The GTX 770 4 GB, the R9 290 4 GB and the GTX 780 3 GB are all competing cards. One is slow, one is loud and one is $100 more expensive. There should be a good way to visualize this.

In the case of the 290x, the reader could then make their own decision on uber vs. quiet. On future cards, rediculous perforance settings would stick out like a sore thumb on a scatter plot instead of dominating the review outside of the one chart on the temps power and noise page.

Finally, I personally think that the reviews have more than enough info and wouldn't think it would be a bad idea to drop to ~5 games if it makes to for more analysis and interesting charts.

 

[Shmee]

For the record, my card came with the Uber mode bios set. It was a powercolor, and it seemed to have higher out of the box clock speeds too. I haven't tried OCing it yet, just adjusted fan max and thermal target.

 

[3DVagabond]

For the record, my card came with the Uber mode bios set. It was a powercolor, and it seemed to have higher out of the box clock speeds too. I haven't tried OCing it yet, just adjusted fan max and thermal target.

Wouldn't it be interesting if only the review samples were sent out with the bios set to quiet mode.

 

[Jaydip]

I believe you should only bother with Uber mode and at the same time maximize the temperature and power settings for all the NV cards.Win Win for all.

 

[Unoid]

I for one am slightly disappointing in ANANDTECH. I come here many hours a day (mostly forums) to learn about the best and newest computer hardware tech.

If I read a review and they purposely use 290x results that are of a special quiet mode vs the high performance Uber mode to compare to another new nvidia card; then I'm being shown apples and oranges.

We come here because we're hardware enthusiasts. Not scrubs with super sensitive hearing aids. I run two 680GTX's in SLI, the sound sis LOUD! but I want the best performance.

Passing off 290/x's with quiet mode vs a 780ti only tells me that AT wants to show the Nvidia in upset glory, and with AMD at worst.

Why the heck do I visit this site if you pull this crap on us?

I run nvidia and own a nvidia SHIELD> But you show bias to Nvidia. I'm disappointing. I also enjoy AMD for their competitive products.

the 290 is the BEST bang for buck enthusiast product. Please argue different because you cna't unless you make 6 digitas a year as a single man with no hobbies.


At least HardOCP isn't letting me down also!

 

[VulgarDisplay]

Stop him from what? Talking about something that you don't want discussed? AFAICT, I've seen him explain why what he is talking about is on topic. So, stop him from what? You telling him to knock off the "off topic" is more "off topic" than him discussing it. Put the "off topic" card away.

It is off topic though, and it's been talked about to death. EVERY one of us agrees the cooler needs to be better. That doesn't have anything to do with Uber mode results being used in reviews.

Absolutely nothing. Stop trying to make something out of nothing with everyone telling blackened to pick something new to talk about. It's all he's done this last week and he's usually one of the less annoying posters here.

Sounds like lonbjerg talking about physx at this point.

 

[BallaTheFeared]

I think default, or the quite mode should be compared to whatever is similar.

That said it's hard since AMD is louder in quite than Nvidia, and since both are playing by temps it gives them an advantage and they know this.

Uber mode is just an overclocked card and should be compared with similar moderate overclocks. Again this is hard because Nvidia even at the same fan speed is quieter, and doesn't have the sound issues AMD does. The more we try to hone these cards to be similar the more dissimilar they seem.

But this does nothing to address going forward, what if the GTX 780 Ti shipped with default and 100% fan and then we'd be having a discussion about why it is 30% faster than Uber 290x because it is hotter/louder...

You need to consider who is saying what, and what (who) they value because the choice you make today will only become the wrong choice when that choice proves to negativity affect one company or the other..


Personally I think the only way to solve this is to test default, then test manual OC vs manual OC. But then you're looking at silicon lottery issues, binning, and cherry picked review samples (though I'm sure we see some of that already).

Whats the right move? Do you adjust fan speed so both exhibit similar db? What about the difference in the actual noise? Where do we start, where do we stop? Should we do it based on power used per game? The only thing I'm sure of is that for weeks after 290x released nobody here (probably most still can't) tell you what the actual out of box performance of the 290x is with any certainty because of the sample variation and some throttling, others not, some doing cold tests, some not.... All 290x has really done is made reviews of stock out of box performance nearly worthless because of the huge variation the temp/fan speed based performance 290x exhibits. Which in turn only makes reviews and reviewers pointless until we start to see the cards in the wild, in user hands.

 

[Silverforce11]

Balla, you do realize the thermal and fan setting on Uber mode just isn't a simple "OC".. it's not the ASUM switch.

ALL IT IS is a higher fan speed cap, thats it. It's covered with warranty, its encouraged for people to use it IF THEY DO NOT MIND THE NOISE and its recommended CF setup be set on Uber.

As a review site, you test as much as possible to gather data and present facts. You can make subjective calls, its too loud, I agree its too loud. Doesn't mean jack, because many gamers isolate the case away and game with headphones. To them, what exactly is the issue with gaming in Uber mode?

OC is pushing the clocks ABOVE the manufacturer's settings. Quiet and Uber are AMD's settings.

As for moving forward, who is going to make multiple bios switches for multiple thermal/noise threshold? Really its simpler to do extra modes in software. AMD is using an obsolete approach.

 

[BallaTheFeared]

When fan speed and therefore thermal headroom dictate the clock speed adjusting fan speed is the same thing as increasing the core clock on a card that isn't limited by the fan speed in the first place.

These gamers can flip a switch on their card to enable a higher fan speed profile but can't do similar with a slider bar in afterburner? Seems people want to make out PC gamers as idiots, but give them the benefit of the doubt when it happens to benefit one company or another.

 

[Jaydip]

I think default, or the quite mode should be compared to whatever is similar.

That said it's hard since AMD is louder in quite than Nvidia, and since both are playing by temps it gives them an advantage and they know this.

Uber mode is just an overclocked card and should be compared with similar moderate overclocks. Again this is hard because Nvidia even at the same fan speed is quieter, and doesn't have the sound issues AMD does. The more we try to hone these cards to be similar the more dissimilar they seem.

But this does nothing to address going forward, what if the GTX 780 Ti shipped with default and 100% fan and then we'd be having a discussion about why it is 30% faster than Uber 290x because it is hotter/louder...

You need to consider who is saying what, and what (who) they value because the choice you make today will only become the wrong choice when that choice proves to negativity affect one company or the other..


Personally I think the only way to solve this is to test default, then test manual OC vs manual OC. But then you're looking at silicon lottery issues, binning, and cherry picked review samples (though I'm sure we see some of that already).

Whats the right move? Do you adjust fan speed so both exhibit similar db? What about the difference in the actual noise? Where do we start, where do we stop? Should we do it based on power used per game? The only thing I'm sure of is that for weeks after 290x released nobody here (probably most still can't) tell you what the actual out of box performance of the 290x is with any certainty because of the sample variation and some throttling, others not, some doing cold tests, some not.... All 290x has really done is made reviews of stock out of box performance nearly worthless because of the huge variation the temp/fan speed based performance 290x exhibits. Which in turn only makes reviews and reviewers pointless until we start to see the cards in the wild, in user hands.

+1 very well said.As I said earlier we are dealing with too much nonsense since boost came to GPU.Both of the vendors are trying to play along while being handily outclassed by Intel.

 

[Silverforce11]

When fan speed and therefore thermal headroom dictate the clock speed adjusting fan speed is the same thing as increasing the core clock on a card that isn't limited by the fan speed in the first place.

These gamers can flip a switch on their card to enable a higher fan speed profile but can't do similar with a slider bar in afterburner? Seems people want to make out PC gamers as idiots, but give them the benefit of the doubt when it happens to benefit one company or another.

It dictates the clock speed UP to the setting by the manufacturer, thus its not running out of spec and its therefore NOT AN OVERCLOCK. Its actively encouraged for gamers to flip the switch if they don't mind the noise, knowing full well its designed to run within that spec, its not voiding their warranty.

As i've said, its an obsolete gimmick, but as long as its here and as long as people who folk out $550 for it are willing to put up with the extra noise... why not test it out? It's just lazy. The excuse of future scenarios of WHAT IFs are just that. Wait til somebody is stupid enough to release 4 different bios switches to flip then put the feet down. As it is, there's only two.

 

[BallaTheFeared]

Overclocking doesn't void the warranty, and if AMD set the max clock for 1200, but could only achieve 800 on silent, and 1000 with uber, what then, putting a block on it to hit 1200 wouldn't be overclocking either?

Whats the difference between switching the bios to uber and putting +250/300 into Precision for the 780 Ti without adjusting fan or voltage?

 

[railven]

Overclocking doesn't void the warranty, and if AMD set the max clock for 1200, but could only achieve 800 on silent, and 1000 with uber, what then, putting a block on it to hit 1200 wouldn't be overclocking either?

Whats the difference between switching the bios to uber and putting +250/300 into Precision for the 780 Ti without adjusting fan or voltage?

If it was a feature of the card such as that lovely HoF. Nothing wrong. Fair game. Flip that switch!

 

[GaiaHunter]

Thanks for your responses A few thoughts:

1) Recommendation to just test 2 modes (slowest and fastest), this makes sense until one manufacturer decides to make their fastest mode something ridiculous. For the folks who suggested this, do you believe it's still a good option if the high performance mode is something like twice the noise of the next lowest setting as well as the high performance mode on a competing card?

This may sound like a contrived scenario but you would be really surprised at the sort of stuff we have to try and defeat before it ever gets proposed as a part of the reviews process. I feel like that's the major concern here, that a willingness to test more than the default setting (and especially a commitment to test the highest performing setting) will almost certainly result in a manufacturer creating a ridiculous highest performance setting to game reviews. The beauty of making it a point to only test default settings is that it encourages manufacturers to optimize for the primary use case of their customers rather than being able to go off and optimize for how the cards would be benchmarked instead. No manufacturer would ship a card in an obnoxious/unusable/insane default configuration for obvious reasons (returns, poor customer experience, etc...).

Take care,
Anand

That is why you test noise as well as performance.

What is this idea of equalizing cards?

Why only noise? Why not power? Why not price? Why not performance? Why not memory amount?

A card that is twice as noisy but it is the best performer is something for the consumer decide not for the reviewer decide - "ah card X is actually faster but at that setting it consumes more power and it is noisier so we've decided that to be fair for the competition we will equalize the card".

And being twice as noisy doesn't actually mean it is unbearable if the other card isn't that loud to start with.

I don't know how it works in the US but in the EU someone buying a graphics card finding that the noise of the 290X uber mode is unbearable and that the performance of the silent mode is not adequate can go to the retailer and return the card.

This sounds like the "dumb down" of reviews - "it might be too hard for the consumers read all the information and make their own decision so we will just cut down some information so we can point them in the right direction".

 

[BallaTheFeared]

It's pretty easy to see what happened with boost this generation.

Nvidia went with a quite fan profile, limiting the potential of the card to obtain it's performance at low noise levels.

AMD took the same concept, and used it to push the clock speeds up at the expense of noise to gain the performance they needed with a less optimal solution.

So how do we compare such cards in a review, specifically when AMD knew Uber would be loud, louder than any recent video card, louder than even the GTX 480 thus felt compelled enough to add a switch between a more reasonable fan profile, albeit slower, and one that is far and way exceeding anything put out by the competition? (RUN ON BABY)

The problem AMD has created for reviewers is how do you merit such actions, when the end result will simply be 100% fan speed options with boost clocks none of the cards will actually hit, and no base clock to control the bottom level of performance you could get with more reasonable fan speeds/noise?

How can you define the performance of a video card which is what reviewers do, if there are so many variables, and how do you compare such a card that requires user input without giving similar input to the other cards (example: adding an offset in precision to the 780 Ti without adjusting fan or even voltage)?

 

[GaiaHunter]

Overclocking doesn't void the warranty, and if AMD set the max clock for 1200, but could only achieve 800 on silent, and 1000 with uber, what then, putting a block on it to hit 1200 wouldn't be overclocking either?

Overclocking might not void the warranty, depending of the specific manufacturer, but it isn't guaranteed that the card will work at whatever OC settings.

If AMD states that with better cooling (be it water or be it higher fan speed) 1200 is stable on every card than it isn't an OC and you can return the card for not reaching its specs.

 

[BallaTheFeared]

Just like if the card runs at 600MHz, you can't return it because it doesn't meet the expectations you had based on reviews because you run the fan at 30% instead of 55%?

Sorry but I see no benefit for arguing this strategy as valid for the end user, only for the corporations.

 

[GaiaHunter]

It's pretty easy to see what happened with boost this generation.

Nvidia went with a quite fan profile, limiting the potential of the card to obtain it's performance at low noise levels.

AMD took the same concept, and used it to push the clock speeds up at the expense of noise to gain the performance they needed with a less optimal solution.

So how do we compare such cards in a review, specifically when AMD knew Uber would be loud, louder than any recent video card, louder than even the GTX 480 thus felt compelled enough to add a switch between a more reasonable fan profile, albeit slower, and one that is far and way exceeding anything put out by the competition? (RUN ON BABY)

The problem AMD has created for reviewers is how do you merit such actions, when the end result will simply be 100% fan speed options with boost clocks none of the cards will actually hit, and no base clock to control the bottom level of performance you could get with more reasonable fan speeds/noise?

How can you define the performance of a video card which is what reviewers do, if there are so many variables, and how do you compare such a card that requires user input without giving similar input to the other cards (example: adding an offset in precision to the 780 Ti without adjusting fan or even voltage)?

Reviewers jobs is to review the cards and present correct and useful data.
It is the consumers rights to decide what is reasonable for them or not.

Additionally there is loads of Ifs that have no connection with the current situation.

By the way the 290 also has 2 bios files that are the same. In fact since the 6970/6950 many of the AMD flagship reference models come with 2 bios files.

So they didn't just added a switch to the 290X - it is a useful feature to have 2 BIOS files stored.

 

[Abwx]

From the other thread it seems that Nvidia s coolers are left
uncovered and as a consequence most of its heated air flow
will be blown inside the case...if there is one..

Measuring cards noises in isolation require careful study
of the measurement relevancy , in that case the Nvidia card
will heat the inside of the case much more than the Radeon
wich blow its air outside of the case , hence the card will be for
sure more silent but only to transfer the burden to the case
extracting fan that will spin harder to get out the heated air,
in short both the air extraction and subsequently a part of the
noise will be transfered to the case fans wich will negate
the noise advantage of the Nvidia cards when measured
in isolation.

 

[GaiaHunter]

Just like if the card runs at 600MHz, you can't return it because it doesn't meet the expectations you had based on reviews because you run the fan at 30% instead of 55%?

Sorry but I see no benefit for arguing this strategy as valid for the end user, only for the corporations.

I don't see any factory preset that runs the fan at 30% for the 290X.

But you can return the card if you find the silent performance doesn't meet your needs and the uber setting is too loud.

 

[BallaTheFeared]

This thread is for the reviewers, not for us.

The difference between a bios switch with a backup bios, and a bios switch with two different performance parameters and noise profiles is they're two different cards, one slower with less noise, the other faster with considerably more.

The reason for the what if's is because that is the concern of Anand and Ryan, what if the 780 Ti shipped with a bios switch that had 100% fan speed? People are already willing to argue this isn't overclocking, so what then? Then reviewers will complain 100% was too high, so because 100% might not be too much for the avid fans than they'll need a third option, and a fourth.

Reviewers are under enough pressure right now without having to run every card in their list, and then have to run the same cards two, or three more times to get a review out after only having a sample for a few days prior to the DNA lifting.

I don't see any factory preset that runs the fan at 30% for the 290X.

But you can return the card if you find the silent performance doesn't meet your needs and the uber setting is too loud.

Point was AMD went so far as to not include a baseline performance figure for their card. There is no point where you can RMA based on your stipulation of clock/performance because AMD doesn't guarantee any clock speed, in your example you said if they said it would run up to 1200 and you put it on water and it couldn't reach it you could rma, but you couldn't because up to doesn't mean it will.

 

[Leadbox]

I believe you should only bother with Uber mode and at the same time maximize the temperature and power settings for all the NV cards.Win Win for all.

Were reviewers have tried to equalize things, they lowered the R9s fan speed to 34% to match the noise levels of the geforces. They didn't speedup the geforces fans because that doesn't do to much for their performance as they will still throttle due to power limits yes?
Now you want them to up the geforce power settings because you know increased fanspeed won't do much for their performance compared to what it does for the R9s
Here, have a listen to to noise levels of a 780 at a setting that prevents thermal throttling compared the 47% fanspeed of the 290 and there goes the noise argument

 

[GaiaHunter]

This thread is for the reviewers, not for us.

The difference between a bios switch with a backup bios, and a bios switch with two different performance parameters and noise profiles is they're two different cards, one slower with less noise, the other faster with considerably more.

The reason for the what if's is because that is the concern of Anand and Ryan, what if the 780 Ti shipped with a bios switch that had 100% fan speed? People are already willing to argue this isn't overclocking, so what then? Then reviewers will complain 100% was too high, so because 100% might not be too much for the avid fans than they'll need a third option, and a fourth.

Reviewers are under enough pressure right now without having to run every card in their list, and then have to run the same cards two, or three more times to get a review out after only having a sample for a few days prior to the DNA lifting.

If the 780 Ti came with a 100% fan speed its performance would be exactly the same because the chip can't simply keep increasing its frequency. If it also increase voltage and power consumption, NVIDIA would have to make sure the circuitry was up to the task of extra power.

So NVIDIA would have to have better components (more expensive) would have to have chips that could reach higher speeds (again higher costs on binning with lower chips reaching those speeds/voltages), etc.

How much does NVIDIA allow the power to be increased in the 780 TI? Is it something like 5-7%?

It is a contrived scenario.

Also if some reviewers can't present all the information while some will be able to, the market will decide which reviewers will have more hits.

It is a business after all - that is why when some reviewers started using FCAT a large majority had to start using it as well otherwise their reviews would have less added value, even though it is more time consuming.

Point was AMD went so far as to not include a baseline performance figure for their card. There is no point where you can RMA based on your stipulation of clock/performance because AMD doesn't guarantee any clock speed, in your example you said if they said it would run up to 1200 and you put it on water and it couldn't reach it you could rma, but you couldn't because up to doesn't mean it will.

Up to isn't a free pass for manufacturers, at least not in the EU.
Otherwise AMD would have said "UP to 5GHZ" instead of the "Up to 1GHz" that its uber mode pretty much hit all the time.