Spin Off: AT's Testing Methods & Uber Mode

[blackened23]

You should just add the 290X cooler spiel to your sig... It can save you valuable typing time

I know, dead horse and all that. But you have to admit every single issue that has popped up, every single complaint, goes back to that very issue. Throttling? Uber mode? Noise? It all goes back to that and could have been prevented. I dunno man. I was excited about the 290 and wanted the 290 to be all that and then some - yet because AMD made one bone headed decision along the way, it creates 4 or 5 controversies which were entirely preventable. Even if the card had cost 30$ more with a better shroud, the toms controversy would not exist, complaints of the AT review would not exist, none of this would exist. You know? So. There. AMD is where they are in terms of internet controversy because of one single bone headed decision and it truly is mind boggling. /dead horse. I'll stop.

 

[ViRGE]

So again, since out of the box performance DOES NOT include raising frequency or voltage, why is AT testing that, but can't flip a switch?

Uh, unless I'm missing something here (and it's entirely possible I am), AT only adjusted the settings on their 780 Ti for the overclocking section. Which provides a neat look at the overclockability of the card, but isn't a piece of data AT uses to draw their conclusions.

I don't think AT is trying to imply that they won't test overclocking or uber mode when they're reviewing the specific product in question (e.g. the 780 Ti review). Rather when they're going to be comparing a new product to an old product, they will be drawing those comparisons with both products at their default settings.

EG, if Intel magically came out with a high-end GPU tomorrow, how should it be compared? Stock Intel, stock NVIDIA, stock AMD (quiet mode)? Stock Intel, stock NVIDIA, adjusted AMD (uber mode)? Overclocked Intel, overclocked NVIDIA, overclocked AMD? The matrix of possibilities explodes very quickly, and in a review you can really only use one of them to draw your base conclusions.

 

[GaiaHunter]

AMD uber mode is a factory setting of the 290/290X.

Sure, for reviewers might be annoying to test basically 2 cards but for the consumers it is a flip of a switch covered by warranty.

An easy solution if reviewers have that stance is sending the review samples with the switch on the high performing setting.

 

[tg2708]

This is also getting 'too long in the tooth.' Why complain about a reference cooler when you know for a fact that you will only be buying aftermarket cards? More than half of the people here complaining through out the forums about the reference blower are all using non reference cards. Also the ones who bought the reference cards are the ones that want first dibs to see what can most likely be achieved and will most likely sell the card once aib comes in because they are bound to improve up on certain aspects of the card whether its heat or noise reduction.

 

[chimaxi83]

Uh, unless I'm missing something here (and it's entirely possible I am), AT only adjusted the settings on their 780 Ti for the overclocking section. Which provides a neat look at the overclockability of the card, but isn't a piece of data AT uses to draw their conclusions.

I don't think AT is trying to imply that they won't test overclocking or uber mode when they're reviewing the specific product in question (e.g. the 780 Ti review). Rather when they're going to be comparing a new product to an old product, they will be drawing those comparisons with both products at their default settings.

EG, if Intel magically came out with a high-end GPU tomorrow, how should it be compared? Stock Intel, stock NVIDIA, stock AMD (quiet mode)? Stock Intel, stock NVIDIA, adjusted AMD (uber mode)? Overclocked Intel, overclocked NVIDIA, overclocked AMD? The matrix of possibilities explodes very quickly, and in a review you can really only use one of them to draw your base conclusions.

I agree with you, I was just presenting another view. Not every site will cater to everyone, that's why you read more than one review. It would be a ton of numbers and combinations if every review showed every piece of hardware in every configuration.

 

[Erenhardt]

Note that this has been spun off from another thread in order to keep this subject from sidetracking the original subject
-ViRGE

What is worth mentioning are reviewers thoughts below graphs:
Metro:

Meanwhile looking at GTX 780 Ti SLI performance, the SLI setup tops the charts at 2560 for everything short of the 290X in uber mode, though in this case (like most cases) two high-end GPUs is on the verge of being overkill even at 2560. Otherwise looking at 4K, NVIDIA’s poor 4K scaling on Metro once again makes itself present here, with NVIDIA’s performance only minimally benefitting from the second card. In the case of Metro at 4K, the 290X CF is going to be by far the faster option.

In case someone didn't see graphs - SLI 780ti was 3 fps behind CF 290X(Uber) in best case scenario. So, topping charts but not.
Bioshock:

Meanwhile for the AFR matchup, with a pair of GTX 780 Ti’s we’re either looking framerates that will make a 120Hz gamer happy, or enough horsepower to take on 4K at our highest settings and still come out well ahead. At 57.3fps the GTX 780 Ti is several frames per second ahead of the 290X CF, coming up just short of averaging 60fps even at this very high resolution.

By "several frames per second ahead of 290X CF" they meant actually 2fps behind it.

BF3:

Moving on to SLI performance, the GTX 780 SLI is once again a chart topper. Even 3840x2160 and with Ultra quality, the GTX 780 Ti still more than enough to deliver more than 60fps, making this a fully playable resolution with minimum framerates that should easily be over 30fps.

Chart topper (ahead in 1 of 3 charts)

And my favorite one in

Moving on, even when we double up on cards the GTX 780 Ti and 290X remain close. At 2560 it’s a virtual tie at 87fps apiece, while at 4K the GTX 780 Ti SLI takes a slight lead.

88 for SLI 780Ti vs 98 CF 290X is a tie and at 4K SLI takes slight lead being 20% slower?!

I case someone want to jump on me. I see what they did there. Next time on anandtech: amazing race through firestrike benchmark on idle clocks! See which card idles faster!

 

[Ryan Smith]

I noticed that too, but Ryan didn't seem to comment much on multi-card performance.

We covered it in far more detail in the 290X launch article. For the GTX 780 Ti review we made the performance data available in our charts, but from a commentary perspective the scaling factors and the reasons behind them have not changed since the 290X article where we compared GTX 780 SLI to 290X CF.

Erenhardt: From our testing notes on page 3:

Meanwhile on a housekeeping note, we want to quickly point out that we’ll be deviating a bit from our normal protocol and including the 290X results for both normal (quiet) and uber modes. Typically we’d only include results from the default mode in articles such as these, but since we need to cover SLI/Crossfire performance and since we didn’t have 290X CF quiet mode results for our initial 290X review, we’re throwing in both so that we can compare the GTX 780 Ti to the 290X CF without being inconsistent by suddenly switching to the lower performance quiet mode numbers. Though with that said, for the purposes of our evaluation we will be focusing almost entirely on the quiet mode numbers, given the vast difference in both performance and noise that comes from using it.

Our policy is to always compare out of the box performance when possible. The inclusion of the uber mode numbers is essentially a one-time event.

 

[ViRGE]

An easy solution if reviewers have that stance is sending the review samples with the switch on the high performing setting.

Okay. So then AT reviews it as a high performance, high noise card, and calls it out on the fact that it's 59dB loud, peaking at over 61dB. That gives the best performance numbers, but now the baseline for a 290X would be that it's a really loud card (relatively speaking) with fairly high power draw (since the limited cooling keeps the power draw down some).

Not that I'm saying you're wrong, BTW. I'm just thinking this out and trying to see where this would lead.

 

[GaiaHunter]

Okay. So then AT reviews it at a high performance, high noise card, and calls it out on the fact that it's 59dB loud, peaking at over 61dB. That gives the best performance numbers, but now the baseline for a 290X would be that it's a really loud card (relatively speaking).

Not that I'm saying you're wrong, BTW. I'm just thinking this out and trying to see where this would lead.

Obviously the most informative for the readers is having both settings tested.

Why is AT deciding which one is the baseline?

Especially since noise is sujective?

They can say "look guys we don't really like the uber mode for this so and so reason", no problem, but why omitting it when it is a factory feature of the card?

 

[Creig]

I'm a bit confused by Ryan's stance as well.

Personally, I'm glad to see the existence of this switch because now the card can appeal to an even greater audience. Those that want good performance with lower sound levels can leave the switch in the quiet position. Those people who wish to wring every last possible FPS out of the card can move the switch to the uber position.

Both positions are covered by the factory warranty. You aren't modding or hacking anything. And if you are smart enough to be able to flip a light switch, you can probably figure out how swap between quiet and uber modes.

As has been pointed out already, these two modes mean double the work while benchmarking. And I'm sure that sucks for reviewers. But more options are better than fewer options in my opinion.

 

[Owls]

I wonder why Ryan hasn't responded to these allegations of bias. We all come to AT for in depth reviews and objectivity. This review was well below the standards AT has provided.

Clearly the review was biased and we deserve answers.

 

[ViRGE]

I wonder why Ryan hasn't responded to these allegations of bias. We all come to AT for in depth reviews and objectivity. This review was well below the standards AT has provided.

Clearly the review was biased and we deserve answers.

Your tone is exactly why no one in their right mind would respond. It's accusatory, hostile, and altogether rude. It's a no win scenario. If he comes, he'll just get ganged up on and pummeled by trolls.

 

[krumme]

Some marketing newbees at AMD probably thought that they could get both a high perf. and low noise review by this switch because it is a difficult as to flip a light switch.

Well in a market where NV owns the brandvalue, technical support, the "relations" and very much the cognitive perception, that is a very naive thinking.

Would they eg. have predicted the 5,5db difference between an 290 and 780ti lead to untill now unseen calculation of perf. with equal db - or nearly a page of noise rant from Ryan. For 5.5dB ! And then the sli vs cf more or less the same talk. lol.

Typical lousy cooler + bad marketing decisions + angry biased reviewer = crazy stuff and new inventions = punishment for newbees

 

[krumme]

Your tone is exactly why no one in their right mind would respond. It's accusatory, hostile, and altogether rude. It's a no win scenario. If he comes, he'll just get ganged up on and pummeled by trolls.

Ofcourse its difficult to come here.

We have explained why we think the review is off. There is plenty of examples.

We are critical and thats okey. Ryan "tone" in the 290 review is not exactly rosy either. Thats how it is. Its perfectly fine not to agree.

 

[krumme]

Imagine if the responsible for the 290 blower decision from AMD made his way to this forum. lol.

 

[realibrad]

I think the main issue to be addressed is, what is the purpose of benchmarks? Empirically testing things like FPS, wattage and db are based on the "out of the box" settings. But then, how would you test a dual gpu card? What does out of the box mean? Opening the case, putting the card in, running the cables, setting up the software that is specific to the dual gpu card, then bench marking.

It may be assumed that a dual gpu will need to have CF or SLI enabled to fully test the cards, but that is something that has to be enabled, and would not be set up out of the box.

If you dont want to test another factory setting, because you make a subjective decision, thats fine, but lets not pretend its objective. Yes, including 2 sets of data on top of OC tests will make more work and that does cost something somewhere. If this cost is not worth it to you, fine. AT is not obligated to test anything, no matter how much users complain, but, users are not obligated to agree on opinions either.

Most cards ship with 1 setting that is determined by the creator to be the best balance of noise, power, heat and performance which makes things easy. AMD did something different by allowing 2 options. Luxury car manufactures do this with sport and cruise mode. What AT did was test cruise mode against cars with a mixed mode. Sure, AMD's cruise mode was not all that efficient, but you cant use it as the base line, simply because thats what is enabled by default.

 

[EliteRetard]

I'm going to add fuel to the fire here:

Looking back at the GTX480 review, the card was far hotter, louder, and used more power than any other single card. In fact, it was as hot and louder than the 290x. GTX480 at 64.1db 290x at 58.9db.

Did they flat out say it was a crap card and totally not recommend it for purchase?
No, it was simply recommended to get one with a lifetime warranty (chance of failure from heat).
It was 6 months late and marginally faster than the 5870 with huge trade off in heat, noise, power, AND a higher price...but it gets the go ahead.

But if AMD makes a faster card that costs less with a heat/noise trade off its a terrible thing.

 

[Atreidin]

I believe this is a case of policies lagging behind innovation and changing markets, and as a result not accurately reflecting how people actually use the product in a way that the manufacturer intends.

As has already been pointed out, there is an analogy with luxury cars. Nobody is going to be driving their luxury car and think, "Gee, I sure wish I could use sport mode, but when I drove the car off the dealer's lot, it was in cruise mode. Oh well." Similarly, car reviewers aren't going to refuse to test one official supported driving mode simply because the switch wasn't flipped to that position from the factory, or else nobody will take them seriously.

The fact is that AMD should get some credit for their innovation. It is a new way of making graphics cards, and I like it, because it gives choice on how to run your card. As long as you cover the difference between both modes, I don't see the problem with reviewing them both. Nobody is stopping Nvidia from implementing this themselves. But the fact is AMD did it first, and any claims of it being "unfair" to look at both modes are just wrong. In my idealized world, reviews would cover how people actually use the product, and even if you add the rule "and also how the manufacturer intends it to be used (or within warranty)", well, AMD has 2 ways it can be used, both within warranty. Nvidia has 1. AMD came up with a novel idea, and even if you don't like it, they should get some credit for trying. Nobody is forcing a 290X in uber mode into your computer.

Of course nobody *has* to review any aspect of any product if they don't feel like it. You could test which card has better throwing distance or first releases magic smoke when dumping water on it. Put up what you want, and readers will read what they want. It doesn't have to be more drama-filled than that.

 

[TerryMathews]

I don't think AT is trying to imply that they won't test overclocking or uber mode when they're reviewing the specific product in question (e.g. the 780 Ti review). Rather when they're going to be comparing a new product to an old product, they will be drawing those comparisons with both products at their default settings.

And you don't see anything wrong with building comparison charts with the cards set slowly on slow mode?

I truly am not understanding why in this day and age we would leave data off a chart? I mean, you dont have to make it fit on an 800x600 screen anymore.

For completeness sake, include over clocked Nvidia cards if you need to (although they aren't warrantied in that state so I don't believe it's apple to apples...), just don't leave half the data off the chart due to some slavish adherence to a policy that makes no sense.

 

[GaiaHunter]

Talking about the GTX 480 decibels level, some other forum members said that you can't compare since the test procedure might have changed.

But the currently test procedure is at 1 foot distance.

So unless they kept the mic just by the fan, I'm not seeing how the GTX 480 could have become so quiet in the recent 290 review, unless they are using a non reference one.

 

[VulgarDisplay]

Am I nuts or do I see uber numbers in all the charts?

 

[TerryMathews]

Am I nuts or do I see uber numbers in all the charts?

You aren't nuts. The discussion is surrounding Ryan's comment that the inclusion is a one-time exception.

 

[Meekers]

You aren't nuts. The discussion is surrounding Ryan's comment that the inclusion is a one-time exception.

And even though they were in the charts the article pretended they did not exist.

My main problem with all of this is that Ryan eviscerated AMD for sound/heat from Uber mode and used that to refuse to recommend the card, then goes ahead and then goes ahead and ignores its performance. He seems to be going out of his way to paint the card in a bad light. Between the two reviews he used the sound/heat from uber and performance from quiet to trash the cards.

 

[ViRGE]

So unless they kept the mic just by the fan, I'm not seeing how the GTX 480 could have become so quiet in the recent 290 review, unless they are using a non reference one.

Looking at the articles, it looks like they switched cases, CPUs/coolers, and benchmarks between then and now. The old noise data was only taken from FurMark, whereas they're now primarily focusing on a gaming workload (Crysis 3). That last bit would explain a large piece of the difference, as the 480 has a much larger difference between gaming and FurMark due to the lack of modern power management.

In the 290 article they threw in the GTX 480 tested to modern standards; it was 54.7db with Crysis, 65db with FurMark. If their primary frame of reference is now games as opposed to Furmark, that would certainly change what's average and what's acceptable. Especially since that would mean the 290 is louder than the GTX 480.

My main problem with all of this is that Ryan eviscerated AMD for sound/heat from Uber mode and used that to refuse to recommend the card, then goes ahead and then goes ahead and ignores its performance. He seems to be going out of his way to paint the card in a bad light. Between the two reviews he used the sound/heat from uber and performance from quiet to trash the cards.

Huh? The 290 doesn't have an uber mode. It only has the one mode, and was trashed based on that. 290X does have two modes, but that certainly didn't get trashed.

 

[Face2Face]

Everyone knows how loud the 290X is in uber mode but what enthusiast reading these reviews is going to keep the switch set to Quiet mode while gaming?