Testing Nvidia vs. AMD Image Quality

[SirPauly]

Rather than having nvidia pr send a card to bfg, who is on record of preferring nvidia --- send it to me. Hell kyle and the rest have all had their cards and given their judgment. This is just nvidia trying to present an issue most don't see as being critical.

I strongly disagree. BFG hammered nVidia on the GTX 470 in some older titles.

 

[GaiaHunter]

Yes, but a lot of other websites don't use canned benchmarks. Sites like PCGameshardware, Hardwarecanucks, ABT, gamegpu.ru use manual run demos too. With [H], the apples-to-apples benchmarks are just limited to 2560x1600 as of late. That's not very useful for 99% of people, and especially not when the reviewer starts to imply what a "smooth playable level is" in any game. The smooth/playable FPS is unique for every gamer. Some are OK with 30 fps, some want 60 fps, some want 80-100, etc.

You are not understanding how [H] test it.

[H] actually plays a large portion of the game and then chooses one of the most intense area to bench it - and that is where the numbers you see in the review come from.

So basically their reviews transmit their own gaming experience - that is why sometimes a game at 33.5 fps is considered playable and another at 31 isn't at those settings.

That is why they don't review a dozen of titles because they make sure they played a sizeable sample of the game, even if they only show a benchmark covering a few minutes of game play.

Is their gaming experience relevant for all of us readers/consumers?

Guess some will be left out.

For example some people wouldn't play Starcraft II without AA. For me, that play mostly multiplayer the AA is irrelevant. Someone playing the campaign though, might see a better benefit due to cinematics (and the 3d portraits of units on the HUD) really showing aliasing.

Is it subjective? Yes.

But are benchmarks concerning a few minutes of a game really representative of the entire game?


To resume [H] plays a substantial portion of a given game, determines the highest IQ/resolution that feels playable (on their own opinion) on the entire game and then chooses an area from that game to present numbers for the reader.

 

[SirPauly]

There was a time when Kyle just offered his subjective findings and it was tough to gauge. There was discussion on this at Beyond3d and offered my views without attacking Kyle personally. I found it did take great courage to try to change the ways that many reviewers evaluate product but with his review style was tough to gauge a foundation to build upon and needed apples-to-apples coupled with his subjective findings; to help readers.


http://forum.beyond3d.com/showpost.php?p=1124616&postcount=53

 

[Teizo]

In a 6850 follow up review [H] tested a 6850 (I think it was the Asus DirectCu) and compared it the reference 460, and when it came to BFBC2 they said the 460 was not capable of 4xAA with playable frame rates. They said it was 'almost' capable...I found that interesting because I can play at 4xAA just fine with my 460. It was even more interesting that the apples for apples showed the reference 460 only 1 fps behind, and when it showed that, they downplayed it as much as possible.

That is not to say they are pro or anti anything I think because they did the same thing with the 580 vs 5970 comparison, favoring the 580 overall even though it got less overall framerates, but when I read someone saying something (the 460 is BFBC2) is not playable at a certain setting and saying it is definitely so, when I know myself it is definitely not, I am forced to not take their opinion too seriously.

 

[SirPauly]

Thing for me, is I have to read all the reviews because like gamers themselves, reviewers' tolerances and subjective areas may differ, too. Ya like to keep subjective out but if one can showcase differences through wording, screenshots, evaluation, videos, coupled with synthetics and areas that are GPU limited in titles that can be easily reproduced.

Still feel that IQ is not really subjective but tolerances are subjective.

 

[GaiaHunter]

In a 6850 follow up review [H] tested a 6850 (I think it was the Asus DirectCu) and compared it the reference 460, and when it came to BFBC2 they said the 460 was not capable of 4xAA with playable frame rates. They said it was 'almost' capable...I found that interesting because I can play at 4xAA just fine with my 460. It was even more interesting that the apples for apples showed the reference 460 only 1 fps behind, and when it showed that, they downplayed it as much as possible.

That is not to say they are pro or anti anything I think because they did the same thing with the 580 vs 5970 comparison, favoring the 580 overall even though it got less overall framerates, but when I read someone saying something (the 460 is BFBC2) is not playable at a certain setting and saying it is definitely so, when I know myself it is definitely not, I am forced to not take their opinion too seriously.

And are you sure your 460 isn't OC'ed?

 

[Teizo]

And are you sure your 460 isn't OC'ed?

Ha. Yeah, it is factory overclocked. I guess I should have mentioned that as undoubtebly that helps. Just about every 460 you buy is too though. Sorta does make benching them for proper end user comparison a bit difficult, but no other review site on the web I've seen listed the reference 460 as having 'trouble' playing BFBC2 maxed with 4xAA 16xAF @ 1080 in the fashion he did. And even when the apples to apples showed effectively a draw, he downplayed the results to favor his opinion.

His reviews aren't crap, and they do offer useful information to readers, but I much prefer complete apples to apples comparisons which don't allow for any user bias. And, as mentioned, I don't think he's biased either way (though he has given nVidia an open lashing or two of late), but it is only human to favor one's opinion and argue that opinion in the face of conflicting data (such as down playing the apples to apples comparison in that review in that game). And, when you do that it makes it hard to take a person's opinion as seriously as he would like you to.

 

[Dribble]

The problem with [H] reviews is they are really only the most optimal setting for Kyle. He has his pet loves and hates. He loves high res, preferably lots of monitors in eye infinity. He doesn't care much about frames per second as long as to him it looks smooth.

There's a whole other school of gamer who want really high fps and don't really care about the ultra large screens and high res. Hence Kyle has decided 30fps is enough, where as another type of gamer wouldn't start turning settings up till he was at a steady 120fps.

Personally I only have one screen at 19*10 so I don't care about any other res. I want at least 60fps, more if I can manage it, and will lower settings to medium and no AA to try and get that (but not low cause that normally looks *too* ugly). Hence when I read a [H] review I only get limited info - they tend to run at much higher res and with more AA which means they are often gpu memory bound. I on the other hand don't mind sacrificing high levels of AA so don't have the same gpu memory problems. Sometimes I end up cpu bound (responsible for the min's anyway) and not gpu bound at all.

 

[NoQuarter]

In a 6850 follow up review [H] tested a 6850 (I think it was the Asus DirectCu) and compared it the reference 460, and when it came to BFBC2 they said the 460 was not capable of 4xAA with playable frame rates. They said it was 'almost' capable...I found that interesting because I can play at 4xAA just fine with my 460. It was even more interesting that the apples for apples showed the reference 460 only 1 fps behind, and when it showed that, they downplayed it as much as possible.

They said:
"The NVIDIA GeForce GTX 460 was almost playable at these settings too however when we got into huge firefights the mouse would lag sometimes when moving our sights from one target to another. This meant we had to turn off any AA before our mouse movements stayed smooth."

This is the subjective analysis that doesn't show up in Avg fps numbers and can even be hard to discern from the fps graph but is what made them have to turn off 4xAA. Just because the average fps was only 1 lower doesn't mean it was equally playable. The OC on your 460 is just enough to get it over the hump that was making 4xAA too unplayable during heavy firefights.

 

[Stoneburner]

Yano, [H] does provide objective comparison after the semi-subjective stuff. I'm not sure why it's being criticized, it's overall the most useful testing I've come across.

 

[3DVagabond]

Yano, [H] does provide objective comparison after the semi-subjective stuff. I'm not sure why it's being criticized, it's overall the most useful testing I've come across.

Love the FPS/Time graphs. I wish everyone used them. The highest playable settings gives very good info as well. Apples to apples shows everyone what they want to see, the "horse race" between the brands, but the info given by the other graphs, which is fairly unique from a trusted source, is what sets [H] reviews apart.

Look at this graph and tell me what use knowing just peak - avg. - min FPS would tell you compared to seeing the graph.

 

[Teizo]

Haven't fully read the article yet, but Hilbert has done a write up called "Exploring ATI Image Optimizations"

http://www.guru3d.com/article/exploring-ati-image-quality-optimizations/

/Ah, well...according to him it is definitely there though he states he doesn't feel it is a 'cheat' even though in his opinion it should be disabled by default. So, imho, if Hilbert says it is there and it is not being made up by Nvidia as some stunt...then that finalizes it for me, and should for everyone else whether you care about it being there or not.

 

[CitanUzuki]

From the same article:

But let's take a more real-world example then, please download all three images, and load them up in say Photoshop. Make sure you zoom in at all screenshots to 100% and then start to compare the three images (each lossless 12 MB BMP files).

Mass effect R6850 ATI Optimization Off - Download JPG
Mass effect R6850 ATI Optimization On - Download JPG
Mass effect NVIDIA GTX 580 Defaults - Download JPG
This is Mass effect, 16xAF and Trinlinear filtering enabled in the configuration file. We are in the spaceship and have positioned ourselves with a surface area where the optimization should show really well. Yet we have a more complex scene with nice textures and lots of colors, much less bland then the simple example shown previously. This time just a Radeon HD 6850 with the optimization on and off and a GeForce GTX 580.

We have a hard time spotting differences as much as you do, and while making the screenshots we increased gamma settings to 50% and applied a resolution of 2560x1600 to try it look more visible.

Do you spot the difference ? Probably not, that is the rule we life by here at Guru3D, if you can not see it without blowing up the image or altering gamma settings and what not, it's not a cheat. And sure, we know .. this game title is not a perfect example, it however is a good real world example.

 

[Teizo]

Yeah, he said it is not a cheat so not to be viewed as one, but he agrees it is there and that it should not be enabled by default.

From a reviewers perspective, and an end user as well when I look at reviews, I want to know that the drivers are set to the same quality when looking at the performance numbers so I can make a more properly informed decision on what card really offers more performance before I spend my money. That is something we should all demand. It only makes sense to do so.

So, AMD needs to disable it by defualt, or the proper settings need to be made in order to do a more fair comparison. It doesn't matter if people are willing to game with the optimization on. This is from a review perspective, and giving the end user accurate information about a cards performance in comparison to another.

 

[CitanUzuki]

It sounds as though you are saying nvidia has no optimizations in the default driver settings, but I dont know if that is the case. Some reviewers do set them both to highest quality, which is probably best, assuming both companies high quality setting are equivalent to one another.

 

[Teizo]

It sounds as though you are saying nvidia has no optimizations in the default driver settings, but I dont know if that is the case. Some reviewers do set them both to highest quality, which is probably best, assuming both companies high quality setting are equivalent to one another.

If Nvidia has some optimization that affect things the same way, then yes, it should not be there by default when the cards are being tested. According to those who have looked in to it though, Nvidia does not have the same optimization, so this only is affecting AMD.

 

[tannat]

In a 6850 follow up review [H] tested a 6850 (I think it was the Asus DirectCu) and compared it the reference 460, and when it came to BFBC2 they said the 460 was not capable of 4xAA with playable frame rates. They said it was 'almost' capable...I found that interesting because I can play at 4xAA just fine with my 460. It was even more interesting that the apples for apples showed the reference 460 only 1 fps behind, and when it showed that, they downplayed it as much as possible.

That is not to say they are pro or anti anything I think because they did the same thing with the 580 vs 5970 comparison, favoring the 580 overall even though it got less overall framerates, but when I read someone saying something (the 460 is BFBC2) is not playable at a certain setting and saying it is definitely so, when I know myself it is definitely not, I am forced to not take their opinion too seriously.

Actually, this is what most people don't seem to get. What you or I think is playable have no effect on the outcome of the test whatsoever.

They give you a diagram with the framerates, to back it up, in case you choose to question their credibility. The game need to pass two tests: Visual playability and measured frame rates within defined limits shown in the diagram.

They choose the most most demanding part of the game. Compare with Crysis: You can play through much of Crysis at quite high settings but will need to take IQ down several notches at Ice level and end game.

Additionally they complement with apples to apples, in case your choice of baseline does not agree with theirs.

 

[RussianSensation]

Actually, this is what most people don't seem to get. What you or I think is playable have no effect on the outcome of the test whatsoever.

You are right that our perception of what is playable has no impact on the benchmarks at {H}. But the way they are tested has an impact on their usefulness. For example, if my perception of playable is 60 fps in a racing game, then I have no idea at what settings the cards can produce that in F1 2010 for example. In addition, since in almost all the latest reviews every game is tested at 30-35 fps avg at 2560x1600 w/ AA, those results tell me absolutely nothing about 1920x1080, 1920x1200, etc. So how exactly am I supposed to make a purchasing decision from those results? Kyle will also pick and choose various settings, such as running NV cards with PhysX with no AA vs. AMD cards with no PhysX with AA. How are you supposed to draw any conclusion from those results? You can't compare PhysX to AA (they are completely different visual enhancements).

The idea of plotting fps over time on a graph in the toughest part of the game is a great one. However, unless he introduces other resolutions and considers that not everyone is happy with 35 fps, the execution leaves much to be desired. I see [H] quickly becoming a niche website for those with 2560x1600 monitors. It appears that the people who benefit are those few who game at 2560x1600 with 4AA at 30-35 fps.

 

[tannat]

You are right that our perception of what is playable has no impact on the benchmarks at {H}. But the way they are tested has an impact on their usefulness. Since in almost all the latest reviews, every game is tested at 30-35 fps avg at 2560x1600 w/ AA, those results tell me absolutely nothing about 1920x1080, 1920x1200, etc. and if I prefer 60 fps, I have no idea at what settings the cards can produce that either. So who exactly benefits from those benchmarks? The 10 people who game at 2560x1600 with 4AA at 30 fps? The idea is solid in that you plot on a graph the fps over the toughest part of the game. However, unless he introduces other resolutions and considers that no everyone is happy with 35 fps, I see [H] quickly becoming a niche website for those with 2560x1600 monitors only.

Most people are smart enough to realize that if it is fast enough for 2560 the HP is plenty enough for low resolutions.I would not worry too much about [H] becoming a niche review site for high resolution gaming.


Heck, you can buy a GTx580 and play at 1440x900 and still experience frame drops under 60FPs. That would be a niche review site, focusing on those problems

 

[notty22]

You are right that our perception of what is playable has no impact on the benchmarks at {H}. But the way they are tested has an impact on their usefulness. For example, if my perception of playable is 60 fps in a racing game, then I have no idea at what settings the cards can produce that in F1 2010 for example. In addition, since in almost all the latest reviews every game is tested at 30-35 fps avg at 2560x1600 w/ AA, those results tell me absolutely nothing about 1920x1080, 1920x1200, etc. So how exactly am I supposed to make a purchasing decision from those results? Kyle will also pick and choose various settings, such as running NV cards with PhysX with no AA vs. AMD cards with no PhysX with AA. How are you supposed to draw any conclusion from those results? You can't compare PhysX to AA (they are completely different visual enhancements).

The idea of plotting fps over time on a graph in the toughest part of the game is a great one. However, unless he introduces other resolutions and considers that not everyone is happy with 35 fps, the execution leaves much to be desired. I see [H] quickly becoming a niche website for those with 2560x1600 monitors. It appears that the people who benefit are those few who game at 2560x1600 with 4AA at 30-35 fps.

I agree. I'll read H reviews, but I give his results probably less weight than other sites. IMO, they (Kyle's team) spend a lot of time telling readers 'bad' things about 'canned' benchmarks and people want to believe him. Whatever, I also know there is gainful knowledge gained with more 'scientific' repeatable benchmarks also.

 

[tannat]

I agree. I'll read H reviews, but I give his results probably less weight than other sites. IMO, they (Kyle's team) spend a lot of time telling readers 'bad' things about 'canned' benchmarks and people want to believe him. Whatever, I also know there is gainful knowledge gained with more 'scientific' repeatable benchmarks also.

And thats where I disagree most strongly. Repeatability is not a scientific approach on it's own without testing for significance/relation to the things you really want to know.

Heck, noise is often one of the most repeatable parameters you have to fight in measurements. Canned benches can show a relative relation to real game play from the comparisons I have seen, as often as they really do show to be misleading. Some sites try to better this by using more games but that's like the dentist drilling in the wrong tooth.

You may call canned benchmarking more systematic i guess, but it is IMHO obviously blatantly less scientific than [H]s approach with regards to analyze data of relevance.

I don't think that [H] have perfect testing. Not at all.

However, from the point of a scientist my (slightly pointed) opinion is that I can as well try to get out significance from my telephone book as from canned benches. That's of course only true if I'm not specifically interested in how the card is performing in canned benches only.

 

[BFG10K]

Rather than having nvidia pr send a card to bfg, who is on record of preferring nvidia --- send it to me. Hell kyle and the rest have all had their cards and given their judgment. This is just nvidia trying to present an issue most don't see as being critical.

LOL, why would nVidia PR send me a Radeon? The 6850 is a review sample that was sent to Apoppin from ATi. He’s finished reviewing it, so he’s sending it to me to test.

Yes, I have a personal preference to nVidia, but I’m not biased. The GTX470 review I published caused a ruckus at nVidia HQ because the results painted a less than stellar picture.

As for the topic at hand, I’ve already commented on it many times: run both vendors at their maximum quality, and then compare the differences. I don’t run my cards at their default settings and I don’t expect anyone else to either.

Also disabling Catalyst AI is as silly as deleting all of nVidia’s profiles. The fact is that both vendors employ application detection for compatibility and/or performance reasons.

 

[Keysplayr]

LOL, why would nVidia PR send me a Radeon? The 6850 is a review sample that was sent to Apoppin from ATi. He’s finished reviewing it, so he’s sending it to me to test.

Yes, I have a personal preference to nVidia, but I’m not biased. The GTX470 review I published caused a ruckus at nVidia HQ because the results painted a less than stellar picture.

As for the topic at hand, I’ve already commented on it many times: run both vendors at their maximum quality, and then compare the differences. I don’t run my cards at their default settings and I don’t expect anyone else to either.

Also disabling Catalyst AI is as silly as deleting all of nVidia’s profiles. The fact is that both vendors employ application detection for compatibility and/or performance reasons.

I'm just looking forward to a apples to apples, factual, but non opinionated article from you. That would be great.

 

[apoppin]

LOL, why would nVidia PR send me a Radeon? The 6850 is a review sample that was sent to Apoppin from ATi. He’s finished reviewing it, so he’s sending it to me to test.

Yes, I have a personal preference to nVidia, but I’m not biased. The GTX470 review I published caused a ruckus at nVidia HQ because the results painted a less than stellar picture.

As for the topic at hand, I’ve already commented on it many times: run both vendors at their maximum quality, and then compare the differences. I don’t run my cards at their default settings and I don’t expect anyone else to either.

Also disabling Catalyst AI is as silly as deleting all of nVidia’s profiles. The fact is that both vendors employ application detection for compatibility and/or performance reasons.

Yep, you should have it next week. Hopefully we beat the Christmas rush. And i will miss my HD 6850 :\

Nvidia actually changed the wording of their campaign partly due to your articles, i think. "The world's fastest DX11 GPU" stresses the new Fermi architecture as being much faster than their last generation in new geometry-intensive tessellation-heavy games.

Me .. i am just wondering if AMD did not rush their last couple of sets of drivers in preparation for their new launch. It appears that they changed their default "HQ" to Quality to match Nvidia and it didn't work out so well.

i am still going to use HQ setting for both vendor's CP - not their defaults and i am eagerly awaiting Cat 10-12 to see what happens.

 

[ronnn]

LOL, why would nVidia PR send me a Radeon? The 6850 is a review sample that was sent to Apoppin from ATi. He’s finished reviewing it, so he’s sending it to me to test.

Yes, I have a personal preference to nVidia, but I’m not biased. The GTX470 review I published caused a ruckus at nVidia HQ because the results painted a less than stellar picture.

As for the topic at hand, I’ve already commented on it many times: run both vendors at their maximum quality, and then compare the differences. I don’t run my cards at their default settings and I don’t expect anyone else to either.

Also disabling Catalyst AI is as silly as deleting all of nVidia’s profiles. The fact is that both vendors employ application detection for compatibility and/or performance reasons.

Never said you are biased. Why would I? I certainly have preferences in my reviewers, but would like to think I am only biased against fud and vrzone.