nVidia manipulating reviews?

[thilanliyan]

Original thread is here:
http://www.xtremesystems.org/f...howthread.php?t=208261

EDIT: Here's a post from Shadowmage at Xtremesystems:
"SINCE I'M AN EX-REVIEWER, PEOPLE SHOULD READ THIS POST

It's because of NVIDIA's Reviewer's Guides. These guides contain pre-designed methods of benchmarking, including hardware configurations, driver revisions, game quality configurations, and even *SCORES*. My guess is that those sites merely just copied whatever NVIDIA's reviewer's guide told them to do.

If mods will let me (PM me please, mods, with your decision), I can post one up as proof."

I'm sure nVidia isn't the only company that does this though.

EDIT: Hopefully this is just a "preview" and not a "review"...maybe later on the sites will test a wider selection of games.

 

[chizow]

Not really sure what the problem is here. Is it a problem that reviewers are using 5-6 of the most popular, best-selling games released in the last few months? If anything this is exactly what hardware reviewers should be doing, whether or not NV suggested using them or not as these are the most relevant games that gamers are going to be most interested in.

A bigger issue would be if the results were unverifiable or inconsistent from review site to review site, which they are not. Its pretty obvious that the 180 drivers do significantly increase performance to the point the GTX 260 c216 pulls far ahead of the 4870 1GB where it did not previously with older 178 drivers.

Also, dev mandated requirements are nothing new, and I certainly don't remember anyone raising a stink about the requirements AMD set forth with the R700 preview:

R700 Preview at AT

AMD did have some stipulations for our R700 preview, they were as follows:

1. Previews can be posted any time after 12:01 am on Monday, July 14th.
2. Previews can include benchmarks of any combination of artificial tests or games up to a maximum of four.
3. As PowerPlay has not been enabled in the BIOS on your engineering sample, please stay away from Idle Power tests at this point.
4. In the interest of leaving something for the full NDA lift in August, we'd ask you to keep this high level and not go deep on the architecture at this point.

 

[Zap]

Originally posted by: chizow
I certainly don't remember anyone raising a stink about the requirements AMD set forth with the R700 preview:

That's because all the cool kids hate NVIDIA right now.

 

[nitromullet]

Officially endorsed previews or anything done under NDA can always be (and most likely has been) manipulated by the company allowing access to the goods. This is true for all products and manufacturers. At this point, you are really relying more on the integrity and reputation of the review site than anything else.

Once the products are generally available, and review sites actually have to spend their own dime for the products, you will see more independent reviews.

 

[thilanliyan]

Originally posted by: chizow
AMD did have some stipulations for our R700 preview, they were as follows:

That didn't specify WHICH games to use though. IF nV did stipulate which games and settings to use that's not right since obviously they'd choose which ones make that look best. There isn't outright proof that this happened but from those 4 sites it seems as if that IS what happened.

I think you guys might be right though...maybe this is more of a "preview" rather than a "review". I edited my OP.

 

[chizow]

Originally posted by: Zap

Originally posted by: chizow
I certainly don't remember anyone raising a stink about the requirements AMD set forth with the R700 preview:

That's because all the cool kids hate NVIDIA right now.

lol now? Checks website url....

Originally posted by: thilan29

Originally posted by: chizow
AMD did have some stipulations for our R700 preview, they were as follows:

That didn't specify WHICH games to use though. IF nV did stipulate which games and settings to use that's not right since obviously they'd choose which ones make that look best. There isn't outright proof that this happened but from those 4 sites it seems as if that IS what happened.

You're missing the point. You need to look for motive and potential gain from any backdoor dealings.

What's the motivation for AMD limiting review sample? To limit exposure of poor scaling or outright driver bugs. By limiting number of games reviewed, you're more likely going to hit commonly reviewed and popular titles, which will reflect well on your product.

What's the motivation for NV to review 5-6 of the newest, most popular titles? If they did specify those games, what's the problem with selecting 5-6 of the top 10 selling games for the last 2 months? Its not like they're calling on old bellweather games like Mass Effect or CoH. At best you're suggesting TWIMTBP works and NV cards work better with new games. At worst you're suggesting ATI's monthly driver updates are always a month behind.

 

[thilanliyan]

Originally posted by: chizow
You're missing the point. You need to look for motive and potential gain from any backdoor dealings.

What's the motivation for AMD limiting review sample? To limit exposure of poor scaling or outright driver bugs. By limiting number of games reviewed, you're more likely going to hit commonly reviewed and popular titles, which will reflect well on your product.

What's the motivation for NV to review 5-6 of the newest, most popular titles? If they did specify those games, what's the problem with selecting 5-6 of the top 10 selling games for the last 2 months? Its not like they're calling on old bellweather games like Mass Effect or CoH. At best you're suggesting TWIMTBP works and NV cards work better with new games. At worst you're suggesting ATI's monthly driver updates are always a month behind.

Like I said AMD didn't specify WHICH games to test so you're half right (as I said earlier there's no outright proof of nV limiting to certain games either). I don't really care whether there was any tangible gain from something like this but regardless it should be stated in the review (like AT did in the R700 preview) IF this is in fact true.

Also, it's not me suggesting anything (ie. this is not MY theory)...I posted to that thread and edited my post to suit. I'm not sure how you brought ATI's driver schedule into this but they actually have hotfixes as the games are released for new games don't they...not just on a monthly basis?

 

[BFG10K]

Originally posted by: chizow

Not really sure what the problem is here..

This grabbed my attention:

It's because of NVIDIA's Reviewer's Guides. These guides contain pre-designed methods of benchmarking, including hardware configurations, driver revisions, game quality configurations, and even *SCORES*. My guess is that those sites merely just copied whatever NVIDIA's reviewer's guide told them to do.

What that potentially means is nVidia supplied the benchmark scores and the review sites simply put them up (i.e. the review site never tested anything, they just put up nVidia?s scores).

I'm not sure how anyone can claim that's remotely the same thing as saying "you can only benchmark four games but I'll let you chose which four".

 

[chizow]

Originally posted by: BFG10K
What that potentially means is nVidia supplied the benchmark scores and the review sites simply put them up (i.e. the review site never tested anything, they just put up nVidia?s scores).

I'm not sure how anyone can claim that's remotely the same thing as saying "you can only benchmark four games but I'll let you chose which four".

That's the first thing I checked when I read that (they're not the same), but honestly, do you think NV and reviewers would do something as blatantly stupid as regurgitating benchmarks? The results are different between reviews but all have the GTX 260 c216 consistently beating the 4870 1gb.

If NV did suggest which titles to test while providing sample result differences it would certainly serve the same purpose as AMD dictating number of reviewable titles. Both will ultimately influence the returned results for a favorable and more predictable outcome.

 

[nRollo]

Originally posted by: thilan29

Originally posted by: chizow
AMD did have some stipulations for our R700 preview, they were as follows:

That didn't specify WHICH games to use though. IF nV did stipulate which games and settings to use that's not right since obviously they'd choose which ones make that look best. There isn't outright proof that this happened but from those 4 sites it seems as if that IS what happened.

I think you guys might be right though...maybe this is more of a "preview" rather than a "review". I edited my OP.

Here's the deal with this:

NVIDIA does provide user guides for testing hardware and software.

For example this one contained things like "Set all options to maximum, disable vsynch" in the Left4Dead testing section, and "Disable vsynch, maximize graphics options, and set mouse sensitivity to highest" for Dead Space.

They do include their benchmark scores and rigs, so you can check if you're in the ballpark and not post something way higher or lower that's misrepresentative of their products. My own averages were a hair below theirs, but very close.

I don't think it's a conspiracy or hand picked games, they just wanted to show that their new drivers raised the performance at what are sure to be some of the top selling games this holiday season. Personally I'd rather see benches, and improvements, at these games than the hot games of days gone by.

 

[ViRGE]

From what I know, everything Rollo says is true. I have a copy of the NV35 reviewer's guide in front of me, and while it's admittedly old, it's exactly as Rollo says: They list several modern games, list the best practices for benchmarking (run the test a few times, or make sure the UT2 engine isn't auto-scaling quality, etc; nothing that would benefit NVIDIA), and list what scores they got on their own setup. These scores aren't meant for publication, and I'd hope no one is dumb enough to blindly publish them as opposed to running their own benchmarks to confirm the results.

Anyhow, to my knowledge nothing has changed with regards to reviewer guides, and NVIDIA isn't doing anything to limit what reviewers can do (e.g. only allowing specific games and the like; not that they can, these drivers are publicly available), so there doesn't seem to be anything here to be worried about.

 

[thilanliyan]

Originally posted by: nRollo
Personally I'd rather see benches, and improvements, at these games than the hot games of days gone by.

Thanks for posting about how it works. I myself would like to see those and more games tested though. I suppose this is more of a driver preview than a video card review. They should have also added more than just those cards to the mix (I'm not quite sure why every site only tested with those cards).

EDIT: Overclockersclub also did the same games and settings it seems like. The review sites should really state that the games/settings were suggested by nVidia (if they were) like AT did in the R700 preview.

Here's another quote which I think sums up what one side of this argument thinks (whether this is accurate or not I really don't know):

" Originally Posted by villa1n View Post
Thats not how a fair and objective review works where they are trying to compare the cards. That is not what he s asking for either. Nvidia is tailoring the review to show the settings that the card wins at only. Insteading showing the full scope of settings, that people use it for. It is limiting the games, in the hopes of creating the idea, that these performance trends run for all games, and that these are merely a random sample of the games out.
Wombat may be a amd fanboy, just like your an intel nvidia fanboy, but what he is asking for is reviews that show the spectrum of the card. I m sure he may not be happy if the 260 216 wiped the floor at all settings and all games, but he wouldnt have much to say, if that was demonstrated. The point is, this isn't reviewing, it is marketing masked as reviewing to gain credibility and garner sales. If nvidia, or amd want to sell cards, make a better product, don't try and dupe the masses. Just look at the bush presidency. Didnt work out to well."

 

[BFG10K]

Originally posted by: chizow

but honestly, do you think NV and reviewers would do something as blatantly stupid as regurgitating benchmarks?

Not in this case, but it sounds like he?s implying it happened in the past.

I wouldn?t be at all surprised to learn some dumb reviewer in the past has posted the figures nVidia supplied them without bothering to run their own benches.

The results are different between reviews but all have the GTX 260 c216 consistently beating the 4870 1gb.

I would agree based on the scores we?ve seen so far.

Anyway, I'll be testing the drivers at some stage and you can you bet I'll test any game I please.

 

[Keysplayr]

Nvidia has always released reviewer guides. I'm sure ATI does this as well. What makes this time any different? Because they are doing so well? LOL, probably.

 

[nRollo]

Originally posted by: BFG10K
Anyway, I'll be testing the drivers at some stage and you can you bet I'll test any game I please.

I can tell you right now BFG- they did not put any time into improving Serious Sam 1 performance and you'll have to be satisfied with 234fps average at 25X16 8X16X SSAA.....



I kid, I kid- happy testing.

:beer:

 

[Keysplayr]

Originally posted by: thilan29

Originally posted by: nRollo
Personally I'd rather see benches, and improvements, at these games than the hot games of days gone by.

Thanks for posting about how it works. I myself would like to see those and more games tested though. I suppose this is more of a driver preview than a video card review. They should have also added more than just those cards to the mix (I'm not quite sure why every site only tested with those cards).

EDIT: Overclockersclub also did the same games and settings it seems like. The review sites should really state that the games/settings were suggested by nVidia (if they were) like AT did in the R700 preview.

Here's another quote which I think sums up what one side of this argument thinks (whether this is accurate or not I really don't know):

" Originally Posted by villa1n View Post
Thats not how a fair and objective review works where they are trying to compare the cards. That is not what he s asking for either. Nvidia is tailoring the review to show the settings that the card wins at only. Insteading showing the full scope of settings, that people use it for. It is limiting the games, in the hopes of creating the idea, that these performance trends run for all games, and that these are merely a random sample of the games out.
Wombat may be a amd fanboy, just like your an intel nvidia fanboy, but what he is asking for is reviews that show the spectrum of the card. I m sure he may not be happy if the 260 216 wiped the floor at all settings and all games, but he wouldnt have much to say, if that was demonstrated. The point is, this isn't reviewing, it is marketing masked as reviewing to gain credibility and garner sales. If nvidia, or amd want to sell cards, make a better product, don't try and dupe the masses. Just look at the bush presidency. Didnt work out to well."

In bold ^
Does he mean, all settings highest quality with AA/AF. VSync off, set mouse to highest sensitivity? Those settings? How would any of these settings show Nvidia in a better light over ATI? Wouldn't that be apples to apples? What settings would show the ATI card shining more? Only thing left is to reduce the quality settings, right?
So, I dunno what this guy is talking about.

 

[Keysplayr]

Originally posted by: nRollo

Originally posted by: BFG10K
Anyway, I'll be testing the drivers at some stage and you can you bet I'll test any game I please.

I can tell you right now BFG- they did not put any time into improving Serious Sam 1 performance and you'll have to be satisfied with 234fps average at 25X16 8X16X SSAA.....



I kid, I kid- happy testing.

:beer:

::sigh:: I was hoping that SpongBob Squarepants would receive a nice performance boost.
Along with Earthworm Jim.

BFG. All in good fun bud.

 

[chizow]

Originally posted by: thilan29
That didn't specify WHICH games to use though. IF nV did stipulate which games and settings to use that's not right since obviously they'd choose which ones make that look best. There isn't outright proof that this happened but from those 4 sites it seems as if that IS what happened.

Like I said AMD didn't specify WHICH games to test so you're half right (as I said earlier there's no outright proof of nV limiting to certain games either). I don't really care whether there was any tangible gain from something like this but regardless it should be stated in the review (like AT did in the R700 preview) IF this is in fact true.

Also, it's not me suggesting anything (ie. this is not MY theory)...I posted to that thread and edited my post to suit. I'm not sure how you brought ATI's driver schedule into this but they actually have hotfixes as the games are released for new games don't they...not just on a monthly basis?

I still don't see the problem here. If NV said "hey, review the top 5 PC games in terms of sales and popularity" how is that not right or somehow unfair? How are they choosing games make their cards look best? If they were cherry picking obscure games or ones that have historically favored NV parts you'd have a point but that's clearly not the case as all of the titles in question have been released in the last month or two and are at the top of the PC sales charts.

As for how this relates to AMD's restrictions on R700 testing, again it serves the same purpose as it will influence results favorably for both. For example, by limiting to 4 titles, AMD limits their exposure to driver/CF/scaling problems, as reviewers will gravitate to popular titles readers expect to see reviewed. If they set the limit to 2 games for instance, there's a 100% chance you'd see Crysis or COD4 in every review. This is a common criticism of profile-based SLI/CF, that only popular/heavily reviewed titles will get consistent support simply because they're in the public eye and highly scrutinized. AMD can focus on making sure CF drivers work for popular titles and by limiting number of titles reviewed, will ultimately have a higher % of properly scaling games in reviews.

NV is doing the same thing if they are including bench guidelines and results (as has been confirmed by both nRollo and ViRGE). If NV says "hey guys look, the last 5-6 most popular, best-selling titles all run better on our hardware" and show some gaudy benchmarks, reviewers would be more likely to review the same titles because they're probably 1) playing those games already and genuinely interested in how they bench 2) haven't benched them yet, so a driver review would pull double-duty and 3) want to see if the results are accurate and the gains are reproducable. Honestly if you had to choose which mandate was less restrictive or oppressive, I'd say its clearly NV's approach.

 

[chizow]

Originally posted by: BFG10K

Originally posted by: chizow

but honestly, do you think NV and reviewers would do something as blatantly stupid as regurgitating benchmarks?

Not in this case, but it sounds like he?s implying it happened in the past.

I wouldn?t be at all surprised to learn some dumb reviewer in the past has posted the figures nVidia supplied them without bothering to run their own benches.

The results are different between reviews but all have the GTX 260 c216 consistently beating the 4870 1gb.

I would agree based on the scores we?ve seen so far.

Anyway, I'll be testing the drivers at some stage and you can you bet I'll test any game I please.

Who's implying it happened in the past? An ex-reviewer that goes by "Shadowmage"? At some point you have to filter reliable information from the BS on various forums, and in this case, it doesn't make much sense to claim cherry-picked or regurgitated benchmarks. All the titles in question fall in the Top 10 for PC games and were released in the last two months and none from what I've seen are the same from review to review.

 

[Genx87]

Originally posted by: BFG10K

Originally posted by: chizow

Not really sure what the problem is here..

This grabbed my attention:

It's because of NVIDIA's Reviewer's Guides. These guides contain pre-designed methods of benchmarking, including hardware configurations, driver revisions, game quality configurations, and even *SCORES*. My guess is that those sites merely just copied whatever NVIDIA's reviewer's guide told them to do.

What that potentially means is nVidia supplied the benchmark scores and the review sites simply put them up (i.e. the review site never tested anything, they just put up nVidia?s scores).

I'm not sure how anyone can claim that's remotely the same thing as saying "you can only benchmark four games but I'll let you chose which four".

I would assume scores are to give a general idea of performance. So if for instance in game x you are scoring 20 fps when Nvidia claims 80. There may be an issue with the setup or the card. If any review site is dumb enough to reprint Nvidia's benchmark results they deserve a public lashing.

 

[apoppin]

Originally posted by: nRollo

Originally posted by: thilan29

Originally posted by: chizow
AMD did have some stipulations for our R700 preview, they were as follows:

That didn't specify WHICH games to use though. IF nV did stipulate which games and settings to use that's not right since obviously they'd choose which ones make that look best. There isn't outright proof that this happened but from those 4 sites it seems as if that IS what happened.

I think you guys might be right though...maybe this is more of a "preview" rather than a "review". I edited my OP.

Here's the deal with this:

NVIDIA does provide user guides for testing hardware and software.

For example this one contained things like "Set all options to maximum, disable vsynch" in the Left4Dead testing section, and "Disable vsynch, maximize graphics options, and set mouse sensitivity to highest" for Dead Space.

They do include their benchmark scores and rigs, so you can check if you're in the ballpark and not post something way higher or lower that's misrepresentative of their products. My own averages were a hair below theirs, but very close.

I don't think it's a conspiracy or hand picked games, they just wanted to show that their new drivers raised the performance at what are sure to be some of the top selling games this holiday season. Personally I'd rather see benches, and improvements, at these games than the hot games of days gone by.

agreed

they are testing guidelines .. and they do not have to be followed to the letter

and the reviews are as good as the reviewer; the integrity comes from the tech site itself
- some reviewers *return* their HW samples; others keep them; other sites refuse to accept any HW but buy all of their own; it is up to the site how they do it

the "ex-reviewer" is seeking attention ,, nothing new or shocking or immoral or unethical
- except perhaps the 'ex'

 

[geoffry]

Anand shows 0 performance increase

http://anandtech.com/video/showdoc.aspx?i=3462&p=4

 

[chizow]

Originally posted by: geoffry
Anand shows 0 performance increase

http://anandtech.com/video/showdoc.aspx?i=3462&p=4

Actually the other Release 180 reviews don't show performance increase/decrease, they show relative performance between parts with the latest drivers. Also, the games tested by AT aren't mentioned as games that show improvement in the patch notes and they aren't the top 5 titles used in other reviews. They might've been addressed in a previous patch, but its hard to say since AT's last full review is still using launch 177.34 drivers. These are the kind of pitfalls and bad habits a review site gets into when using archived results from review to review.

 

[geoffry]

Originally posted by: chizow

Originally posted by: geoffry
Anand shows 0 performance increase

http://anandtech.com/video/showdoc.aspx?i=3462&p=4

Actually the other Release 180 reviews don't show performance increase/decrease, they show relative performance between parts with the latest drivers. Also, the games tested by AT aren't mentioned as games that show improvement in the patch notes and they aren't the top 5 titles used in other reviews. They might've been addressed in a previous patch, but its hard to say since AT's last full review is still using launch 177.34 drivers. These are the kind of pitfalls and bad habits a review site gets into when using archived results from review to review.

Ah, when I saw the TR link to the XS boards talking about big time performance gains and then I seen Anand's results I was a bit shocked.

 

[BFG10K]

Originally posted by: nRollo

I can tell you right now BFG- they did not put any time into improving Serious Sam 1 performance and you'll have to be satisfied with 234fps average at 25X16 8X16X SSAA.....

Actually I'm thinking of testing other modern games, like Stalker Clear Sky. I'd be happy with more performance in that one as I find the game is almost as demanding as Crysis, if not more-so.

Originally posted by: Chizow

Who's implying it happened in the past? An ex-reviewer that goes by "Shadowmage"? At some point you have to filter reliable information from the BS on various forums, and in this case, it doesn't make much sense to claim cherry-picked or regurgitated benchmarks. All the titles in question fall in the Top 10 for PC games and were released in the last two months and none from what I've seen are the same from review to review.

The fact that they?re top 10 titles doesn?t mean anything. What, you think nVidia provides figures for ancient games?

Again, I already stated I don't believe it's happening for the Big Bang driver reviews. But do I think it's happened in the past? Sure. I wouldn?t be surprised there were reviewers out there that simply reposted nVidia's figures. And ATi?s too, if they provide such figures.