Benchmarkers need to stop using biased games when testing GPUs.

[BFG10K]

I benchmark the games I actually play. Does that make them biased?

Additionally, the settings used are in most cases the settings I use (or would use) when I game on the cards.

 

[Wreckage]

I couldn't agree m----and you just couldn't resist throwing your credibility away.

How did I throw my credibility away? AMD dumped a bunch of money on Dirt 2. Even had it delayed for their code. Both companies do it. Rumor has it that Crysis 2 may come with heavy tessellation and physx.

 

[buckshot24]

How did I throw my credibility away? AMD dumped a bunch of money on Dirt 2. Even had it delayed for their code. Both companies do it. Rumor has it that Crysis 2 may come with heavy tessellation and physx.

So if Crysis 2 gets benchmarked in future video card reviews then they are merely trying to help nVidia?

 

[solofly]

 

[Mistwalker]

How did I throw my credibility away?

First, you said politics should be left out of game selection for benchmarking. Sounds pretty reasonable.

You then followed by saying Dirt 2 is included by sites "to help AMD" implying AMD needs help or there aren't very valid reasons to include the title on its own merits (first DX11 racing title, extremely popular, argued by many to be the best looking racer even today, etc.). Finally, you attacked the game itself for looking no better than a PS3 game.

Do you see how singling out Dirt 2 (and AMD) like this is very much political, and contradictory to your first statement?

Both companies do it.

And here is plain and simple truth, unfettered by opinions or agenda. No need for anyone to be defensive about it.

 

[digitaldurandal]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

Wreckage if you believe that Dirt2 looks worse than GT5 you need to go check your brain.

 

[Gloomy]

Wreckage if you believe that Dirt2 looks worse than GT5 you need to go check your brain.

If I were him I would get a copy of the game before I resort to that. :sneaky:

 

[footballrunner800]

Come on Wreckage! You were so close of finally writing a 100% unbiased and rational comment.

I agree with your first paragraph though.

Thats exactly what i was going to say!!

 

[dust]

I benchmark the games I actually play. Does that make them biased?

Additionally, the settings used are in most cases the settings I use (or would use) when I game on the cards.

The following does not apply to you at all, because I know you actually like playing games:

Do you think every other reviewer does the same?

What's the general criteria for cherry picking the games to be benched anyway?

I always thought the games were added to the reviewers suite because of the demand from the gamers community. How many copies must sell for a game to become benchmark. I know HAWX sold somewhere North of 1 million copies on 4 platforms if I'm not mistaken. I couldn't find a suitable statistic for the HAWX2 yet,but I wonder what's all this fuss about it as it's not my type of game at all. If the game has sold in impressive quantity then by all means, let it be benched, if not, I'm sure there are other games that are waiting in line.

Just because the game is new, it doesn't make it good too, so why bother with it anyway?

The same story is probably applying to Farcry2. I personally liked the game but it got a huge BOO from the gaming community and yet it was on the lips of every other reviewer out there.

 

[RussianSensation]

When NV's cards are faster (i.e., Lost Planet 2, Metro 2033 with tessellation, STALKER: Sun Shafts, Dirt 2, Hawx 2), the benchmark is automatically labelled as biased. When AMD's cards are faster (i.e., F1 2010, Mafia II without PhysX, tons of benches at 2560x1600 8AA), AMD cards are viewed as superior.

I benchmark the games I actually play. Does that make them biased?

Biased: "a systematic as opposed to a random distortion of a statistic as a result of sampling procedure."

In other words, testing 10 OpenGL FPS games (some of which are 5+ years old) is not representative of a random sample size. We still enjoy your benches anyways.

 

[Spikesoldier]

this may apply to both camps but i think this is true for NV:

the reasons why you see a lot of games reviewed with a heavy advantage is because as terms of supplying a review site with hardware, the reviewer agrees to run a list of predetermined games to bench.

i would figure that with the minimum list of games they MUST review, that a lazy reviewer would say well thats enough and just go with that list.

 

[Leadbox]

Those of you saying that AMD should do the same and 'help' game developers, how exactly do you see this working? Do they approach the same game developers that Nvdia does with 2 million bucks and a couple of software engineers or perhaps go to others, try and get in before Nvidia does. If they approach the same developers, obviously they have a different idea of how certain effects should be rendered by leveraging the strengths of their gpus with the reverse being true for Nvidia. How does this pulling and tugging help pc gaming? Lets not forget that the ubisofts, the EAs, cryteks and edios' of this world are multi million dollar giants

 

[Lonyo]

When NV's cards are faster (i.e., Lost Planet 2, Metro 2033 with tessellation, STALKER: Sun Shafts, Dirt 2, Hawx 2), the benchmark is automatically labelled as biased. When AMD's cards are faster (i.e., F1 2010, Mafia II without PhysX, tons of benches at 2560x1600 8AA), AMD cards are viewed as superior.



Biased: "a systematic as opposed to a random distortion of a statistic as a result of sampling procedure."

In other words, testing 10 OpenGL FPS games (some of which are 5+ years old) is not representative of a random sample size. We still enjoy your benches anyways.

That's not biased in the way the OP is claiming though, so it's fine. He's testing games he plays, which doesn't discriminate against AMD or NV supported games, so with regards to NV vs AMD it is a random sample, even if it's biased in other ways.

 

[SirPauly]

Personally expect the latest and greatest titles to be sponsored and desire more to be -- because it shows to me that the IHV's are out there trying to improve PC gaming. PC gaming is much more than a benchmark but the experience the platform creates.

 

[RussianSensation]

That's not biased in the way the OP is claiming though, so it's fine. He's testing games he plays, which doesn't discriminate against AMD or NV supported games, so with regards to NV vs AMD it is a random sample, even if it's biased in other ways.

But besides Hawx 2, what games are biased? Obviously certain game engines will run faster on certain hardware. Right now NV is still faster by a bit in DX11 games, while AMD has an advantage in CF scaling and performs better at 2560x1600 8AA. Some games are faster on AMD and some on NV. That's the point of choosing a videocard - pick based on the games you play. :thumbsup:

I find it amusing that there always has to be a "winner". As far as I am concerned for example, HD6970 and GTX570 are equally good (the 570 is quieter, but the 6970 has other cool features like multi-monitor support and 2GBs of Ram). Toss a coin. Instead our forums are still filled with arguments about these 2 cards...I just don't get it.

 

[SirPauly]

I don't understand the logic of winning unless one has a camp mentality and a desire to see one win over the other. The consumer does win when there is compelling choice to consider to buy to me, It is this strong competition that creates differentiation and value and may be enough choice to match what is important to the complex individual and their subjective tastes and different wallet sizes.

Even with compelling choice, value, market share around 50/50, some still see agendas, bias, because their company of choice is getting a perceived bad rap.

 

[Lonyo]

But besides Hawx 2, what games are biased? Obviously certain game engines will run faster on certain hardware. Right now NV is still faster by a bit in DX11 games, while AMD has an advantage in CF scaling and performs better at 2560x1600 8AA. Some games are faster on AMD and some on NV. That's the point of choosing a videocard - pick based on the games you play. :thumbsup:

I find it amusing that there always has to be a "winner". As far as I am concerned for example, HD6970 and GTX570 are equally good (the 570 is quieter, but the 6970 has other cool features like multi-monitor support and 2GBs of Ram). Toss a coin. Instead our forums are still filled with arguments about these 2 cards...I just don't get it.

According to the OP, biased games are any which have some form of support from either NV or AMD, which is most games on major engines/most DX11 games/most new big titles. Which is most of the games being benchmarked, hence the whole bias thing being stupid.
Basically the OP doesn't want any relevant games to be benchmarked.

 

[Grooveriding]

this may apply to both camps but i think this is true for NV:

the reasons why you see a lot of games reviewed with a heavy advantage is because as terms of supplying a review site with hardware, the reviewer agrees to run a list of predetermined games to bench.

i would figure that with the minimum list of games they MUST review, that a lazy reviewer would say well thats enough and just go with that list.

This is entirely true. A good example is sites that still bench Far Cry 2. A historically NV game. I roll my eyes every time I see a recent review using Far Cry 2 and generally stop reading the review at that point or looking on it skeptically.

The game is ancient at this point and a $100 card can run it maxed out, but you still see reviews using that turd because NV has it in their list.

 

[Wreckage]

The reason games bench better on NVIDIA cards is that NVIDIA has better cards. Occam's razor.

 

[RussianSensation]

The reason games bench better on NVIDIA cards is that NVIDIA has better cards. Occam's razor.

True from a tessellation perspective, but you have to look at the full picture.



So NV has a better card than the $155 HD6850?
So NV has a better card than the $215 HD5870?
So NV has a better card than the $299 HD6950?
So NV has a better card than the $369 HD6970 for 2560x1600?

 

[shangshang]

Those of you saying that AMD should do the same and 'help' game developers, how exactly do you see this working? Do they approach the same game developers that Nvdia does with 2 million bucks and a couple of software engineers or perhaps go to others, try and get in before Nvidia does. If they approach the same developers, obviously they have a different idea of how certain effects should be rendered by leveraging the strengths of their gpus with the reverse being true for Nvidia. How does this pulling and tugging help pc gaming? Lets not forget that the ubisofts, the EAs, cryteks and edios' of this world are multi million dollar giants

In the business world, money talks. AMD has been making a BIG profits over the last year because of their superior 4xxx and 5xxx series cards, right? (or so everyone here thinks so).

Time to put that profits to work! Money talks. NV puts out $2 million? So what. AMD put out $3 million then! Money is the great motivator and mover here. That's the kind of world I'm accustomed to.

..and this thread has all the tone of "yet another AMD fanboy whining about unfair competition from NV because NV dumps tons of money into developers' laps.." Well that's how the real world works. No free rides, eh. Want free rides? Well there's linux and opengl, lol.

 

[apoppin]

This is entirely true. A good example is sites that still bench Far Cry 2. A historically NV game. I roll my eyes every time I see a recent review using Far Cry 2 and generally stop reading the review at that point or looking on it skeptically.

The game is ancient at this point and a $100 card can run it maxed out, but you still see reviews using that turd because NV has it in their list.

i roll my eyes when i see biased and FUD-filled posts like these and everyone should stop reading them.
:thumbsdown:

A historically Nvidia game?
- it is on AMD's benching list; they bench it.

Ancient?
- it is two years old

A$100 card can run it?
- try 2560x1600 with 8xAA

A turd?
- it has a metacritic score of 85

Review sites are biased that use it?
- HardOCP praised the benchmark and they claim they had input into it's creation

Enough FUD about Far Cry 2

 

[Wreckage]

So NV has a better card than the $155 HD6850? Yep, it's called the 460 1gb
So NV has a better card than the $215 HD5870? Yep, it's called the 470
So NV has a better card than the $299 HD6950? See above
So NV has a better card than the $369 HD6970 yep the 570

There ya go.

 

[Gloomy]

There ya go.

You can't be serious. The 6950 is much more robust than the 470, and more expensive. The 470 does not answer the 6950, it answers the 6870, which is similarly priced and offers similar performance.

The 6950 is on its own right now, priceerformance-wise.

 

[Troll Trolling]

So NV has a better card than the $155 HD6850? Yep, it's called the 460 1gb
So NV has a better card than the $215 HD5870? Yep, it's called the 470
So NV has a better card than the $299 HD6950? See above
So NV has a better card than the $369 HD6970 yep the 570

There ya go.

460 1GB is not faster. It is cheaper. And there is a reason to the high HD 6850 price be right: You don't lower your price if you are selling very well.
470 is not on pair with the HD 5870, you may distort in any way you want, in real games the Radeon is faster.
HD 6950 is faster than the HD 5870, therefore, faster than the GTX 470, although it is more expensive.
HD 6970, well, here you are right, they are about the same speed, but the Radeon is more expensive :|