Benchmarkers need to stop using biased games when testing GPUs.

[RavenSEAL]

First off, let me just make it clear that both nVidia and AMD are great manufacturers.

But honestly, with the purchase of a new video card 2 days ago, i have been doing a lot of research around the web.

I stumbled upon several websites sporting their benchmarking test all over the page...But the one thing i have noticed, is that fact that they actually dare to compare cards head-to-head, on games that are optimized for nVidia GPUs.

Now, you might be saying..."OMG, here we go, AMD fanboy".

Wrong, i have owned about 6 video cards, split even between both AMD and nVidia( Til' now)

ATI:

ATI 9800
ATI HD4870
ATI HD5670
Now AMD HD6850

As far as nVidia:

nVidia 8800GT
nVidia 9600GT
GTX260
GTS450

Now, i'm gonna go ahead and admit that AMD needs to pull their heads out of their *** and fix the Catalyst drivers, period.

nV is light years ahead of AMD as far as drivers. This is where the Radeon cards get destroyed, even though the hardware it self seems to be superior. I personally believe that with properly coded drivers, AMD cards might even start beating the nV card on the same market level, BUT we can only hope...

But the point is, why do these so called "Pro" benchmarkers actually bother testing games/software that actually takes advantage of nVidia's proprietary architecture?

I personally don't see anything wrong with nVidia going to developers and saying "Here, we'll give your our architecture to take advantage of, we'll pay you a little bit and just make sure you put our name on the game"

But honestly, we all know all the damn nVidia cards are going to beat out the AMD cards! These games were made from scratch using nVidia's architecture!

Why waste your time doing that when the results are already known as soon as you boot up the game?!?! If you haven't notice, a loud *** "nViiiiiiiidaaaa" goes off as soon as you start the game, following by a funky "flashy" movie with green stripes all over the screen.

So, come on, people need to start taking off all those games that nVidia has clearly paid to see their name all over and start using more software that truly benchmarks the GPUs regardless of manufacturers.

Just my two cents...

 

[happy medium]

So, come on, people need to start taking off all those games that nVidia has clearly paid to see their name all over and start using software that truly benchmarks the GPUs regardless of manufacturers.

OR
AMD should get of there A$$es ,get with the program,and start to work better with game developers mabe? Just a thought.

 

[Wreckage]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

 

[GaiaHunter]

Generally in those biased games the extra FPSs one company gets are FPSs that don't improve gaming experience at all - "wow 120fps vs 100 fps big crushing win. Pity my monitor refresh rate is 60Hz*".

For example what games can the GTX460 play that the 6850/70 can't at similar resolutions/IQ? And you can do the same for most of the card tiers.

*I know some people have monitors with higher refresh rates but those aren't common - of course distinguish individual frames when you are at 60+ fps is a tad hard.

 

[Barfo]

It seems to me that most games run better either on Radeons or Geforces, obviously nvidia will try to get reviewers to use the games that favor their cards, and AMD will do the same.

 

[Barfo]

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

Those two games look much better than GT5 with my AMD card, perhaps nvidia has IQ issues if they look crappy for you.

 

[happy medium]

Generally in those biased games the extra FPSs one company gets are FPSs that don't improve gaming experience at all.

For example what games can the GTX460 play that the 6850/70 can't at similar resolutions/IQ? And you can do the same for most of the card tiers.

Hawx 2?

 

[nitromullet]

What games not on either of these lists do you really want to see bench marked?

http://in.nzone.com/object/nzone_twimtbp_gameslist_in.html
http://sites.amd.com/us/game/games/Pages/games.aspx

 

[tviceman]

Every single game that doesn't perform within the normal average of most games respective to the graphics cards being tested could be considered bias. F1 could be considered bias against Nvidia cards. Hawx 2 could be considered biased against AMD cards. But the fact is these are actual programs, and "biased" or not, they are representative of how these cards will perform within these games.

Months and months before Fermi came out, I'm sure Nvidia fans thought that Dirt 2 and Stalker: CoP would be biased since they were DX11 games heavily marketed by AMD, but as it turns out those games typically perform better on Nvidia based cards. Similarly, Mafia II is a TWIMTBP game yet when physx isn't being used the game typically performs better on AMD cards. So honestly, how are you to decide what is biased and what isn't?

 

[Lonyo]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

Name another recent PC racing game.
Name another DX11 game (that doesn't already get benchmarked).

Sites test games for typically one or more of four reasons.
1) It's a modern game, <1 year old probably
2) It's a very popular game or uses an engine that people play lots of games based on (e.g. Left 4 Dead 2 for Source)
3) It's one of the most modern examples of its genre
4) It's a DX11 game.

Dirt 2/F1 2010 falls under 1, 2 and 4.
Unless you have a suggestion for a different racing game they should be benchmarking, or a different DX11 game they should be benchmarking?

This also relates to the OP.
Benchmarks are based around the most demanding/useful/feature heavy games, typically. There's no point in benchmarking cards on older games where all cards get 100+ FPS really. They try and focus on newer games which are more demanding, and use things like DX11, or are based on popular engines. There's no bias in selecting games in such a manner, it makes the most sense. This leads on to what nitromullet points out. Most advanced/popular games are "supported" in some fashion or another by AMD or NV, so selecting ones which aren't basically rules out most relevant titles.

 

[Rifter]

I think they should benchmark whatever the new games are regardless of if they run better on red or green fuel. If your game plays better on one than the other you should know that so you can buy the appropriate card.

Agree about the drivers, ATI needs improvement, especially in non windows OS's.

OR
AMD should get of there A$$es ,get with the program,and start to work better with game developers mabe? Just a thought.

Could not agree more !

And yeah Wreckage since i just bought GT5 today and put in a few hours and own dirt2 and F1 i can tell you that while GT5 does look awesome its just not on the level of the top tier PC racing games. At the end of the day it still has to run on PS3 hardware.

 

[notty22]

I argue against this statement often, "nvidia biased game".
Thats just a excuse imo, to discount the really few little differences between video cards.
There are only 2 companies, and we all have to play the same PC games.
So if certain games make one card shine, I'm sorry to me, thats just a big PRO for that card.
Popular games usually make in to reviewer test suites.

Dirt 2 and F1 use the same game engine, but its obviously tweaked in F1 and AMD cards seem to be getting higher fps.
Is that now a AMD bias game ?
No, I don't think so. It might be a pro, or a reason to buy a AMD card if you really want to get in to that game. Of course it will play fine on Nvidia cards, just like TWITBP games play fine on AMD cards.

 

[DaveSimmons]

I disagree with the OP. Knowing that (popular game X) runs much better on (GPU family) is useful, especially if you might play that game, regardless of why it happened.

Just because it was sleazy of a game developer to accept cash or services to favor one GPU family over another doesn't change the fact that if you buy the "wrong" card that game will run worse for you.

I wouldn't mind if the reviewers added disclaimers, such as "this game accepted 'assistance' from AMD to improve the DX11 performance on only Radeon cards" or "nvidia's suitcase full of cash made them add useless tessellation that heavily favors the 4xx/5xx cards."

Also, the reviews should include a mix of games not just the "biased" games.

 

[Unkle_Tar]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

GT5 looks worse than Dirt 2 on console, let alone on PC.

 

[Troll Trolling]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

Come on Wreckage! You were so close of finally writing a 100% unbiased and rational comment.

I agree with your first paragraph though.

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

 

[OCGuy]

The games are still games. AMD really needs to work with more developers to encourage the growth and ensure the sustainability of PC gaming.

Without PC games, nobody is shelling out $1K for a 580 SLI system, and $250 for the PSU to run it.

Without a constant "need to upgrade" mentality, people will sit on their CPUs/Mobos/RAM longer as well.

Basically I am trying to say that a lot is riding on a healthy PC gaming market. It would suit all of us if MS, AMD, nV, Intel, and all of the AIBs did more.

There are no bias games....

 

[-Slacker-]

I disagree with the OP. Knowing that (popular game X) runs much better on (GPU family) is useful, especially if you might play that game, regardless of why it happened.

Just because it was sleazy of a game developer to accept cash or services to favor one GPU family over another doesn't change the fact that if you buy the "wrong" card that game will run worse for you.

I wouldn't mind if the reviewers added disclaimers, such as "this game accepted 'assistance' from AMD to improve the DX11 performance on only Radeon cards" or "nvidia's suitcase full of cash made them add useless tessellation that heavily favors the 4xx/5xx cards."

Also, the reviews should include a mix of games not just the "biased" games.

^Yup. What he said.

 

[NoQuarter]

Just need a game review site that benches a sampling of graphics cards as part of the game review, instead of a graphics card review site that benches a sampling of games as part of the graphics card review.

 

[taltamir]

I disagree with the OP. Knowing that (popular game X) runs much better on (GPU family) is useful, especially if you might play that game, regardless of why it happened.

Just because it was sleazy of a game developer to accept cash or services to favor one GPU family over another doesn't change the fact that if you buy the "wrong" card that game will run worse for you.

I wouldn't mind if the reviewers added disclaimers, such as "this game accepted 'assistance' from AMD to improve the DX11 performance on only Radeon cards" or "nvidia's suitcase full of cash made them add useless tessellation that heavily favors the 4xx/5xx cards."

Also, the reviews should include a mix of games not just the "biased" games.

all true.. furthermore, if review sites did not benchmark "biased games", they will have no games to benchmark at all.
Lets face it, the majority of games are made using older technology and can easily be maxed out on outdated hardware... the ones actually worth benchmarking are the ones who push the envelope on what can be done, and there are only a handful of those a year, and all of them "biased" as you call it.

besides which, its only sleazy if nvidia pays them to sabotage AMD performance or vice versa... there is nothing wrong with nvidia helping a game run better on nvidia hardware and amd helping a game run better on AMD hardware.

I think the batman thing comes to mind... AMD claims that nvidia paid them to sabotage AMD by preventing it from running AA on AMD cards... that would be wrong if true.

nVidia claims that they provided code to run AA on nvidia cards because the game developer did not want to spend the money writing their own code to do so. And when the developer contacted AMD telling them what nvidia did and asking them to provide code to run AA on AMD hardware AMD refused. This means that nVidia didn't do anything wrong...

now, which one of them is telling the truth? Hell if I know.

 

[WelshBloke]

Meh, you have to benchmark what people are playing otherwise it becomes a fairly pointless academic argument.

 

[shangshang]

Meh, you have to benchmark what people are playing otherwise it becomes a fairly pointless academic argument.

yep. Bench what the mass is playing.

AMD needs to stop being a cheapass and spend some money on game developers. And IMO, money spent on game developers, the very people connecting the hardware makers and the entertainment crowds,.. should be encouraged and applauded. For this reason alone, I praise NV. NV put their money where it counts, not like stupid cheap AMD.

 

[rgallant]

-if the reviewers want to spend 1 min. per game per card and not even play the games with said cards\settings[ 1 hr. ]. I 'am not sure if any so call real reviews[gaming] are any better than the fake reviews ,including tweaked code or not.
-you can have any ,game ,card crash 3 times a night ,then turn the settings lower and not crash ,-so said card can't play said game at said settings - so how does a 1 min, bench mark = the faster\better card ? for any buyer.
-most of us look at benches [site gets hits $$$] as we don't have access to more than the video cards we own . yet all reviews post ,winners and losers base on price not game play@settings+tweaked code = sad [BF2 BC comes to mind]

 

[Powermoloch]

Nvidia:
FX5200>6600>7600GT>XLR8 250GTS
Ati:
X850XT PE>HD3650>HD4650>2xHD5770>2xHD6850

I just buy cards that are the best bang of my buck. So far the ATi takes the cake, but I like Nvidia drivers...but not too much.

I go for simplicity

 

[Mistwalker]

I say just benchmark the latest and greatest games regardless of politics. If those games happen to play better on NVIDIA cards, well then maybe you should get an NVIDIA card. Otherwise you just asking for brand favoritism.

I couldn't agree m--

Although, some sites try to help AMD by including Dirt 2 or F1, even though they look worse than a PS3 racing game like GT5.

--and you just couldn't resist throwing your credibility away.

Anyway, bench what people are playing. Try to have a variety of genres represented. The end.

If Nvidia is actually spending money and resources so developers take advantage of their cards, that's a value add for Nvidia owners.

 

[happy medium]

If Nvidia is actually spending money and resources so developers take advantage of their cards, that's a value add for Nvidia owners.

:thumbsup: