What would you like to see in a graphics card review.

[Tempered81]

something completely comprehensive that shows a full comparison of nvida & ati at stock and overclocked speeds using the exact same in-game settings (unlike hardocp) and on similar nvidia/amd platforms for an even duel.

something like this, but using and overclocking more cards for the comparison:
http://www.jmax-hardware.com/i...&limit=1&limitstart=13

yes it's time consuming, buy you asked

 

[Acanthus]

CPU Scaling when a new card is reviewed.

Especially multi GPU reviews.

 

[panfist]

I would like the benchmarking methods to be detailed so that I can repeat the same test on my system. If I need to download any demos or whatnot, they should be available. I know that reviewing a video card isn't a science, but the whole point if scientific rigor is to only publish repeatable results.

Publishing minimum framerates is also a must. Who cares if I can get over 60fps on average if it dips to 20 every time I shoot? Or turn?

Also, at least one set of benchmarks should compare whatever cards are the main subject of the review to some older cards. For example I just upgraded from an x1900XT last week and it was a huge pain in the balls trying to figure out what kind of performance gains I would get.

It would be nice if the benchmarking method used from review to review was consistent so that if you compare a chart published on March 30th to one published on January 30th, the numbers would line up as you would expect. At least for one set of benchmarks per review. And 3dmark doesn't count. Granted, a lot can change in the market in just two months, and sometimes it doesn't make sense to review a new enthusiast (or budget) video card with the same settings as you reviewed a video card last month, but as much as possible should stay the same, for consistency's sake. I like it when I see an apples to apples comparison, but it would be nice to get apples to apples to apples to apples, between different reviews.

On the other hand, lately I've been liking and trusting the reviews on hardocp more and more. Apples to apples is nice, but it's also nice to get an editorial perspective saying, "here is the best gameplay experience you will get with this card, and this is how it compares to the best gameplay experience you will get with this other card."

Lastly, this site does a really good job of comparing stock clocked cards to overclocked cards, but in reality not everyone will be able to achieve the same overclocks. How about calculating some kind of performance increase index, that says clock for clock this is how much framerate increase (both minimum and average) you would expect to get per incremental clock increase. Also, does a maximum overclock result in a better gameplay experience, like is it smoother? can you enable higher settings?

I've noticed that, especially nvidia and since about 3 years ago, graphics cards have different specifications other than clock speed which have more of an impact on performance than any overclock you can possibly achieve. For example the number of ROPs or texturing units. A few years ago the manufacturers would have the same exact chip at a few different clock speeds and price points. Now they just release "OC" versions at different price points. It really doesn't matter to me because I'm going to make my decision based on real-world performance, not model numbers or clock speeds; however, it would be nice to see some editorial perspective, like "no matter how high the overclock is the real-world benefit is negligible," or "those gamers interested in playing this game and this resolution will get an increase in value for higher clocked cards."

 

[BFG10K]

Edit: sorry, wrong thread.

 

[BFG10K]

• (For high-end boards), mandatory 16xAF with 4xMSAA and 8xMSAA, plus TrMS or TrSS depending on the game. I don?t want to see AA-less benchmarks unless AA is not possible AA in the game.
  
• (For high-end boards), mandatory 16xx x YYYY or better resolutions.
  
• No 3DMark reviews. Please stop wasting time and space on junk like this.
  
• A mix of classic (e.g. Doom 3, Far Cry, etc) and new titles, including reporting driver bugs. After running a benchmark fire up a few levels in the game and have a quick run around to see if any visible driver bugs are present.
  
• Don?t publish benchmarks of manual runs through games but instead always use integrated playback.

 

[BFG10K]

drivers dont increase performance that much

I disagree with this. Drivers can make a huge difference to performance especially if there was a driver bug holding back performance to begin with.

 

[Acanthus]

Originally posted by: BFG10K

drivers dont increase performance that much

I disagree with this. Drivers can make a huge difference to performance especially if there was a driver bug holding back performance to begin with.

I agree with BFG, i have seen 45%+ gains from drivers in the past.

 

[BFG10K]

Aye, in my 8800 GTS testing I went from 59 FPS to 229 FPS in Star Trek Elite Force after a driver update.

This is the most extreme example I?ve seen but it shows what a huge difference drivers can make.

 

[Keysplayr]

Lots of great info here guys. Keep em coming. I'm going to condense all the comments in a short while. Just want to give this thread some more time to grow.

Cheers!

 

[MustangSVT]

maybe if you use a demo version or a test that we can run on our own machines to see how our systems compare with the new cards?

 

[Keysplayr]

Originally posted by: MustangSVT
maybe if you use a demo version or a test that we can run on our own machines to see how our systems compare with the new cards?

That would be great. But in some games, I can't create timedemo's. CoD4 for example. You can record a demo, but the playback function is broken. Numerous sites report crashing to desktop when trying to playback a recorded demo. /demo or /timedemo.

And in Bioshock. No way to record a demo that I know of.

Crysis, STALKER and CoD2 work fine when recording and playing back our own timedemo's.

HardwareOC has some benchmarking utilites for some older games like FarCry and CoD2. I wish they would keep up. LOL.

 

[apoppin]

Originally posted by: keysplayr2003

Originally posted by: MustangSVT
maybe if you use a demo version or a test that we can run on our own machines to see how our systems compare with the new cards?

That would be great. But in some games, I can't create timedemo's.
And in Bioshock. No way to record a demo that I know of.

LOL.

Piece of
... damn ... no cake!

You wanna know how?
:gift:
-i just recorded a nice one of BioShock - just to see - just now. .
--of course you have to use the Full Version of FRAPS

 

[LOUISSSSS]

don't know if its possible but... uploaded videos of the recording (if used) with a program like FRAPS showing the avg/min FPS.

of course... testing should be done on a PC that THE MAJORITY of people own... such as an 5000 X2, E6600, E8400, or Q6600. and run them at stock speeds and also at a mild overclock, like 200-400mhz+ just to show if a higher clock speed would matter.

 

[nevbie]

Originally posted by: BFG10K

• (For high-end boards), mandatory 16xAF with 4xMSAA and 8xMSAA, plus TrMS or TrSS depending on the game. I don?t want to see AA-less benchmarks unless AA is not possible AA in the game.
  
• (For high-end boards), mandatory 16xx x YYYY or better resolutions.

I disagree. It is not a satisfying comparison if you change benchmark settings based on expected results. I want to see AA-less benchmarks of higher-end cards compared to AA-less benchmarks of not higher-end cards.

Yes, stressful settings are results too. But it is more important to be able to do a direct comparison between cards.

 

[BFG10K]

CoD4 for example. You can record a demo, but the playback function is broken. Numerous sites report crashing to desktop when trying to playback a recorded demo. /demo or /timedemo.

Well actually it does work, you just have to use the MP executable and load the demo as a mod.

It is not a satisfying comparison if you change benchmark settings based on expected results.

Changing settings based on hardware is exactly what reviewers should do. Unfailingly running a battery of tests at the same settings regardless of hardware isn't necessarily useful if you fail to capture the differences you?re looking for.

I want to see AA-less benchmarks of higher-end cards compared to AA-less benchmarks of not higher-end cards.

AA-less benchmarks on high-end boards generally add CPU/platform limitations and at that point you?re testing more than graphics cards. Again it's all about customizing based on the hardware being tested.

You wouldn't (for example) test CPUs at 2560x1600 so why would you run GPU tests at low detail settings?

 

[waffleironhead]

When the folding@home group release their new graphics based folding app with support for latest ati cards. I would like to see folding benches in the reviews.
Right now they only support ati x16xx-x19xx series so its a no go for the moment, but hopefully in the future.

Looks like the new client may be out in the near future.(crosses fingers)

 

[Keysplayr]

Originally posted by: BFG10K

CoD4 for example. You can record a demo, but the playback function is broken. Numerous sites report crashing to desktop when trying to playback a recorded demo. /demo or /timedemo.

Well actually it does work, you just have to use the MP executable and load the demo as a mod.

Keys : Thanks BFG. I have found that the demo has to be inside a "demos" folder, within the "modwarfare" folder. And it does work. The demo then runs, albeit at the speed of light, hehe. But it runs! :thumbsup:
EDIT: Ah, the /timedemo command runs it at warp 9, but the /demo command runs it normally. I just need to use FRAPS for the framerates. Cool.

It is not a satisfying comparison if you change benchmark settings based on expected results.

Changing settings based on hardware is exactly what reviewers should do. Unfailingly running a battery of tests at the same settings regardless of hardware isn't necessarily useful if you fail to capture the differences you?re looking for.

I want to see AA-less benchmarks of higher-end cards compared to AA-less benchmarks of not higher-end cards.

Keys : Benchmarks can be ran with and without AA. The reader can then draw what conclusion they need from it. It only takes a few extra runs to include both.

AA-less benchmarks on high-end boards generally add CPU/platform limitations and at that point you?re testing more than graphics cards. Again it's all about customizing based on the hardware being tested.

You wouldn't (for example) test CPUs at 2560x1600 so why would you run GPU tests at low detail settings?

 

[nevbie]

Originally posted by: BFG10K
AA-less benchmarks on high-end boards generally add CPU/platform limitations and at that point you?re testing more than graphics cards. Again it's all about customizing based on the hardware being tested.

You wouldn't (for example) test CPUs at 2560x1600 so why would you run GPU tests at low detail settings?

I would consider AA/AF and such more as universal filters/effects (whichever suits better) than mere game settings. Typically games separate various detail settings and AA/AF options, so I wouldn't call running without them "low detail".

Let's say card A gets 60fps and card B gets 60fps with AA off. If card A scales better with AA than card B, and you only compare "high-end" cards with AA on, you lose information. Besides, how do you know what card is "high-end" and what card is not, without running both cards with the same settings? I'm just trying to avoid the hardocp graphics card review style. Apples vs oranges and so on.. =P

to keys: you can satisfy most of the suggestions here by going the "let's get more results"-route..

btw, the fps graphs at hardocp reviews are fun! I'm not exactly sure how useful they are, but they are the sole reason I check their reviews nowadays. Probably goes to the "tell us minimum fps values!" suggestion category.

 

[ArchAngel777]

Originally posted by: BFG10K
Aye, in my 8800 GTS testing I went from 59 FPS to 229 FPS in Star Trek Elite Force after a driver update.

This is the most extreme example I?ve seen but it shows what a huge difference drivers can make.

That was such a great game. Probably one of the best games that ever came out based on the Q3 engine.

In regards to the performance, I'd say that is a good example, however, looking back it was clear that the driver was bugged, as an old game like that running so poorly on an 8800 class card was deplorable.

 

[Munky]

Some good points here, some not so good. Firstly, I want to see high end cards benched with AA and AF whenever these features can be enabled and the performance is reasonable. Jaggies and blurry textures are not only ugly, but they are unrealistic settings for a high end card. Likewise, highest ingame settings should always be used, to stress the cards as much as possible. If the performance is unacceptable for some cards at those settings, then the review should additionally test those cards at lower settings.

Secondly, cpu scaling should be detailed in a separate article. When testing video cards, you want to eliminate all possible bottlenecks, and therefore should use a fast cpu. Instead of testing 27 different platform combinations, I'd like to see more video cards, including the ones from last generation, so people have a clear picture of how much improvement they can expect by upgrading to a newer card.

 

[Munky]

Originally posted by: nevbie
I would consider AA/AF and such more as universal filters/effects (whichever suits better) than mere game settings. Typically games separate various detail settings and AA/AF options, so I wouldn't call running without them "low detail".

Let's say card A gets 60fps and card B gets 60fps with AA off. If card A scales better with AA than card B, and you only compare "high-end" cards with AA on, you lose information. Besides, how do you know what card is "high-end" and what card is not, without running both cards with the same settings? I'm just trying to avoid the hardocp graphics card review style. Apples vs oranges and so on.. =P

I consider AA and AF no different than other game settings, and no less important. If you are going to turn down settings, then why stop there? Why not turn down shadows, particles, lighting, and just make Crysis look like FarCry? If a certain card has bad performance with AA and AF, then the reader should know about it, and likewise there will be situations when a certain card has bad performance with HDR or many shadows, about which the reader should know as well. The review should show how a game plays on a $300 card compared to a $150 card, and if you don't use maximum visual quality settings, then you're not providing an accurate picture.

 

[ArchAngel777]

Originally posted by: munky
I'd like to see more video cards, including the ones from last generation, so people have a clear picture of how much improvement they can expect by upgrading to a newer card.

I agree with that idea, but would prefer it to go two generations back. Not everyone updates their graphics card every generation. Since G80 is so old, it would be ok to have only one generation back to say the 7800GTX.

 

[nevbie]

Originally posted by: munky
I consider AA and AF no different than other game settings, and no less important. If you are going to turn down settings, then why stop there? Why not turn down shadows, particles, lighting, and just make Crysis look like FarCry? If a certain card has bad performance with AA and AF, then the reader should know about it, and likewise there will be situations when a certain card has bad performance with HDR or many shadows, about which the reader should know as well. The review should show how a game plays on a $300 card compared to a $150 card, and if you don't use maximum visual quality settings, then you're not providing an accurate picture.

Ah, but you aren't providing an accurate picture if you don't use medium visual quality settings either. What if no card provides good enough performance with maximum settings? Different cards will probably scale differently when different settings are lowered, an inferior card with maximum settings may become relatively better with lower quality settings. I wouldn't judge the cards by maximum settings in this case. There seem to be many kind of potential scaling questions involved. They'd really deserve an article (or ten).

Of course if current reviews lack the maximum setting results, it would be a step forward to add them.

Once a specific benchmark is chosen (testbed, game, settings, what to record) it would be desirable to run it with as many cards as possible. That is a comparison of cards in my opinion.

note: I think a review shouldn't care much about the price of the products, because the readers will always have that information anyway.

 

[Munky]

Originally posted by: nevbie

Originally posted by: munky
I consider AA and AF no different than other game settings, and no less important. If you are going to turn down settings, then why stop there? Why not turn down shadows, particles, lighting, and just make Crysis look like FarCry? If a certain card has bad performance with AA and AF, then the reader should know about it, and likewise there will be situations when a certain card has bad performance with HDR or many shadows, about which the reader should know as well. The review should show how a game plays on a $300 card compared to a $150 card, and if you don't use maximum visual quality settings, then you're not providing an accurate picture.

Ah, but you aren't providing an accurate picture if you don't use medium visual quality settings either. What if no card provides good enough performance with maximum settings? Different cards will probably scale differently when different settings are lowered, an inferior card with maximum settings may become relatively better with lower quality settings. I wouldn't judge the cards by maximum settings in this case. There seem to be many kind of potential scaling questions involved. They'd really deserve an article (or ten).

Of course if current reviews lack the maximum setting results, it would be a step forward to add them.

Once a specific benchmark is chosen (testbed, game, settings, what to record) it would be desirable to run it with as many cards as possible. That is a comparison of cards in my opinion.

note: I think a review shouldn't care much about the price of the products, because the readers will always have that information anyway.

That's the point I'm getting at. The games should be tested with maximum visual quality settings, including AA and AF, except in cases like Crysis, where the performance hit from AA is so massive that no current video card can run it smoothly, except maybe at low resolutions. But in other games I don't see why AA, and AF in particular, should be optional. Blurry textures are really ugly, even when combined with the best lighting and shadowing effects.

 

[taltamir]

Originally posted by: ArchAngel777

Originally posted by: munky
I'd like to see more video cards, including the ones from last generation, so people have a clear picture of how much improvement they can expect by upgrading to a newer card.

I agree with that idea, but would prefer it to go two generations back. Not everyone updates their graphics card every generation. Since G80 is so old, it would be ok to have only one generation back to say the 7800GTX.

you know, that is a very good point. Usually a person is looking at reviews to upgrade from last gen to current gen cards. With benchmarks only comparing current gen cards he knows how fast they are compared to another, but not how fast they are compared to the currently owned card...