What changes would you make to video game benchmarking

[digitaldurandal]

1) What games would you use?

2) What settings would you use?

3) What type of charts and information do you think is appropriate?

1) DA:O or A, black ops, bfbc2, mafia ii, dirt2 or GRID, batman:ac (when released)

2) a range of settings, definitely including surround/eyefinity for the high end cards!

3)list of max playable settings for each card for each game in a chart similar to anandtech's gpu bench - where you add the information to a database that can be accessed for each resolution and game.

Plus of course the standard information + fps/s graph to show actual playability with check boxes to compare the graphs of two or more cards. This way instead of having to crawl through and manually compare the graphs of two cards, you could access the information more readily.

 

[w1zzard]

1) all
2) all
3) minimum, average, maximum, binned histogram, median, std dev fps, confidence interval, accuracy, resolution. charts with error bars would be nice to see
4) all
5) all from lowest to highest
6) all from lowest to highest
7) bonus question: compare to which cards in reviews? all
8) bonus question 2: run on which hardware platforms? all, including 32-bit & 64-bit comparison

suggestion: look at the reviews you can find online. make a matrix what you like and what you don't like. come up with ideas how those can be improved, try to implement them. look at your personal fields of expertise and get those added to the review. also scrap items from the list above based on the time you want to spend on a review. expect manufacturers to drop cards on you 2-3 days before launch.

performance benchmark data is only a very small subset of what is required to deliver a useful review

 

[tannat]

3) Full plot of FPS over time. This gives much more info than any histograms. Statistical data in histogram form does not ad very much if anything.

2)a1 High resolution - Eyfinity/surround + @ most common resolution 1920x1080. More is not needed

2)b Compare with no AA/AF and at highest playable settings

Additional thought. I find canned benchmarks more or less meaningless. It's just not representative or related enough to real gameplay to waste time and statistics on. Bench real gameplay or just don't do it.

 

[w1zzard]

uh, whats min fps over time? min fps is a single number. are you asking for a normal fps plot ? or do you mean some kind of SMA ? highest frametime in a 1 sec bin?

Statistical data in histogram form does not ad very much if anything.

percentage of frames below x fps ?

 

[Minas]

HardOCP's style is my favourite. I like the fps charts from actual gameplay, and apples-to-apples comparisons. I'd prefer to see more benchmarking like that, only with more cards compared in each review. Games that consistantly run at >60fps with high settings really need to go (mass effect 2 and wolfenstein are glaring examples in AT's game list, although they don't change that list very often).

 

[Lonyo]

HardOCP's style is my favourite. I like the fps charts from actual gameplay, and apples-to-apples comparisons. I'd prefer to see more benchmarking like that, only with more cards compared in each review. Games that consistantly run at >60fps with high settings really need to go (mass effect 2 and wolfenstein are glaring examples in AT's game list, although they don't change that list very often).

They are two valid games, being the most recent iD Tech game (and one of the few modern OpenGL games AFAIK), and being an Unreal Engine 3 game, and Unreal Engine 3 is widely used. They could replace it with something like a MoH single player benchmark, which also uses UE3, but those two games to have a useful role in the grander scheme of things.

 

[Keysplayr]

uh, whats min fps over time? min fps is a single number. are you asking for a normal fps plot ? or do you mean some kind of SMA ? highest frametime in a 1 sec bin?



percentage of frames below x fps ?

Min fps I think is the most important aspect of benchmark results.
Do you remember the built in FEAR benchmark? Gave percentage of benchmark time below 30fps, between 30 and 59, and 60 and above as percentages. I like those kind of results. Line charts are great for showing the amount of time spent at the low fps range, how often the fps dips below playable at given settings.

Oh, yes, and games that don't even stress the GPU should be dropped from the bench suite. No need to see 100+ frames with everything maxxed at 25x16.

 

[Jacky60]

HardOCP's style is my favourite. I like the fps charts from actual gameplay, and apples-to-apples comparisons. I'd prefer to see more benchmarking like that, only with more cards compared in each review. Games that consistantly run at >60fps with high settings really need to go (mass effect 2 and wolfenstein are glaring examples in AT's game list, although they don't change that list very often).

I agree that games flying along at 100fps need dropping. Arma 2 features in very few benchmarks but is just the sort of game you could never run on a console. Drop the console ports and introduce more demanding PC games -Stalker is a good example

 

[Troll Trolling]

First, I agree with the gentlemen saying that real gameplay is better (doesn't have to be like HardOCP, maybe something like Computerbase or Bit-tech)

I think there should be a graph of FPS over the gameplay; and most important, marked where it stutters: stutter defined as when the the frame interval delta is 'x' (took 'x' more time to render) times higher than the previous (it should be tested what delta is considered stutter). Of course, when the frame rate is already low, it shouldn't count, because the game is already unplayable.

With this information, we could have a playability score, calculated by a formula like this:
(AvgFPS/"time spent on a fps lower than 25")
1+0,5*"times it stuttered"

Standardized so that a score (1, 10, 34, 1000...) or higher is playable.

Something like this

 

[Borealis7]

uh, whats min fps over time? min fps is a single number. are you asking for a normal fps plot ? or do you mean some kind of SMA ? highest frametime in a 1 sec bin?
percentage of frames below x fps ?

is this W1zzard from TPU? member since 2002, only 4 posts lol...

 

[SirPauly]

I don't care if games are for the console or PC; as long as they're quality games. However, would like to see multi-platform games take advantage of PC abilities to improve gaming experience value.

For reviewers, it's usually the same, benchmark, maybe touch on IQ a bit -- conclusion. If an IHV offers an ability, in my mind, is part of the the product and should never be ignored based on reviewers' preference.

More investigations, less canned, concentrate on GPU limitations and sustained minimums, more intense testing on the actual features the cards offer.

No leading for the readers, an allow data to lead, and allow the reader to make up what is best for their needs based on the data offered.

Have three formats, easy, moderate and advanced formats so one can go back to read the review formats as they slowly digest data and learn.

A format that evolves and not static -- a dynamic review that evolves with future drivers or features. One of the things that sucks is when one reads a review of a product -- sometimes the data is old, yet has improved with future features and drivers.

 

[nemesismk2]

HardOCP's style is my favourite. I like the fps charts from actual gameplay, and apples-to-apples comparisons. I'd prefer to see more benchmarking like that, only with more cards compared in each review. Games that consistantly run at >60fps with high settings really need to go (mass effect 2 and wolfenstein are glaring examples in AT's game list, although they don't change that list very often).

HardOCPs style is also my favourite

 

[tannat]

uh, whats min fps over time? min fps is a single number. are you asking for a normal fps plot ? or do you mean some kind of SMA ? highest frametime in a 1 sec bin?



percentage of frames below x fps ?

FPS over time, sorry.

In order to see how the FPS drop behave and where min FPS is.
Is it a single dip or is it several small dips. Does the cards dip at the same place? Is it visible on screen etc.

 

[Anarchist420]

1) What games would you use?

2) What settings would you use?

3) What type of charts and information do you think is appropriate?

1) DA:O or A, black ops, bfbc2, mafia ii, dirt2 or GRID, batman:ac (when released)

2) a range of settings, definitely including surround/eyefinity for the high end cards!

3)list of max playable settings for each card for each game in a chart similar to anandtech's gpu bench - where you add the information to a database that can be accessed for each resolution and game.

Plus of course the standard information + fps/s graph to show actual playability with check boxes to compare the graphs of two or more cards. This way instead of having to crawl through and manually compare the graphs of two cards, you could access the information more readily.

I would list minimum, max, and avg framerates for each benchmark and I'd have in game settings set to their maximum quality, except for MSAA and SG-SSAA both of which I would always benchmark at 4x. I wouldn't test higher than 1920x1200. I wouldn't test lower than 1024x768.

I would also show a screenshot of the games graphics settings.

I'd also enable Vsync with a 120Hz monitor, and I'd force HQ for both vendors.

I'd set audio quality as high as the game allows.

Finally, I'd use 1 low range, mid range, and high range CPU for each benchmark.

 

[cmdrdredd]

1) What games would you use?

2) What settings would you use?

3) What type of charts and information do you think is appropriate?

1) nothing older than 2 years (except for perhaps crysis)

2) Maxed out only (no "gamer" quality in crysis). Only the highest settings and I'd turn on various levels of AA and AF for each test. 2x/4x/8x/16x at all resolutions. The more demanding the better. It will show more clearly the performance advantage of each card over the other.

3) I think a bar graph with the averages followed by the minimums is the best way. I hate charts that show a card as a % of performance against another card. For example, I hate taking a 6950 and making it the 100% and showing a GTX 580 being 140% faster in some game. That tells me nothing. I also think maximum fps is useless. If I max out at 60fps but never drop below 40 then there's no problem. I don't care if at some point looking at a blank wall it jumps to 500fps. That's silly. I also don't care for HardOcp's "max playable". Some games are playable at lower fps numbers (crysis was and is like this), while others rely on high fps to not get you killed (CoD and UT3). I won't turn off settings just to get a magic number. Crysis was very plable with maxed settings albeit with no AA/AF running.

 

[KeithP]

I would add a low to mid range computer system to check the video cards with. Tests always seem to be conducted with some high end rig. That is great for showing the full potential of the card but seems unrealistic it regards to what people are actually using.

-KeithP

 

[cmdrdredd]

I would add a low to mid range computer system to check the video cards with. Tests always seem to be conducted with some high end rig. That is great for showing the full potential of the card but seems unrealistic it regards to what people are actually using.

-KeithP

Very good point. Many are using C2Q, and Phenom II based systems which we know are not going to match an overclocked i7.

 

[Jacky60]

I would add a low to mid range computer system to check the video cards with. Tests always seem to be conducted with some high end rig. That is great for showing the full potential of the card but seems unrealistic it regards to what people are actually using.

-KeithP

I'd agree before I was high end CPU/GPU wise I always wondered how my PC would perform in the same test. BUT the reason I guess is to remove all other variables so perhaps not. No point in benchmarking a GPU which is cpu limited.