Help Design The Next AnandTech GPU Benchmark Suite

[Nintendesert]

The biggest thing for me is ease in getting a good representation of the potential gains in performance I can expect from an upgrade. Now this is really difficult with your reviews because you don't go back far enough in the types of cards you use in the tests. I understand time is short but most of us don't upgrade every generation.

Look at STEAM and what most people run, their hardware isn't all that top of the line. Including the flagship cards from a few generations ago will do a lot towards choosing which card to buy.

I know we have the database and can use that, but when a new card hits and on that day 1 decision on whether to upgrade I need to know if my old 8800GTS is getting absolutely smoked by the newest 560. Right now I need to look at what the 560 does against the 460, then go back to what the 460 does against the 280 or whatever and so on and so on.

Make it easy on us when you're doing one of those DX9 games and throw in a 4850, or a 9800 whatever like was asked earlier in the thread. I hope to see a 5870 vs. Kepler and the 7xxx series because I and many others skipped the 5xx and 6xxx generation.

 

[AtenRa]

New games to be included for GPU Benchmarking within 2011.

Battlefield 3 : New Frostbite 2 Game Engine with DX10-11 only support

Rage : New ID Tech 5 Game Engine with huge maps (OpenGL)

Elder Scrolls V – SKYRIM : New engine Large open landscapes.


Power Consumption: I would use 4-5 games and measure their power usage and have the average, by measuring only one app or game we don’t have the data necessary to make a good evaluation.

Also if you going to use GPGPU apps I would have a second measurement in those apps in order to see the difference in performance/power in different apps environments.

Lately, more and more people use there graphics cards for more things as GPGPU apps, multimedia and not only for gaming, so I would divided the power consumption section in to three (3) categories, Gaming, GPGPU and Multimedia (Video/Blu Ray playback).

Minimum Frames: Just by showing the minimum frame in the benchmark is not showing the whole picture, I would use a chart with the entire time line of the benchmark (like the charts [H] use).

I know you cant put more than 3-4 cards in the same chart but it would be nice to see the 2-4 direct competitors (example could be the new HD7970-50 vs GTX680-70 or HD79xx vs HD69xx).

Eyefinity/Surround : with prices low as $150-200 for 23-24” monitors im betting that three (3) monitor setup will gain momentum in the coming years, so I would really like a follow-up review of the new cards in Eyefinity/Surround after the original single monitor review.

Same goes for a single 120hz 3D monitor setup, a lot of people will start to use them if they haven’t already. (Eyefinity/Surround 3D is too much for now)

If I think anything more ill let you know

 

[bryanW1995]

For GPGPU applications I think that SETI is your best bet. It's the most popular other than F@H, and as far as I can tell there are no unfair advantages/optimizations for either company.

Also, I would like to ask that you keep the top 2-3 cards from each camp's previous generations going forward for 3 generations. So, for AMD you could include 4850/4870, 5770/5850/5870, and 6950 and 6970 in 2012 gpu roundup in addition to any other 6xxx and 7xxx cards that you wanted. For Nvidia I would pick gts250/gtx 260 (core 216)/gtx 280, gtx 460 1gb/gtx 480, and gtx 560ti/gtx580.

Obviously, the key for these is the older cards b/c the most recent round will automatically be included. But having the most popular cards from older generations will enable AT bench to be a more useful tool for the many people who wait 3-4 generatios before upgrading. As a case in point, I've had several instances recently in which I tried to compare a gtx 260 to a newer card for somebody looking to upgrade, but since that (very popular) card is already out of the AT bench for 2011 I had to go back to 2010 and pick the "best fit" comparison instead.

Sorry, I was thinking this was for AT bench, not for benchmarking reviews.

 

[busydude]

So let me ask you guys this: do you play many UE3 games (i.e. am I overestimating its importance?), and given that we largely use the same benchmark suite from top to bottom, would you be okay with UE3 not showing up in mainstream/low-end video card reviews?

UE3 is the most widely used game engine right now. Almost everyone I know in PCG forum has played ME, ME2 and Batman: AA. I am not sure which game is the best implementation of UE3, but I think including a UE3 title would be much appreciated.

 

[Stuka87]

I would like to see at least 2 generations back for some cards. For instance, AMD's 4000 (5000 if the 7000 comes out soon) series and nVidia's 200 series.

I would also like to see a UE3 based game. Yes the engine is not used to its full potential so frame rates are high, but there are many games that use it that are all very popular.

And while I would love to see an AMD and an Intel test for at least some of the games, I can see how that would be a major pain to test everything more than once.

 

[busydude]

I would like to see at least 2 generations back for some cards. For instance, AMD's 4000 (5000 if the 7000 comes out soon) series and nVidia's 200 series.

I guess Ryan made it clear that would not be possible. I forget where he posted that, but I definitely remember him being against that(Possibly due to Dx11).

 

[Termie]

...
Obviously, the key for these is the older cards b/c the most recent round will automatically be included. But having the most popular cards from older generations will enable AT bench to be a more useful tool for the many people who wait 3-4 generatios before upgrading. As a case in point, I've had several instances recently in which I tried to compare a gtx 260 to a newer card for somebody looking to upgrade, but since that (very popular) card is already out of the AT bench for 2011 I had to go back to 2010 and pick the "best fit" comparison instead.

I guess Ryan made it clear that would not be possible. I forget where he posted that, but I definitely remember him being against that(Possibly due to Dx11).

Yeah, I think we have to be realistic here. With so many games being DX11 now, testing the 8800gt, GTX260, 4850/4870, etc., just isn't meaningful, at least for those games, since the graphics settings aren't equivalent. On the other hand, I do think Anandtech should continue to bench those cards in games that aren't DX11, like Crysis Warhead, Mass Effect 2, etc.

By the way, I'd think that Mass Effect 2 could stand in for other UE3 games. That's a game people still play. I also understand keeping an OpenGL game in the benchmark (but not two).

As for the suggestion of Skyrim, I don't think it will be out in time for this round of testing. Mass Effect 3 might not either. But man, not having BF3 in there...sad day for sure! Are we really going to see AMD cards before Oct. 25th anyway?

Ryan - we'll all give you a pass on the general rule against including new games in the benchmark. Please include BF3!

 

[Grooveriding]

This is actually an interesting perspective since I have the opposite point of view - I'm currently looking for a new UE3 title to keep in the suite. UE3 is used in so many games that I'm hesitant to ignore it, but you're right in that the framerate on most games is rather ridiculous on high-end cards.

So let me ask you guys this: do you play many UE3 games (i.e. am I overestimating its importance?), and given that we largely use the same benchmark suite from top to bottom, would you be okay with UE3 not showing up in mainstream/low-end video card reviews?

Rage right now is on the short-list for future inclusion, so long as we don't find any problems when vetting it. As long as games are still using OpenGL I believe it's important to have an OpenGL benchmark as part of the suite, and as the first commercial OpenGL 3.x title it's going to be a much more challenging game than the current collection of id Tech 4 games.

If UE3 is going to be included, a good option may be Red Orchestra 2, coming out the end of this month. It is using a modified UE3 engine and should be a popular title and more recent than a lot of the UE3 games out there.

I concur with another poster here about some sites that use UE3 games in their benches and declare one card a winner over another in their final overall conclusion because it got 200FPS in some ancient titles while the other got 175FPS. It's a meaningless comparison as both cards are so far beyond what is needed and serves no purpose beyond giving skewed review conclusions for whatever reason.

Games being tested should be pushing even the 6990 hard enough to show sub 60 framerates, at least when we are talking about resolutions like 2560x1600 or even 1920x1200.

 

[Spjut]

Yeah, I think we have to be realistic here. With so many games being DX11 now, testing the 8800gt, GTX260, 4850/4870, etc., just isn't meaningful, at least for those games, since the graphics settings aren't equivalent. On the other hand, I do think Anandtech should continue to bench those cards in games that aren't DX11, like Crysis Warhead, Mass Effect 2, etc.

I can agree about DX11 games without DX10/10.1 fallbacks being unnecessary to include older GPUs for, but for those games that do support DX10/10.1 as well I definitely want to see how they compare to the DX11 GPUs in DX11 games...
It is especially nice for those games that just suck up performance in DX11, why go from eg a HD4850 supporting DX10.1 and playing the game smoothly, to a DX11 GPU that's running worse than your DX10.1 card in DX11 or just is too weak to enable those DX11 goodies anyway?
Benchmarks showing that contrast would be well appreciated

Apart from Crysis and TW2,most gaming DX10/10.1 GPUs have no troubles running DX9 games and therefore don't give us an urge to upgrade, what we need to see is how our older GPUs holds out in the modern games

As for the difference in settings, the articles can simply mention that the GTX260 only supports DX10, the HD4870 DX10.1 etc and are tested in those modes, whereas all other tested GPUs are DX11 and have extra effects not available in lower DX versions, most folks would certainly understand that without difficulty

 

[nemesismk2]

The games I would like to see included are:- Battlefield 3, Crysis 2 with DX11 etc, Fear 3, Witcher 2, Red Faction Armageddon and Dirt 3.

 

[sandorski]

Re: BitCoin

Is there a way to ensure doing the same task between runs? I'm not sure there is, because time to completion of a task is not predictable. AKA, I'm not sure it even has identifiable Work Units. Could be wrong though.

Milkyway, OTOH, is predictable with unique Work Units.

 

[mnewsham]

Re: BitCoin

Is there a way to ensure doing the same task between runs? I'm not sure there is, because time to completion of a task is not predictable. AKA, I'm not sure it even has identifiable Work Units. Could be wrong though.

Milkyway, OTOH, is predictable with unique Work Units.

With mining you would be looking at the hash rate not the work units done.

 

[sandorski]

With mining you would be looking at the hash rate not the work units done.

Ah ok.

 

[scooterlibby]

What about a change in methodology? I never really feel comfortable with 'one run' tests. What are the chances that the difference in FPS for a game between 2 cards is just a fluke? I'd wager there's a good chance that's the case if you only test once per card.

The only way to get around that is to do iterative testing - multiple runs per game per card. Then do a difference of means test or just something more statistically sound than a one and done.

-statsnerd

 

[Tempered81]

Hi everyone, your friendly neighborhood GPU editor here;

I'm currently in the process of building our next GPU benchmark suite. Ostensibly this is for when 28nm GPUs hit the market over the next year, but this will also cascade down in parts into our other GPU and IGP reviews.

Anyhow, as some of you may know, before any major overhaul of the GPU benchmark suite I like to gather some feedback from you guys on what you'd like to see in our benchmark suite, so that we aren't overlooking something that you guys may find critical. Given that we now have our Bench comparison system, feedback is all the more important this year since the data will be available through Bench for years to come.

So if you guys have any games or GPGPU applications that you'd like to see on our next GPU benchmark suite, now's the chance to speak up; we're interested in what you guys think. With that said if you could please provide a basic rationale/reason behind your request it would help the process a lot. And on a side note, for GPGPU benchmarks they need to be cross-platform (e.g. F@H is out because they don't have a proper AMD client right now).

Finally, just to give you guys an idea of what we're roughly looking at so far, we're looking at 7-9 games (one of them will be Crysis). New this fall will be a SSAA benchmark on one of our DX9 titles, since we have a couple different games that both NV and AMD SSAA work on. Our load testing methodology will also be changing slightly since both AMD and NV now use intricate throttling systems on their latest cards.

Anyhow, please let us know what you'd like to see. The more feedback we get, the better we can tune things to meet your needs.

-Thanks
Ryan Smith

It would be cool if you guys could test the cards power draw like Techpowerup & Xbitlabs do. I know this requires a special device for measuring PCI power and PSU Power rails.

It would be good to see 10+ titles benchmarked instead of 7.


Thanks!

 

[bryanW1995]

Yeah, I think we have to be realistic here. With so many games being DX11 now, testing the 8800gt, GTX260, 4850/4870, etc., just isn't meaningful, at least for those games, since the graphics settings aren't equivalent. On the other hand, I do think Anandtech should continue to bench those cards in games that aren't DX11, like Crysis Warhead, Mass Effect 2, etc.

By the way, I'd think that Mass Effect 2 could stand in for other UE3 games. That's a game people still play. I also understand keeping an OpenGL game in the benchmark (but not two).

As for the suggestion of Skyrim, I don't think it will be out in time for this round of testing. Mass Effect 3 might not either. But man, not having BF3 in there...sad day for sure! Are we really going to see AMD cards before Oct. 25th anyway?

Ryan - we'll all give you a pass on the general rule against including new games in the benchmark. Please include BF3!

Yeah, I misunderstood the OP, I was thinking that it was for AT bench instead of reviews.

Now that we're on GPU's, I'll just mention that I prefer no OC cards from either camp in testing. I think that we already voted on that last year, but just in case you're considering it again I'd strongly caution against it!

 

[RussianSensation]

I think if you are testing next generation GPUs, then the most important factors are:

1) You will want to pick games over a wide variety of genres, so it is representative for a wide variety of gamers
2) You will want to use the most modern / demanding game engines (otherwise it defeats the purpose of testing modern GPUs)
3) You will want to find engines that benefit from additional VRAM to test this limitation for future game engines. This is going to be the most challenging since most games still don't need more than 2GBs of VRAM.

With that out of the way:

Racing:
- Dirt 3 or upcoming F1 2011 (whichever is more demanding). Dirt 3 seems to be a breeze though. While F1 2011 may be more demanding, I have a feeling more people will play Dirt 3. Both use the same engine (so it likely won't matter) Alternatively, you can try to pick the latest Need for Speed: The Run, which should be coming out shortly.

FPS:
- Battlefield 3 (so many people will play it online for years to come). This is a must have.
- Rage (the most recent OpenGL game from iD), although unlikely to be demanding. So a maybe. I have a feeling most modern cards will max this game out. So I may leave this as a backup since it will likely to be outdated as a benchmark within 12 months as Wolfenstein was.
- Crysis 1 & Crysis 2 with DX11 (because it's the best implementation of DX11 thus far). If I had to choose, I'd ditch Crysis 1 and just go with Crysis 2 since it's a great benchmark for Tessellation, POM, advanced DOF DX11 settings.
- Metro 2033: Because Metro 2034 will launch in 2012. And also it's a Tessellation and DOF killer (So a prime candidate for foreshadowing future DX11 game engines): http://www.computerbase.de/artikel/grafikkarten/2011/test-19-grafikkarten/23/

Although, this one is a maybe since many people have beaten this game. So again, not so relevant from trying to choose modern cards.

Role-Playing:
- Elder Scrolls V: SKYRIM - Will likely to be very popular, plus will have a modding community. Later on this presents you with an opportunity re-test this game with the most demanding modern mods 2-3 years from now. Your inclusion of previous Elder Scrolls game proved to be an excellent choice for GPU testing in the past as well. I doubt this game will be a disappointment in that regard

- Witcher 2 (arguably the most beautiful game now, also allows us to quickly check Super Sampling performance of modern cards).

- Dragon Age 2: Still demanding and it does run out of VRAM at 2560x1600 8AA: http://www.computerbase.de/artikel/grafikkarten/2011/test-19-grafikkarten/21/
But, again, depending on how demanding SKYRIM is, it's a toss up. I feel that most people will have beaten this game by now though; so it's less relevant compared to Elder Scrolls V.

Strategy:
- Shogun 2 - easily uses 2GBs of VRAM and is insanely demanding: http://www.computerbase.de/artikel/grafikkarten/2011/test-19-grafikkarten/24/#abschnitt_shogun_2

- Starcraft 2: At 2560x1600, also can exceed 1GB of VRAM: http://www.computerbase.de/artikel/grafikkarten/2011/test-19-grafikkarten/15/

With MLAA, it's playable on AMD cards, but with standard FSAA, NV has a huge lead here. Arguably though, it is a better CPU bench I feel. However, an argument can be made that since it's one of the most popular strategy games and will still have 2 major campaign expansions coming out, that you can't do wrong by including it.

GPU Specific:
- Photoshop CS5/6 - we have so many threads asking what card is better for Photoshop (since NV has hardware acceleration support and AMD plans to bring it starting with CS6). It would be invaluable to see if buying a faster NV card for Photoshop acceleration is better than laying out $1000 for a Sandy Bridge E 6-core 12 threaded processor for example. How much does video acceleration really help in Photoshop? This bench will never be outdated imo.

- Milkyway@Home OR PrimeGrid - More modern GPGPU engines, scale proportionally with double-precision GPU performance:
http://forums.anandtech.com/showthread.php?t=2064356

Since Kepler is supposed to have 3x the DP performance over Fermi and AMD continues to focus on DP performance, this is a great benchmark to test GPGPU performance / watt and absolute DP performance. In other words, although this particular bench may not be useful on a per use basis since not many people use this, it provides a direct window to see how a card will perform in scientific applications that benefit from DP performance.


***The only only caveat with my game selection is that more than 53% of all PC gamers play Role-Playing and Strategy Games - far more than any other genre individually. In other words, if I was designing a long-term bench, I'd pick those most popular games first (regardless if they are more demanding or not), simply due to a broad usefulness of such a benchmark. Perhaps, these are more of a must have for your CPU bench however. You can see the difference in performance fluctuates significantly for the Strategy Genre (33.6% of all PC games sold in 2010) depending on the CPU:
http://www.techspot.com/review/305-starcraft2-performance/page13.html

Take a look a the PC Gaming Genre Sales in 2010 to help with your selection:
http://www.techspot.com/news/44167-esa-facts-and-figures-about-the-gaming-industry-in-2010.html

I suppose, you can either pick the most demanding games in your new GPU bench, or keep the most popular games like SC2, WOW and include the most demanding games in your regular GPU reviews. That could work also.

 

[Termie]

Russian nails it with his detailed response.

I just checked the Skyrim site and it's not out until 11-11-11. If we make an exception for BF3 being a new game, maybe we should make it for Skyrim too (assuming new cards aren't out before these games hit the market).

BTW, I really think keeping Crysis 1 is pretty critical for inter-generational comparisons. If we had to cut back on an FPS game, I'd drop Metro 2033. Crysis 2 provides many of the same DX11 challenges.

 

[RussianSensation]

BTW, I really think keeping Crysis 1 is pretty critical for inter-generational comparisons. If we had to cut back on an FPS game, I'd drop Metro 2033. Crysis 2 provides many of the same DX11 challenges.

That's a good point. Many people still have a fascination with Crysis' performance. At the same time, I feel that Witcher 2 is the best example of the most demanding DX9 game. If we have Crysis 1 and 2, that's 2 / 7 games devoted to Crysis. If I had to choose an inter-generational game, I'd pick Witcher 2 for 3 reasons:

1) It's DX9, so that means just like Crysis 1 you can test cards all the way back such as the 9800GT and see exactly how much faster modern cards have become.

2) The Role-Playing genre's sales are larger by size than the FPS genre. So this bench will likely be more useful for more gamers than Crysis 1 is (I mean how many people are going to be replaying Crysis 1 with a GTX680 SLI setup?).

3) On Ultra settings with Super Sampling, this game is more demanding than Crysis 1 is:
http://gamegpu.ru/RPG/Rollevye/The-Witcher-2-Assassins-of-Kings-versii-1.2-tect-GPU.html

Witcher 2 also supports 3D vision, not sure if Crysis 1 does?

 

[tigersty1e]

Shogun 2. One of the very few demanding dx11 games that has amd support. To balance out the nvidia optimized titles.

 

[Gikaseixas]

I guess Ryan made it clear that would not be possible. I forget where he posted that, but I definitely remember him being against that(Possibly due to Dx11).

but it would have been so good to see the gains a GTX 280 could get by being tested with a 2600K cpu...

They could test them all in DX10 just for that comparison

 

[Deleted member 284126]

Video card noise levels (and in general all computer noise) really annoys me.

I would like to see more thorough noise level testing. I know sound is very subjective and environmental conditions play a huge part, so accurately conveying noise through a review may be iffy.

I mean there are websites completely dedicated to showcasing quiet computer components, so noise is very much a dealbreaker for a lot of people.

 

[Ryan Smith]

Minimum Frames: Just by showing the minimum frame in the benchmark is not showing the whole picture, I would use a chart with the entire time line of the benchmark (like the charts [H] use).

We get this request a lot. Unfortunately minimum framerates are extremely unreliable - by definition it's an absolute and not an average, so otherwise minor variations between runs make a huge difference. We do want to do minimums where it makes sense, but there are only a few games that have ever proven to offer reliable minimum framerates.

Also, I would like to ask that you keep the top 2-3 cards from each camp's previous generations going forward for 3 generations. So, for AMD you could include 4850/4870, 5770/5850/5870, and 6950 and 6970 in 2012 gpu roundup in addition to any other 6xxx and 7xxx cards that you wanted. For Nvidia I would pick gts250/gtx 260 (core 216)/gtx 280, gtx 460 1gb/gtx 480, and gtx 560ti/gtx580.

For older cards it's basically going to be the opposite of what you've requested; if we already have a 4870, then there's little value in adding the 4850 since its performance is fairly close to the 4870. I have to get the most out of what time I have here.

I can agree about DX11 games without DX10/10.1 fallbacks being unnecessary to include older GPUs for, but for those games that do support DX10/10.1 as well I definitely want to see how they compare to the DX11 GPUs in DX11 games...
It is especially nice for those games that just suck up performance in DX11, why go from eg a HD4850 supporting DX10.1 and playing the game smoothly, to a DX11 GPU that's running worse than your DX10.1 card in DX11 or just is too weak to enable those DX11 goodies anyway?
Benchmarks showing that contrast would be well appreciated

Testing cards at different settings fundamentally breaks apples-to-apples testing. Not that such testing doesn't have a purpose, but for our general benchmark suite it's outside the scope of what we're evaluating.

What about a change in methodology? I never really feel comfortable with 'one run' tests. What are the chances that the difference in FPS for a game between 2 cards is just a fluke? I'd wager there's a good chance that's the case if you only test once per card.

The only way to get around that is to do iterative testing - multiple runs per game per card. Then do a difference of means test or just something more statistically sound than a one and done.

-statsnerd

Virtually everything is run 3 times right now for that reason.

Now that we're on GPU's, I'll just mention that I prefer no OC cards from either camp in testing. I think that we already voted on that last year, but just in case you're considering it again I'd strongly caution against it!

Yeah, that was bad. You won't be seeing that again.

1) You will want to pick games over a wide variety of genres, so it is representative for a wide variety of gamers

It's interesting that you mention that. We're seeing the usual requests for FPS/RPG/Strategy/Racing games, but nothing for flight sims and the like. Are there any genres other than the standard four that you guys would like to see?

Since Kepler is supposed to have 3x the DP performance over Fermi and AMD continues to focus on DP performance, this is a great benchmark to test GPGPU performance / watt and absolute DP performance. In other words, although this particular bench may not be useful on a per use basis since not many people use this, it provides a direct window to see how a card will perform in scientific applications that benefit from DP performance.

My one concern with DP benchmarking is that consumer cards are artificially limited. That doesn't make the test invalid, but it means the tests don't provide as much meaningful information about the hardware's capabilities.

 

[sm625]

I just want to be able to compare new low end hardware with yesterdays high end cards. Take a HD3870 for example. If I find one for $15 I want to be able to tell if it is worth that much. Or an X1950 pro. So dont leave us scavengers behind. I especially need to know how these things compare with llano.

 

[Will Robinson]

One of the popular Bitcoin mining apps would be appropriate as it is a good test of the compute functionality of modern graphics cards.
Given how much positive press has been given to NVDA's Folding@home prowess over the last few years I'm sure NV fans will want to test their stuff on such a popular new application as BC mining.