Potential New Anandtech Video Card Benchmarking Suite?

[blastingcap]

My vote is against mods unless they are so huge as to be games unto themselves (and standardized). If mods keep changing then on standardization grounds alone they aren't good. Imagine Skyrim Mod X, v. 1.00, and doing a bench of it and then 2 years later trying to re-bench on that outdated mod on a different card. No, just no. You can get anecdotal stats off mod forums or whatever if you want, but don't fragment official reviews. Just my 2 cents.

To Ryan: I'd include one indie game in there to tip my hat off to those games that do not use the common engines like Unreal. There are a lot of games in development via Kickstarter, at least some of which will be good and popular. This helps out with the "four UE3 FPS games" problem.

RE: Frametimes I understand that nothing short of a high-speed camera will really tell the whole truth, but as it stands fps is not telling all the story...

I agree on the MSAA 8x decision, and more generically, I agree to not test wild settings that don't really impact image quality.

Naturally I would support testing more games at Eyefinity resolutions...

I agree about repeatability and fragmentation which you elaborated on re: MMOs... I wrote my anti-mod rant prior to reading it though. Sorry for being redundant.

 

[notty22]

However, most users WILL mod their Skyrim with high res stuff. Knowing this they would probably want to know what card won't limit them as much. If a 2GB card struggles so what? Note it and say "this card cannot run the mods like this". That's Nvidia's fault for only using 2GB as standard and they SHOULD be ridiculed for it if indeed some title really has issues. I know for a fact that ENB Skyrim with 4k textures does NOT use a full 2GB. Close, but not fully.

Saying you won't use mods because it will expose the vram limitations (you mentioned 2GB vs 3GB which translates to Nvidia vs AMD) sounds like you are afraid to paint Nvidia in a poor light. Playing favorites are we? If performance tanks we deserve to know as readers. Journalists are supposed to be neutral and simply show us the facts as they find them. Let us draw the conclusions.

What you seem to be saying by removing key titles like Metro and not wanting to expose potential VRAM limitations by using simple mods for Skyrim is that I should find another site for my GPU reviews. I know you do Notebook and mobile GPUs Jarred but if I were buying a gaming laptop or desktop replacement I want to know how it handles all this stuff that my desktop could.

This is all my opinion. Not meant to be directed at anyone at all.

I recognize you were not answering a specific poster. I'm responding to your gut-wrench idea of needing to find a new web site to read reviews. You should read multiple sites for reviews. It can't hurt.
Going back to GTA IV, you can expose ram limitations, here is a article.
How Much RAM Does Your Graphics Card Really Need?
What good does it do, to focus in on obsoleting a specific high end card model. Which could be easily done. You could make a 3gb gtx 660TI look better (what's the definition of better) than 2gb gtx 670's, with mods. What good does that do the community, the hardware companies, less knowledgeable new buyers etc.
And I don't understand your line of thinking that Nvidia should be ridiculed for 2gb, the standard implementation of 4gb is simply not necessary right now for such a high % of gamers that it's a unnecessary consideration. The option is there for 4gb cards for the multi-monitor crowd. But not as a standard across the board standard configuration.

 

[DiogoDX]

Out: Dir3, Batman, Starcraft, Portal 2

Keep the others and add SSAA for Skyrim.

Possible adds: Sleeping Dogs, Farcry 3, Crysis 3, Metro LL, The Witcher 2, F1 2012 and Max Payne 3.

 

[f1sherman]

8x MSAA: Don't expect us to use 8x MSAA in any benchmarks. The quality improvement from 8x is extremely minor; I'd rather turn on TrAA/AAA or SSAA first than to blow performance like that

This guy. Listen to him, he knows the stuff.

 

[RussianSensation]

They don't.
Name one site that benchmarks every card with every driver with every patch of every game.

I never said AT should test every driver with every patch of every game. Not sure how you got that out of my posts. What I asked is how do other websites include newer games and update their test bench throughout the year? This is how they keep themselves relevant and increase readership/membership. AT's test bench remains static for 1 year and next year if they don't allow Metro Last Light and Crysis 3 to be added, it would be a big mistake in my opinion. You can disagree but no one cares how Diablo 3 or Batman AC runs on a $500 high-end videocard in 2013. If you wanted to play Diablo or Batman AC, you played them already and neither of those games needs a $500 GPU upgrade.

Why are you anchoring on the particular FPS? As long as the test is representative it doesn't matter if the GPU gets 10 or 10000FPS in it.

Silverforce11 hit the nail on the head. People who will want to upgrade their GPU in 2013 or buy a high-end GPU or even a current $300-1000 GPU aren't looking to play Diablo 3 or Torchlight II as even a $100-150 GPU can do it.

Of course it matters how demanding the game is which is why not including Crysis 3 and Metro Last Light in 2013 would ensure that people continue using other hardware websites for their needs.

If a game is perfectly playable on even a $100 GPU, it's not worth benchmarking any longer since any dedicated GPU you buy will play it. Your statement itself is contradictory since if you aren't picking relevant (i.e., GPU demanding games), what's the point of conducting a GPU-related test just to conclude that any card can do it? What do you do later for the remaining 11 months of the year? That particular game then becomes a waste of slot space in the test bench.

For a test to be relevant for 12 months, it can't be a game like Dishonored or Diablo 3 because the test bench has to be relevant not only for current generation of GPUs, but serve a function for testing HD8000 and GTX700 series. This is why it's important to at least pick somewhat GPU demanding games and not only the most popular games. Otherwise, the entire test suite would be filled with MMOs and Blizzard games and BF3 only.

5) Modded games are out. Sorry. Someone else already said some of this, but basically you're opening a can of worms and when you use a mod you run a major risk of unfairly penalizing companies. NVIDIA or AMD might work hard to get Skyrim to run acceptably at max settings, and then you add a mod that pushes things so far that it might overflow the RAM on a 2GB card but not on a 3GB or 4GB card. Now it looks like the 2GB card sucks at Skyrim, but the 4GB card still runs well. The reason the game didn't ship in a state that it would use more than 2GB is because the developers already looked at that and said, "That doesn't make sense."

While I realize that some people have problems with including mods in Skyrim, then I think SSAA should be forced in Skyrim. In the current state, the Skyrim GPU benchmark has run its course at AT. Look at the charts - you are looking at a CPU limited scenario.

Most of the cards are hitting 86 fps CPU wall and some are these are just $300 GPUs. Next year it's going to be 86 fps again with 2013 GPUs. If you don't force SSAA, this bench tells us the game is maxed out with HD6970 and any GPU above $300 is CPU bottlenecked at 1600P. So what's the point of including it in this form for 2013? No point.

2) Updating the gaming suite more than once a year makes comparisons with older hardware virtually impossible, particularly on the mobile side of things.

How about a high-end after-market GPU round-up mid-2013 then that includes newer GPU demanding games like Metro LL and Crysis 3 (and some other possible GPU demanding games)? This way you keep the main test bench selection that is decided on this year but still test next generation's games on next generation high-end GPUs?

Think about it, no Metro LL or Crysis 3 for you guys next year? Maybe two of the most graphically-demanding anticipated games and you won't have them until 2014? But instead games like Batman AC, Diablo 3, Civilization V will be used to test HD8970 and GTX780?

Until the list is finalized, thanks for the input! I'll point Ryan at this as well, since most of you seem to be talking more towards his realm of testing (desktop GPUs).

I actually think that's a key difference here. Since you test laptops, a lot of the popular games like Diablo 3, MMOs and strategy games will be more likely to be played on laptops since they don't require a $2000 Clevo. In addition, many laptop users who play games will have low-end or mid-range GPUs at best. People who build a dedicated desktop PC gaming rig aren't going for a GTX650 most of the time. Your comment that it doesn't make much difference if you guys drop Batman AC and replace it with Crysis 2 isn't entirely accurate for the desktop market since desktop gamers are always looking for guidance as to what game requires an upgrade to play smoothly. Batman AC simply doesn't push modern desktop GPUs anymore to warrant testing it (at least without cranking MSAA to 8x). For the laptop market, it could be entirely different since even the highest GPU in laptops is nowhere near as fast as a GTX680 on the desktop.

Also, you say that if a game gets 150 fps, it might still warrant testing. Look at Dishonored.

If we test a game like that, what do we find out? Well for starters that even HD6790 gets 70 fps average, meaning almost no one who has a fairly modern PC gaming rig needs to upgrade for Dishonored. Secondly, this game is DX9 and is based on the ancient UE3. Knowing how next generation GPUs perform in this game doesn't help anyone if an HD6790 can max it out. It doesn't tell us how next generation GPUs will handle DX11, tessellation, deal with high resolution textures, and other next generation graphical effects like global illumination, bokeh depth of filed, contact hardening shadows, etc. If you include such games, then you are just telling your readers what they already know - this game doesn't need a GPU upgrade. So once you run that test once, what you end up is 11 months of useless testing for the same game. This is what made Crysis, Metro 2033 and BF3 so special. When people had older GPUs, they knew if they upgrade, they would actually benefit in those titles. If someone goes out and buys a GTX780 next year, they aren't doing it to play Dishonored, or Diablo 3 or Batman AC, which is why testing games that net > 100 fps on $300 GPUs doesn't truly help gamers who are going to be upgrading for the sake of graphically demanding games.

 

[RussianSensation]

(3) Elder Scrolls: Skyrim SSAA or The Witcher 2 Enhanced Edition - these are both DX9 RPGs, so it's hard to see how they both belong in a 2013 test suite.

I think you should test Witcher 2 enhanced edition for 2 reasons:

1) It includes native SSAA (Uber-Sampling)
2) It's way more GPU demanding than Skyrim is even without SSAA

You hit 2 birds with one stone, allowing you to have a GPU demanding RPG that has a native SSAA mode. You can test mid-range GPUs without Uber-Sampling and still stress high-end GPUs with Uber-Sampling, solving the testing dilemma of finding a game that is great for testing low- to mid- to high-end GPUs.

Also, Skyrim is generally played with mods, so testing it in vanilla form is most likely not relevant to anyone actually playing it.

(7) Dirt Showdown or F1 2012

Dirt Showdown. You have next generation global illumination vs. F1 2012 which is largely a carryover from F1 2011.

F1 2012 gets 100 fps on a GTX660Ti at 2560x1440 and 63 fps with AA. Do we really need to test F1 2012?

If you want a brand agnostic racing game, maybe use Project CARS or wait for the new Need for Speed Most Wanted.

(8) Sleeping Dogs or Max Payne 3

For the same reasons I voted Witcher 2 over Skyrim = flexibility. Sleeping Dogs is way more GPU demanding than MP3 is, and gives the tester the ability to test normal FXAA for slower GPUs, while having the option to evoke native SSAA. That makes it a very flexible GPU test.

Also, this game is remarkably shader bound, which is ideal for testing next generation GPUs as it scales very well with added GPU horsepower. MP3 on the other hand is not that GPU demanding with FXAA, which doesn't really stress the ROPs or memory bandwidth of a videocard. In MP3, MSAA produces an unusually large performance hit while hardly smoothing edges, making FXAA more ideal with MP3. With FXAA though, MP3 is playable at 2560x1600 already on modern GPUs , which would make this test less than ideal at 1080P and below.

Since the test bench may not be updated until December/January, Medal of Honor Warfighter, Far Cry 3, Hitman Absolution may be wild cards to look at later this year.

Adding games mid-cycle: As Jarred already noted, the sheer amount of work it takes keeps us from frequently changing the benchmark suite (just go look at how many GPUs I have in bench, and I haven't done nearly as much SLI/CF testing this year). This doesn't preclude adding a game mid-cycle, but it means there needs to be a very good reason due to all of the prep-work required.

What about doing a special high-end GPU roundup? For example, take 1 fastest single-GPU from AMD and NV and have them in CF and SLI. You could do that once a year (mid-year or end of the year) to compare them in the most demanding titles that didn't make it into the 2013 Test Bench and at the same time reflect on SLI/CF scaling, micro-stutter. For example, MSI Lightning HD8970 vs. Asus Matrix / DirectCUII GTX780 showdown or something along those lines.

8x MSAA: Don't expect us to use 8x MSAA in any benchmarks.

SSAA in Skyrim would be welcome! Games like Witcher 2 and Sleeping Dogs have native SSAA from the in-game settings. Lots of flexibility to push GPUs with those 2 titles. :biggrin:

Finally, since we're on the subject of new games, let me throw something back at you. Coming up with first and third person games is fairly easy. My problem is that I really could use new strategy and sim games so that we aren't so FPS/TPS heavy. Is there anything you guys would like to see, keeping in mind our repeatability requirements?

For a sim racing game, Project CARS (short for Community Aided Race Sim) is a community funded racing simulator currently under development from Slightly Mad Studios, the developer of the Need for Speed: Shift series.
http://www.wmdportal.com/projects/cars/

 

[JarredWalton]

What about doing a special high-end GPU roundup? For example, take 1 fastest single-GPU from AMD and NV and have them in CF and SLI. You could do that once a year (mid-year or end of the year) to compare them in the most demanding titles that didn't make it into the 2013 Test Bench and at the same time reflect on SLI/CF scaling, micro-stutter. For example, MSI Lightning HD8970 vs. Asus Matrix / DirectCUII GTX780 showdown or something along those lines.

It's worth noting that this is precisely what I did with the GTX 680M P170EM article that apparently started this discussion, as I compared it with the HD 7970M. My intention all along was to merely provide a set of additional games in a one-off comparison. That's why there's some redundancy of sorts (DiRT 3 and DiRT Showdown), games that won't make any permanent testing list of mine (Diablo 3 -- just not demanding enough), etc. Skyrim without mods is still quite relevant on laptops, and in fact all of the games can push a laptop into sub-60 FPS with the exception of Portal 2 (unless you enable SSAA). I come from a different realm than Ryan, though, and jaggies while gaming do not usually irritate me all that much. I'll turn on 4xAA and even TXAA or other forms of AA if I'm well above 60FPS, because why not? But if I already drop into the mid-30s at times (which I do in gaming even on HD 7950 or GTX 580), I'm okay with stopping at 4xAA and I almost never bother with SSAA. In my experience, SSAA often causes periodic stutters, even if average frame rates are high.

 

[Jaydip]

Hey Jarred your grudge match review was awesome man well done
But I have to disagree with the Diablo 3 thing.If you play with 3 other friends on inferno it will cut your fps by more than half.I know it is hard to test for you guys but people saying Diablo 3 is easy on gpus haven't played it much.

 

[Red Hawk]

The new list seems alright to me. Some games like Diablo or Skyrim belong not because they particularly push the GPU, but because they are popular games. Graphics card reviews are useful not just because they reveal what card has superior theoretical performance, but because they reveal performance in specific games, and that's just as useful in low end cards like Cape Verde and GK107 as it is up at the high end. It helps readers decide which card to get. The fact is that Diablo 3 currently stands as the fastest selling PC game of all time, so people are going to be looking for performance reviews of it. Diablo 3 or Portal benchmarks may be useless to high end reviews, but then what is the use of 2560x1400 benchmarks of Battlefield 3 or Crysis on low end cards? Either both belong or we simply need separate lists for high end and low end cards.

I won't begrudge Crysis Warhead being removed, but I feel that Crysis should always be represented on benchmark lists because that series stands as a symbol for the goal of buying high end graphics cards. I'd use Crysis 2 (patched and with the official high resolution texture pack, of course) instead of Arkham City or Borderlands. Both represent Unreal Engine, PhysX, and TWIMTBP; Arkham City is a better benchmark IMO because of its DirectX 11 features but Borderlands is more recent

I agree with others that Dirt 3 should go as it's redundant to have both it and Dirt Showdown, and I would replace it with an AMD-tilted Gaming Evolved title instead. Perhaps Deus Ex: Human Revolution.

 

[SPBHM]

I'm pleased with Witcher 2 being added, the game is a real challenge for even the fastest GPUs with ubersampling on...

I would prefer to have F1 2012 instead of Dirt Showdown.