Help Design The Next AnandTech GPU Benchmark Suite

[happy medium]

Not true, the biggest performance killer by far is shader quality, by a factor of 3-4 times. The two you mentioned aren’t even close.

I thought it was DOF? or is that just in some games?

 

[Mir96TA]

I agree, adding GPGPU tasks such as bitcoin mining or F@H numbers would be an excellent supplement to 3D results.

I believe this would give our bench an edge over competition as these GPGPU applications have significantly been rising in popularity in the past year.

I realise that for both F@H and Bitcoin, one of the two brands is at a disadvantage. But if this is ment for a future GPU bench, i think stuff like that should be considered as the cherry on top of all the other good stuff.

For 28nm, AMDs GCN architecture and Nvdias Kepler might improve or balance the scores for applications such as these.

Bitcoin mining i dont think can be done really as Nvidia is getting really bad performance. And there is no standardized client etc. etc.

We have to go buy what are most popular client software and widely have been used.
Second it will also give a chance to Chipset Manafac to assign a team to make a client or activily off load their help to people who makes those client.
Lot of people have bought Nvidia or Ati based chipset Video Card because of that reason.

***************
Also I think bench mark should also include based on a system which does not have a TOP CLASS MB or CPU combo.
It can be second class or budget machine with those Newer or High end Video card. So Cx get an idea if they just want to do a video card upgrade what sort of perfomance increase they migt see

 

[Ryan Smith]

Is milkyway@home any more or less valid then? GPGPU is already the direction amd/nv are going, whether it is for deferred rendering performance or double precision distributed computing.

I'm afraid I don't know much about milkyway@home. Could you please tell me more?

 

[gorobei]

its another distributed computing ala boinc. they are mapping stellar streams in an attempt to learn about stellar development. some of the guys in the AT DC forum can tell you more about it than i can.
http://en.wikipedia.org/wiki/MilkyWay@Home

there are cuda and ati clients, but as it is dp and somewhat hash based the performance deltas favor amd now.

 

[GodisanAtheist]

By "canned benchmarks" I assume you mean timedemos and the like? If so, that's not a request I'm going to be able to meet.

Our (AnandTech's) principle testing methodology is to stick to the scientific method as close as humanly possible. That means running the same exact test on the same exact hardware and only varying the hardware being tested. And that means timedemos and other automated forms of testing when possible. Anything where I can influence the results of a test by my actions is used as a last resort.

There's a place for such testing and I'm not disparaging it, but it's not a methodology we consider suitable for our needs.

-I'm not sure this is what he meant, but I understood it more as built in benchmarks, where games have a nice and convenient "benchmark me" button. Using repeatable segments of gameplay (such as an on rails portion of a game) or using the in game console to "record" button presses and play them back is perfectly acceptable, repeatable and scientific and I think we all appreciate that. We can go to [H] for a more subjective opinion.

However, there are a LOT of built in benchmarks that are hardly representative of actual game play (the Company of Heroes benchmark immediately jumps to mind) that can and do stress aspects that aren't nearly as important as others during gameplay. These benchmarks also provide ample opportunity for targeted optimizations that skew expectations for real world performance.

As a counterexample, your BC2 opening sequence/waterfall benchmarks are repeatable segments that are genuinely representative of actual gameplay that would be very difficult for AMD/Nvidia to specifically optimize for, so thumbs up there.

That's my interpretation of what he says are "canned" benchmarks. He might mean what you thought though, in which case I disagree with him.

 

[Elixer]

I think a more even approach is best.
Have some DX 9 & 10 & 11 games, and also throw in some openGL games as well.
Not sure having any game that is tuned for a specific GPU is a good idea, since the results can be really skewed.
I think the same thing about games that enable APIs that are pretty platform specific, and are really bad with the other platforms. (Physfx comes to mind)

It might also be nice to include this: http://openbenchmarking.org/

 

[ElFenix]

Could you please go into more detail and what you mean with regards to sound and heat characteristics? As for surround gaming, you will be seeing more of it. It's a specialty setup though, so it won't be in every article.

i'm not sure what the other poster was referring to, but with spcr possibly shutting down there might be a real dearth of quality sound related discussion from review sites. recording dB levels from inside your own lab without much real control of the sound floor isn't the same, and frankly, that wasn't the most useful info spcr provided. good quality recordings of the noise and a good subjective evaluation of the sound are a useful purchasing tool.

 

[Borealis7]

Witcher 2 has to be on it for sure, and of course power and temperature charts.

one other thing that interests me, and perhaps other people would like to see it too, is that often times we see threads here in VC&G forum about "will X CPU bottleneck Y GPU??" so a chart to try and find bottlenecking of a GPU by a CPU would be nice to see, and it will help people choose the right GPU for their machine and avoid "overkill" on their puny CPUs.

perhaps take an i7-2500K CPU and underclock/overclock it and see the relative gains in 1 or 2 games for the GPU you are reviewing in terms of (delta FPS over delta Mhz) and see if you hit a plateau.

(EDIT: i now see that Termie has already requested that, so i second his suggestion)

another suggestion i thought of is overclocked performance divided into 2: include 3 bars in the performance chart, one for Core overclock and one for Memory overclock and the 3rd is for combined overclock.

thanks!

 

[Jacky60]

The last I heard of Arma was that it was incredibly buggy. I don't play it myself, so I'm curious to hear as to whether this is still the case.

.

You're right it was incredibly buggy but due to several patches and a good effort by the developer and several patches its very solid now.

The built in benchmark is repeatable and consistent enough to garner meaningful results-usually meaning your gpu needs much more horsepower.

 

[Lonbjerg]

So if you guys have any games or GPGPU applications that you'd like to see on our next GPU benchmark suite, now's the chance to speak up; we're interested in what you guys think. With that said if you could please provide a basic rationale/reason behind your request it would help the process a lot. And on a side note, for GPGPU benchmarks they need to be cross-platform (e.g. F@H is out because they don't have a proper AMD client right now).

So AMD is getting slack for dragging their feat?
For not investing in GPGPU...the benchmarks will neglect this fact?

You should include the most POPULEAR GPGPU software (even BitCoin)...otherwise you present us with a incomplete and rather useless GPGPU comparison.

NVIDIA is SIMD and AMD is VLIW...show us how the architechture infuences program, so that we may jugde better on what to buy for what task.

 

[bart1975]

Use free benchmarks that anyone can get easily so we can compare with our systems.

 

[Arkadrel]

I think they should mostly be DX11 games, and be games that dont favor one maker to much over the other.

How about:
Aliens vs. Predator @ 2560x1600, of course you ll do alot of differnt resolutions. The 580 is down to like 28 fps, and a 6970 around 26 fps. Game can kick modern GPUs in the nuts.

Metro 2033: its still a pretty demanding game. Game can kick modern GPUs in the nuts.

Battlefield 3: Its the most breathtakeing game going by looks so far. Game can kick modern GPUs in the nuts.

And one more game suggestion would be DiRT 3 or soon to be releasing F1 - 2011.

Might not be a bad idea, to have DiRT3 or F1-2011, just so all the games reviewed arnt First Person shooters.

 

[Lonbjerg]

I think they should mostly be DX11 games, and be games that dont favor one maker to much over the other.

Way to make the benchmark irrelevant.
Test the most selling/most populear games to give as broad a foundation as possible to evaluate ones future purchase.

If company X run better in more games than company Y....tell it to me.

Don't hide and mask the performance by cherrypicking games so company X and Y look the same ...show us the real deal.

 

[Spjut]

As it is now, you only seem to test DX11 GPUs in the DX11 games...I would like to see one or two DX10/10.1 GPUs in there, to see how the older generation's performance in DX10/10.1 compares to the modern generation's performance in DX11

 

[Lonbjerg]

As it is now, you only seem to test DX11 GPUs in the DX11 games...I would like to see one or two DX10/10.1 GPUs in there, to see how the older generation's performance in DX10/10.1 compares to the modern generation's performance in DX11

Since a 9800 is the most reported card still in peoples PC's it makes perfect sense:
http://store.steampowered.com/hwsurvey
NVIDIA GeForce 9800 5.01%

I would throw in some 8800/9800 perfomance level hardware, so people can better see what they will upgrade to....how big the "jump" is.

 

[zebrax2]

The Witcher 2
Crysis Warhead
Crysis 2
BF3
Serious Sam 3
Starcraft 2 or another popular RTS
DiRT3 or F1-2011

 

[Borealis7]

i also think there should not be any Unreal Engine 3 based games, as any high end card can do 100+ FPS at those without AA. its getting old.

lol @ 400FPS in Street Fighter IV. you know who i'm talking about.

 

[nenforcer]

It probably goes without saying that the IDTech 5 game, Rage, should be included.

I'm curious as to how it will perform on PC seeing that it is still only DX9 based and will probably be the last big / huge DX9 focused title.

 

[AtenRa]

i also think there should not be any Unreal Engine 3 based games, as any high end card can do 100+ FPS at those without AA. its getting old.

lol @ 400FPS in Street Fighter IV. you know who i'm talking about.

The new Unreal 3 engine can make your card beg for more power (see the Samaritan demo)

It would be great to be able to use the Samaritan demo as a benchmark

I will agree that older games using the first unreal 3 engine are obsolete for benchmarking.

It probably goes without saying that the IDTech 5 game, Rage, should be included.

I'm curious as to how it will perform on PC seeing that it is still only DX9 based and will probably be the last big / huge DX9 focused title.

ID tech 5 Engine used in Rage is OpenGL and not DX-9

 

[Mr. President]

Not true, the biggest performance killer by far is shader quality, by a factor of 3-4 times. The two you mentioned aren’t even close.

It may have been poorly worded on my part. What I meant is that object detail and shadow detail can throw a spanner in the works when you're benchmarking a GPU.

I can't remember the exact numbers, but the difference between those settings on medium and maximum was to the tune of 30 fps vs. 45 fps on my 3ghz Q6600. And that was with everything else on minimum (including resolution) on an HD6850. And it wasn't a fluke either because I retested this on three CPUs and four GPUs, and performance scaled almost linearly with the number of draw calls.

I can't speak for how it is on the i5/i7 series, but it's almost certainly the reason why the game scales with CPU and memory subsystems while still being largely GPU bound. That makes it a confound in testing GPUs is my point.

 

[Ryan Smith]

i'm not sure what the other poster was referring to, but with spcr possibly shutting down there might be a real dearth of quality sound related discussion from review sites. recording dB levels from inside your own lab without much real control of the sound floor isn't the same, and frankly, that wasn't the most useful info spcr provided. good quality recordings of the noise and a good subjective evaluation of the sound are a useful purchasing tool.

Humm, that's an interesting request. I'll have to talk to our other editors and see what we can put together. I'm not sure what we can do with regards to subjective evaluation (we try to avoid subjectiveness as much as possible), but audio recordings may be possible.

Use free benchmarks that anyone can get easily so we can compare with our systems.

I sympathize with the request, but doing so would greatly limit what we can benchmark. Many game demos are out of date and/or lack their benchmarking tools, and other free tools are synthetic benchmarks (which we avoid). Gamers are using these GPUs with retail games, and our tests need to include those games.

Battlefield 3: Its the most breathtakeing game going by looks so far. Game can kick modern GPUs in the nuts.

I'm focusing on games that are out at the moment. Battlefield has been a good benchmark for us in the past, and being the first major DX10+ game makes it a significant entry, but it's far too early to say if we're going to use it. It's not even in beta yet.

i also think there should not be any Unreal Engine 3 based games, as any high end card can do 100+ FPS at those without AA. its getting old.

This is actually an interesting perspective since I have the opposite point of view - I'm currently looking for a new UE3 title to keep in the suite. UE3 is used in so many games that I'm hesitant to ignore it, but you're right in that the framerate on most games is rather ridiculous on high-end cards.

So let me ask you guys this: do you play many UE3 games (i.e. am I overestimating its importance?), and given that we largely use the same benchmark suite from top to bottom, would you be okay with UE3 not showing up in mainstream/low-end video card reviews?

It probably goes without saying that the IDTech 5 game, Rage, should be included.

I'm curious as to how it will perform on PC seeing that it is still only DX9 based and will probably be the last big / huge DX9 focused title.

Rage right now is on the short-list for future inclusion, so long as we don't find any problems when vetting it. As long as games are still using OpenGL I believe it's important to have an OpenGL benchmark as part of the suite, and as the first commercial OpenGL 3.x title it's going to be a much more challenging game than the current collection of id Tech 4 games.

 

[notty22]

I keep seeing a updated UE3 engine mentioned in gaming news, future developements.
Epic Games to provide game engine behind 'Batman: Arkham City

 

[Genx87]

I'd think BF3 should be out around the time AMDs cards show up. I would like to see it included. Crysis 2 in DX9 and DX11.

 

[Anarchist420]

Mainly differences in image quality between the two vendors, and an examination and analysis of what optimizations they may or may not be using.

As for FPS I'd always include the following:
the min and the average matter a lot.
highest quality driver settings, even if they bring a card to its knees
all driver optimizations off, or at least whichever ones can be turned off and if one vendor forces optimizations but the other doesn't, then don't benchmark, or report the frame rate as 1/2 for the vendor who forces optimizations because optimizations should never be used if they decrease image quality, regardless of how few people may be able to tell the difference)
highest quality in game settings and all features on.
1680x1050, 1920x1200, and 2560x1600. Always include something at least as low as 1680x1050.
Classic openGL games with each new hardware release. That will pressure nvidia to improve their GL performance.
In depth analysis of how close they match the reference rasterizer (if Vendor A is further away, it's not always a bad thing because that vendor could have more error because they're forcing higher precision)

 

[cusideabelincoln]

So let me ask you guys this: do you play many UE3 games (i.e. am I overestimating its importance?), and given that we largely use the same benchmark suite from top to bottom, would you be okay with UE3 not showing up in mainstream/low-end video card reviews?

I have UE3 games in my library. Wouldn't you want to keep UE3 games benchmarked for midrange and low end cards? That market segment is where we'll see cards make a more notable impact on playability.