Is Anandtech's gaming benchmark suite obsolete?

[coolking]

Do you think Anandtech's CPU and GPU gaming benchmark suite should be based on frame latency rather than average frames per second?

Techreport has adopted this approach,do you think Anandtech should do the same?

 

[Revolution 11]

Anandtech is supposedly waiting for better equipment before frame latency testing can start. I won't really care about the lack of these benchmarks until the next generation of cards is here.

 

[coolking]

Anandtech is supposedly waiting for better equipment before frame latency testing can start. I won't really care about the lack of these benchmarks until the next generation of cards is here.

So do you think that benchmarks based on these parameters are unreliable as of now?

If so then why so many reviewers are using frame latancy based benchmarks?

 

[Ferzerp]

To be clear, "frame latency" is nothing different than frames per second. The difference comes in when you get more granular (e.g. "frame rate" of each individual frame)

What I think you're trying to ask is:

Is the mean frames per second averaged across an entire test time a good indicator of actual performance?

If, over 1 second, I have 100 frames, each delivered every 10ms, I have 100 fps.

If, over 1 second I have 99 frames delivered in 5 ms increments, and then 1 frame that isn't replaced for 505ms, I also have 100 fps.

Both cards have performed the same raw work within that second, and if they were pre-rendering things, etc, they would, in this case, be of equal value. If that's all we're going for, then both being equal is reasonable.

If our goal is to provide a number that will tell you which will look better to you if you play a game, well, obviously the former would be a great solution, and the latter, while being exactly the same when only looking at the mean, is a useless piece of hardware.


Whether you want to use the term latency or frames per second (they're the same value, just inverses), it makes no difference. You just have to get more granular in such a way that any pauses like the ones that I manufactured in my worst case scenario type example above become apparent.

 

[lavaheadache]

so lets just dump an age old standard in lieu of a new to most people method with hard to quantify measurement as of yet? um..... no!


I think latency results are very important but to just flat out switch to this method bring about erratic and erroneous results that would probably differ drastically from different programs are reviewers. How about we wait till "they" can come up with a reliable and repeatable way to measure latency before jumping off the deep end.

I take it you probably already drive an electric car eh?

 

[DominionSeraph]

Frame latency for single cards does nothing but cater to fanbois who need molehills to make into mountains.

Show me a graph with VSYNC enabled which shows anything meaningful.

 

[BrightCandle]

FPS is dead, the new techniques have shown significant problems with the current implementations, problems that many users report before understanding the data. Frame latency testing is simply good science and Anandtech is totally out of date. Anandtech isn't even at the fps over time point, they were recently using a single fps across an entire run which is obviously rubbish.

Anandtech is about 4 years behind in modern GPU benching, they no longer produce values of any worth at all. We have a lot of religious people here who seem to not believe the cold hard data, and the videos, and the pictures and all the other mounting evidence that shows fps is completely worthless as a metric.

 

[Gryz]

FPS is still a relevant number.
It denotes the raw computation power of a particular videocard in a particular game.

But we need an extra number.
A number to tell us how much stuttering is going on.
And nobody has come up yet with the best number to describe stuttering.
Nor how to name that number.

Maybe "average deviation of the average time to calculate the next frame" ?

 

[Ferzerp]

Maybe "average deviation of the average time to calculate the next frame" ?

There is a problem with using means. 1 outlier doesn't show in a mean, but it is jarring when you're playing. Even if only 1% of the frames have problems, that's enough to ruin the motion illusion.

I find this format is the most telling. Specifically, you can see here how the 680 has a lot of issues delivering frames smoothly (and well, smoothly, slowly in this game) with sleeping dogs. It isn't because of the spike at the end, they all have that, but it's how *wide* it is. This particular example, chosen at random, is a game that nvidia cards have issues with everything enabled.

 

[Will Robinson]

FPS is dead, the new techniques have shown significant problems with the current implementations, problems that many users report before understanding the data. Frame latency testing is simply good science and Anandtech is totally out of date. Anandtech isn't even at the fps over time point, they were recently using a single fps across an entire run which is obviously rubbish.

Anandtech is about 4 years behind in modern GPU benching, they no longer produce values of any worth at all. We have a lot of religious people here who seem to not believe the cold hard data, and the videos, and the pictures and all the other mounting evidence that shows fps is completely worthless as a metric.

Please go post somewhere else then seeing you despise AnandTech so much.D:
I'm sure you could go join the other AT haters at AlienbabyTech they would be bound to welcome you to their little club.

 

[Revolution 11]

So do you think that benchmarks based on these parameters are unreliable as of now?

If so then why so many reviewers are using frame latancy based benchmarks?

I am not saying anything like that. I do not know enough about latency benchmarks to say if they are reliable or not. For the record, I do want Anandtech to examine this issue and include the new benchmarks because they always bring extra detail and reliability to the issue.

But I don't really mind if there are no benchmarks until winter 2013 because there won't be any drastically new cards until then.

Agree 100% with everything lavaheadache stated.

 

[Imouto]

FPS is dead, the new techniques have shown significant problems with the current implementations, problems that many users report before understanding the data. Frame latency testing is simply good science and Anandtech is totally out of date. Anandtech isn't even at the fps over time point, they were recently using a single fps across an entire run which is obviously rubbish.

Anandtech is about 4 years behind in modern GPU benching, they no longer produce values of any worth at all. We have a lot of religious people here who seem to not believe the cold hard data, and the videos, and the pictures and all the other mounting evidence that shows fps is completely worthless as a metric.

We got the message 3 months ago, you don't have to stain every single thread around.

I wonder the degree of obnoxiousness that you will peak once PCPer releases their superduper final test suite.

 

[coolking]

I want to add another thing to this discussion.What do you guys have to say about CPU gaming benchmarks based on frame latency?Because if we take into account only average fps then any new generation CPU,whether it is fx 81350 or i7 3770k,is capable of playing every game.So in this scenario,all CPU gaming benchmarks becomes irrelevant.

Have a look at this article and tell me what do you think?

http://techreport.com/review/23246/inside-the-second-gaming-performance-with-today-cpus

 

[coolking]

I am waiting for your response guys.

 

[Fanatical Meat]

What they should have is updated charts with current drivers. The BF3 results are extremely off for AMD cards

 

[vampirr]

Anandtech results are really off in GPU section and CPU section should also be extensively updated with more games and update the benchmarks of games that they already tested CPU's and GPU's ones.

Anantech would be a primarily site to see benchmarks if they had almost all categories in one if they could do extensive scenarios with various configurations.

Anyway Anandtech benchmark results are so off and do they even update once a year?

 

[Homeles]

so lets just dump an age old standard in lieu of a new to most people method with hard to quantify measurement as of yet? um..... no!

Hooray for straw man arguments. Even The Tech Report still includes FPS averages.

False dichotomy as well...

 

[Fx1]

To be honest i dont think Anands reviews of GPU's has been that good for a long time. I often find myself seeking out far more interesting tests from other websites.

Anand needs to bring his testing into 2013 quickly

 

[parvadomus]

Anand's bench section is outdated, in both drivers (I cant believe some scores really) and games. Its just misleading.

 

[OILFIELDTRASH]

Anandtech reviews are the gold standard IMO. They aren't bought and sold like many of the other sites are. That in itself keeps me coming back.

 

[lavaheadache]

Hooray for straw man arguments. Even The Tech Report still includes FPS averages.

False dichotomy as well...

Is strawman argument the phrase of the month? How is this a strawman argument? I ought to give myself a vaction from this GD place because some of you people really need to get a GD life. Fight, finger point, bicker, use phrases that which you have no idea what they mean but you read about some goon using it somewhere.

 

[Homeles]

I like the articles, the architectural detailing, and insider information that are in their articles. They're well written. The data that they collect is the problem.

 

[Homeles]

Is strawman argument the phrase of the month? How is this a strawman argument?

You ranted about the OP wanting FPS averages completely abolished.

I'm going to give myself a vaction from this GD place because some of you people really need to get a GD life. Fight, finger point, bicker, use phrases that which you have no idea what they mean but you read about some goon using it somewhere.

You're the ignorant one here. You clearly don't know what a straw man argument is. It's an attack on a view that your opponent does not hold. You created the false idea that the OP thinks that FPS averages should be removed from all reviews.

Please do leave. And don't come back until you've educated yourself on logical fallacies, have improved your reading comprehension, and have learned how to make a strong argument.

 

[lavaheadache]

You ranted about the OP wanting FPS averages completely abolished.
You're the ignorant one here. Please do leave.

I ranted?

embellish much?

 

[Homeles]

Rant: to talk in a noisy, excited, or declamatory manner.
http://www.merriam-webster.com/dictionary/rant

You are clearly ranting:

so lets just dump an age old standard in lieu of a new to most people method with hard to quantify measurement as of yet? um..... no!


I think latency results are very important but to just flat out switch to this method bring about erratic and erroneous results that would probably differ drastically from different programs are reviewers. How about we wait till "they" can come up with a reliable and repeatable way to measure latency before jumping off the deep end.

I take it you probably already drive an electric car eh?