Is Anandtech's gaming benchmark suite obsolete?

[lavaheadache]

Maybe if you reread the part that you didn't put in bold you would have realized that I agree with the OP just disagree with an immediate departure of a method that has proven for decades to give a general point of performance unitl we have an accurate way to measure frame latencies.

I know you were very busy trying to be right but there was no need to be right when I wasn't telling him that he is wrong just a little premature

 

[Homeles]

Maybe if you reread the part that you didn't put in bold you would have realized that I agree with the OP just disagree with an immediate departure of a method that has proven for decades to give a general point of performance unitl we have an accurate way to measure frame latencies.

I know you were very busy trying to be right but there was no need to be right when I wasn't telling him that he is wrong just a little premature

But that's not even what he's saying. Hence the straw man.

 

[nextJin]

I believe all sites should have a set standard where a certain set of cards are individual from the actual reviews which is updated every review. For instance;

In the first few pages of the review it is all on a database that is updated everytime a new driver is released (WHQL & latest beta). The last few pages look specifically at the card being reviewed within that series.

The database portion uses a combination of FPS + Latency metrics using a 15 game suite and latest benchmark programs. That way anyone reading any GPU review would get the latest performance information no matter what card they are looking at or when it was reviewed.

 

[lavaheadache]

Is Anandtech's gaming benchmark suite obsolete? - Title of thread

"Do you think Anandtech's CPU and GPU gaming benchmark suite should be based on frame latency rather than average frames per second"- Thread body


"so lets just dump an age old standard in lieu of a new to most people method with hard to quantify measurement as of yet? um..... no!


I think latency results are very important but to just flat out switch to this method bring about erratic and erroneous results that would probably differ drastically from different programs are reviewers. How about we wait till "they" can come up with a reliable and repeatable way to measure latency before jumping off the deep end." - what I said


I'm not sure I see what the problem is

 

[Homeles]

I'm not sure I see what the problem is

The problem is that you don't understand the meaning of words. Since you want to argue semantics, allow me:

Is Anandtech's gaming benchmark suite obsolete? - Title of thread

FPS averages by their lonesome are obsolete. FPS averages plus latency results are not obsolete. You don't have to remove FPS averages in order to catch up with the rest of the industry -- you just need to add frame latency results.

"Do you think Anandtech's CPU and GPU gaming benchmark suite should be based on frame latency rather than average frames per second"- Thread body

Based on suggests that the benchmarking suite should revolve around frame latency. Nowhere does it suggest that it should supplant FPS averages entirely.

Q.E.D.

 

[Grooveriding]

No, but this thread sure is.

 

[notty22]

I think frame latency is important, but it can change as fast as 1 or 2 settings in a games graphics options, and be fixed or broken by a driver.
Here is a site that has done a few reviews on AMD /Nvidia cards, different tier levels and usually 2 graphical setting for each game. Sometimes the settings will give a win to one card, then the other in the same game with different options throughout the review.
http://uk.hardware.info/reviews/404...n-vs-nvidia-geforce-gtx-660-frametimes-review

 

[lavaheadache]

The problem is that you took my comment and made of it what you did. I was mearly saying that I don't think that Anandtech's bench's are obsolete because they aren't measuring frame times yet.

Measuring frame times is still in its infancy with no appropriate or truly accurate way to measure them. I'm sure when there is a good method they will adopt it. Don't put the horse before the carriage and alienate the masses until it is ready to go prime time.

I applaud Tech report and the likes for breaking ground with this stuff. Let them be pioneer's along with sites like ABT and probably a few others. Do we really need every tech site to jump on it and give us their spin on something that is very complicated and hard to get a true measurement.

I'll let you think you are "right" but all you did is misunderstand and then just try to cover it up by dismissing me as stoooopid by using big words and terminology. Kudos to you. I love interwebs smart guiz.

 

[BrightCandle]

The frame latencies are accurate, we have multiple reviews showing that they coincide with the games own reporting of render moments. This technique has already been proven to measure what it says it measures. Further to that we now have some results for the actual screen outputs showing the impact of this on the output that shows that post render that sometimes the picture is much worse (or better) than the frame time measure. Ultimately the state of the art in benchmarking is frame times (which is basically FPS just measured differently and more accurately without averaging) and a frame capture matched up to the frame times.

FPS as a metric was an ok proxy for this but now we have dramatically better techniques on offer that have already been shown to produce useful, reliable and more accurate data it makes FPS as a measure obsolete. There is no question about frame times, many sites have tied them up with the games reported moments, its an accurate dependable technique.

 

[coolking]

I don't want Anandtech to abandon average fps altogether, i want them to add frame latency benchmarks as well. I base my purchasing decisions on their articles and reviews and i want them to be reliable and up to date.May be other reviewers are trying to get more viewer ship by introducing new methods but,at the same time,i expect Anandtech to write atleast an article to tell us why they are not using these new methods in their benchmarks.

 

[njdevilsfan87]

I'm very happy to see this being widely adspoted. Frames alternating between 10ms and 30ms are going to "feel" something like a consistent 25ms, and not 20ms. It's just the way we are hardwired. The former will display 50fps while the latter will display 40fps. Thus the former will be assumed to provide a better gaming experience...

This is why micro-stutter sucks is incredibly noticeable at sub-60fps. As far as keeping your fps above 60 to avoid it's problems, let's say the above frame times are halved; The difference then becomes 100fps vs 80fps. There you just happen to be in a range where both will feel smooth and thus be much more difficult to tell the difference.

Glad to see AFR finally being exposed for what it is. That is, a load of crap when it's advertised as near perfect scaling, because even though it's true in one sense, it's not in the one that completely matters. This is why for those of us who have witnessed this first-hand, we have a tough deciding between something like GTX 690 vs GTX Titan. Because we already sure as hell know that something like GTX 660 SLI is not going to provide a better gaming experience than a single GTX 680 despite putting up higher frame numbers.

And this is a very real issue for those who are still unaware of it (and may even be running CF or SLI setups), otherwise we wouldn't be seeing any of this.

Edit : And the example I gave is actually a very poor one because micro-stutter becomes much more erratic in real gaming scenarios.

 

[Stuka87]

I actually wish AnandTech would move to shows the FPS for the whole test. So we can see the highs and lows. Rather than just showing an average across the board. This to me is much more informative.

 

[coolking]

Have a look at this article,Ryan smith of Anandtech take on frame latency and why they are not including it in their benchmars

http://www.anandtech.com/show/6857/amd-stuttering-issues-driver-roadmap-fraps

 

[lavaheadache]

Have a look at this article,Ryan smith of Anandtech take on frame latency and why they are not including it in their benchmars

http://www.anandtech.com/show/6857/amd-stuttering-issues-driver-roadmap-fraps

After reading that article I have a question regarding this

"The fact that NVIDIA seemed to have figured all of this out much earlier was a point of frustration for AMD. The company likely left non-negligable amounts of performance on the table over the years, which could've definitely helped in close races."

Does anybody think that AMD will implement some of these fixes in legacy drivers? Namely for HD 4XXX and up? I suppose even including them for the HD 2XXX and 3XXX would be feasible since they are all a using unified shaders and can still play most games reasonably well at lowered resolution and details.