[TECH Report] As the second turns: the web digests our game testing methods

[Final8ty]

A funny thing happened over the holidays. We went into the break right after our Radeon vs. GeForce rematch and follow-up articles had caused a bit of a stir. Also, our high-speed video had helped to illustrate the problems we'd identified with smooth animation, particularly on the Radeon HD 7950. All of this activity brought new attention to the frame latency-focused game benchmark methods we proposed in my "Inside the second" article over a year ago and have been refining since.

Correspondence with AMD.
http://techreport.com/blog/24133/as-the-second-turns-the-web-digests-our-game-testing-methods

Driver software to be tweaked to reduce Radeon frame latencies in series of updates
http://techreport.com/news/24136/dr...e-radeon-frame-latencies-in-series-of-updates

 

[Keysplayr]

Excellent!

 

[nextJin]

This is incredible news, the AMD rep actually said they see the problem and even though a simple CAP update wouldn't solve it a new driver revision can.

Catalyst 13.1?

 

[Keysplayr]

This is incredible news, the AMD rep actually said they see the problem and even though a simple CAP update wouldn't solve it a new driver revision can.

Catalyst 13.1?

Right. Now all we need to see is if they do what they say and if the fix has any effect on the performance or the fps reported. It is very good that AMD acknowledged this problem.

 

[A5]

This is incredible news, the AMD rep actually said they see the problem and even though a simple CAP update wouldn't solve it a new driver revision can.

Catalyst 13.1?

Probably more like 13.3. This stuff takes time.

 

[Ferzerp]

Does this mean posters here will stop claiming that Tech Report is incompetent or purposefully misleading?

 

[KingFatty]

Quoting from the TR article:

The most intriguing revelation in Baumann's correspondence, though, concerns one specific technical contributor to the frame latency problems on HD 7000-series Radeons based on the GCN architecture: less-than-optimal memory management in software.

TR quoting the AMD rep:

Additionally, when we switched from the old VLIW architecture to the GCN core there was a significant updates to all parts of the driver was needed – although not really spoken about the entire memory management on GCN is different to prior GPU's and the initial software management for that was primarily driven by schedule and in the meantime we've been rewriting it again and we have discovered that the new version has also improved frame latency in a number of cases so we are accelerating the QA and implementation of that.

Summary from TR:
So a specific portion of AMD's driver code needs some additional attention in order to perform optimally on the year-old GCN architecture—and AMD has accelerated an overhaul of it after discovering that the new revision can alleviate frame latency issues. Wow.

My speculation:
Perhaps it does seem like there is room for further optimization of the GCN architecture, and perhaps there will be overall FPS increase as well as improving the hitching/microstutter.

In other words, perhaps the fix will involve all improvements, without needing to do any 'tradeoffs' between fixing FPS at the expense of microstutter.

 

[Ferzerp]

In other words, perhaps the fix will involve all improvements, without needing to do any 'tradeoffs' between fixing FPS at the expense of microstutter.

Anything like this is just speculation. You won't know until there is released software.

 

[Greenlepricon]

Hopefully we don't see any performance hits at all. Another boost would be nice but I don't think we're going to be seeing that much more on older titles. Well I'm looking to see if I notice anything different at all in the future with my card.

 

[HurleyBird]

Wow, good job TR.

 

[nextJin]

Well if they did fix the stuttering and also somehow improved framerates it would basically slap nvidia in the face. Personally I think they will improve the stuttering (much like nvidia has been doing for awhile now) but I doubt they will have any improvements and possibly a slight decrease in performance which would be odd coming from 12.11

I own a CFX 7970 setup for reference.

 

[SirPauly]

http://techreport.com/news/24136/dr...e-radeon-frame-latencies-in-series-of-updates

Awareness usually is a good thing based on it may improve the products. Good news!

 

[BrightCandle]

What is disappointing in this is that problems were reported to AMD in January 2012. Its taken them a year and this review you finally acknowledge the problem. It'll take longer to fix it. That is shameful customer service.

 

[KingFatty]

By acknowledging there is a problem, would that cause buyers to avoid AMD cards, driving down their price (I hope)?

 

[VulgarDisplay]

Does this mean posters here will stop claiming that Tech Report is incompetent or purposefully misleading?

I don't remember seeing a single person claiming that Techreport was imcompetent or misleading. People did say that more testing was needed to corroborate their findings.

Either way, AMD seems to acknowledge the issue and are going to fix it which is a great thing.

 

[tviceman]

A Tech Lead at Intel also chimed in, which was acknowledged in the OP's techreport link: http://forum.beyond3d.com/showthread.php?p=1689708

TL;DR he says frame latencies is every bit, if not more, important than fps averages. I personally think they are both equally important, as a combination of maintaining high frame rates along with consistent latencies is what ultimate provides the best experience.

 

[tviceman]

I don't remember seeing a single person claiming that Techreport was imcompetent or misleading. People did say that more testing was needed to corroborate their findings.

Either way, AMD seems to acknowledge the issue and are going to fix it which is a great thing.

After TR's posting the initial article, people said the tests were inconclusive.
After TR's posting side-by-side videos, people said more videos were needed and therefore inconclusive.
After another website posted more side-by-side videos (Borderlands 2 and something else), I really don't know what people said at that point I because I quit reading. But the Borderlands 2 hd7970 video had obvious stuttering.

The point is, there were people posting who wouldn't admit to something being off or wrong if AMD themselves had come forward right away and said it. The same thing happens when stuff with Nvidia drivers / performance crops up. People that have their own heads up their GPU maker's a$$'s get offended too easily and refuse to admit problems when they exist.

 

[VulgarDisplay]

After TR's posting the initial article, people said the tests were inconclusive.
After TR's posting side-by-side videos, people said more videos were needed and therefore inconclusive.
After another website posted more side-by-side videos (Borderlands 2 and something else), I really don't know what people said at that point I because I quit reading. But the Borderlands 2 hd7970 video had obvious stuttering.

The point is, there were people posting who wouldn't admit to something being off or wrong if AMD themselves had come forward right away and said it. The same thing happens when stuff with Nvidia drivers / performance crops up. People that have their own heads up their GPU maker's a$$'s get offended too easily and refuse to admit problems when they exist.

Claiming more testing is needed does not even remotely resemble stating that techreport is incompetent and purposefully misleading. What I said stands, no one made those claims. Stop being dramatic.

 

[Homeles]

Looks like I win. I love when I win arguments.

But look, everybody's a winner here. AMD users get smoother frame rates, and Nvidia got some brief publicity.

Claiming more testing is needed does not even remotely resemble stating that techreport is incompetent and purposefully misleading. What I said stands, no one made those claims. Stop being dramatic.

I don't know much about what went on in this board, but over at OCN, there were a lot of arguments made along those lines.

He's not being dramatic — it's the reality of the situation.

 

[VulgarDisplay]

Looks like I win. I love when I win arguments.

But look, everybody's a winner here. AMD users get smoother frame rates, and Nvidia got some brief publicity.

I don't know much about what went on in this board, but over at OCN, there were a lot of arguments made along those lines.

He's not being dramatic — it's the reality of the situation.

I still fail to see how a person stating that tech report's hypothesis needed more testing to confirm it was correct is the same as saying they are incompetent and misleading. Now you are also being dramatic.

 

[lavaheadache]

Why does a frame limiter "fix" these issues? Whether I game on my 30in or 120 hz monitor I use either a 60 or 120 fps frame limit. I do this for the reason of keeping temps and power use down. I don't need my cards running all out needlessly =) Why does this in return create smooth frame delivery?

 

[Ferzerp]

Flat out "call Shens"

http://forums.anandtech.com/showpost.php?p=34366335&postcount=40

So, are we ready to call Shens. on this, yet? There's just too much evidence, even from their own reviews, that contradicts their findings. Even looking at last gens results, where latencies were well into the range we're being lead to believe is noticeable, nobody's cards were stuttering then. And as you point out, nobody was saying AMD was "slower but smoother".

Here you specifically are suggesting it's the games they chose (despite it being a comparison and the issue only occuring on the amd hardware).

http://forums.anandtech.com/showpost.php?p=34366406&postcount=52

My main issue is that they are using two of the least smooth game engines available to test smoothness. Gamebryo and Dunia are two of the buggiest engines on earth.

Here is someone suggesting they are incompetent

http://forums.anandtech.com/showpost.php?p=34366703&postcount=59

Playing Skyrim on 7850 and 7950 through multiple drivers, texture mods etc.. never noticed this kind of behavior. For TR to do a crazy bench that contradicts every other site and their own prior review is too sus. 660ti beating 7950 by huge % in games where every review has it being defeated by huge %? Come on, something is messed up with their rig.

Here you yourself are not asking for more study but are flat out calling the accuracy of the whole thing in to question.

http://forums.anandtech.com/showpost.php?p=34367815&postcount=81

How can you call the frametimes accurate if one card appears to suddenly have lost nearly 50% of it's peformance when compared to past reviews of said card?

This must be the guy you keep calling no one (i.e. "no one made those claims")

http://forums.anandtech.com/showpost.php?p=34374330&postcount=123

This video is incorrect and so is TechReport, although I'm not surprised in the dishonesty in reporting or propogating it. Skyrim has a vsync option by default (called iPresentInterval=0 in the SkyrimPrefs.ini file). The Creation engine really depends on this and some pretty funky glitches happen if you turn it off and get high FPS. TechReport clearly turned this off since their cards are getting over 60FPS with a 69FPS average on the 7950 and 74FPS average on the GTX 660Ti. So not only are people trying to prove a point with only a single game, but what they're actually using is TechReport breaking the game and then providing dishonestly presented data.

Ok, I got bored after going through 1/3 the other thread. That's just a small sample of what doesn't appear to me to be "claiming more testing is needed" but posts that looked designed to suggest that the whole thing was just wrong.

 

[Mezzanine]

So this will be fixed with a driver revision and the 7950 will continue to roflstomp the 660Ti...business as usual.

 

[VulgarDisplay]

Why does a frame limiter "fix" these issues? Whether I game on my 30in or 120 hz monitor I use either a 60 or 120 fps frame limit. I do this for the reason of keeping temps and power use down. I don't need my cards running all out needlessly =) Why does this in return create smooth frame delivery?

Does it, or does it just fool fraps? When I set a frame limiter to say around 60fps on my 120hz monitor my frame times in fraps may look great, but the output is horrendously rough looking. Perhaps I'm just able to detect frames better than most people.

Maybe the upcoming drivers that address the problem will give me smooth output at 60fps on my 120hz monitor.

 

[blackened23]

Why does a frame limiter "fix" these issues? Whether I game on my 30in or 120 hz monitor I use either a 60 or 120 fps frame limit. I do this for the reason of keeping temps and power use down. I don't need my cards running all out needlessly =) Why does this in return create smooth frame delivery?

You can't play skyrim without vsync. If you use the ini modification to disable it, the physics engine breaks. Although, I thought beth removed the option to disable vsync in the .ini file? Because they acknowledged on their forum a long while ago that disabling vsync screws the game up - they mentioned this not long after release.

Stuff like this happens with vsync off:

http://www.youtube.com/watch?v=cRso9WFY6fo

Was TR testing with vsync off?