[TECH Report] As the second turns: the web digests our game testing methods

[Black Octagon]

Frame limiting is a horrible fudge to introduce an artificial bottleneck in the rendering thread at the moment of providing the frame, rather than allowing that bottleneck to be the GPU. In my case I don't need to do this as I can rely on double buffering for this sync point which is based on the physical sync of the card to the monitor. Since I intend no user input there is no chance of input latency issues.

Ok good point. Carry on (and THANKS for what you are undertaking)

 

[VulgarDisplay]

Sigh, typical tiresome VC&G flamewars. Notice how the ONLY on-topic post in this ENTIRE PAGE is the one at the very top?

Back on topic: I agree that the best benchmarks going forward will probably have to show a combination of both raw 'frames per second' as well as frame latency. However at this stage I think the jury is still out on the most effective (and cost efficient) way to test the frame latency. I personally suspect that the high shutter-speed camera method will be the best way to measure true in-use smoothness, however doing it properly could be an expensive investment for many review sites funded mainly (or solely?) by advertising...

Maybe you have more posts per page than I do, but the only thing I see at the top of this page is someone trolling me.

 

[VulgarDisplay]

So far we seem to have seen 3 different problems with frame times. We have seen microstutter (continuous back and forth around an average frame time), small jitters (a jump or dip in frame time followed by the opposite for a number of frames before returning to normal) and spikes (a large enough jump to stop the motion for a long time). I guess the old one of FPS should also be on the list of potential problems.

Seems to me its these four things are what everyone needs to know about human eye perception to get to grips with todays frame time graphs. If we had such data we could search through these charts and find out how many instances there were of these problems. If there is anything else you think should be on the list let me know.

What I am doing is writing a basic piece of software that simulates these changes in simulation time and effective hand crafts every frame of a basic simulation so that its ensured you get the precise frame you are meant to. This necessitates a very simple scene rendered mostly on the CPU to avoid GPU stutters. I can't help using exclusive mode in D3D to do this properly with control over the buffering but hopefully I can avoid problems on the GPU if the scene is 2D and very simple.

Right now I am considering only doing it with vsync on because then I can ensure what I send is what you see and vsync off would mean that I don't know what the GPU is showing at any point in time. The point is to find the thresholds of human perception of these problems and not to minimise latency (a different problem).

The other problem I have with this approach is that fraps would always show a perfect 16ms/8ms frame time and never show the stutters, because the GPU is seeing carefully controlled frames at very regular intervals. So I will have to produce my own timings trace to show the pattern being simulated as fraps does not show it.

Anyhow I am getting there, I have a basic simulation now and I have some microstutter introduced that I can see rendered perfectly at 60fps.I have a lot more to do to make it a releasable piece of software but I am encouraged that I know how to simulate these problems now and can ask the users if the animation they saw was smooth which combined with double blind not knowing what it is they are looking at should give me some decent results.

Thoughts?

This sounds really promising. It's exactly what I was talking about doing, and I'm glad someone has to skills to actually make something work.

If you get your simulation to the point where you can adjust a slider or something to increase or decrease the latency spikes then a tester could do a bunch of blind tests and come up with an average number for when people actually see the issue.

It's obvious to me that the goal should be completely consistent frame times from both vendors that are as low as possible, but if the actual worst case scenario before visible issues are easily apparent then both vendors have a number to shoot for and gamers win.

 

[Lonbjerg]

This sounds really promising. It's exactly what I was talking about doing, and I'm glad someone has to skills to actually make something work.

If you get your simulation to the point where you can adjust a slider or something to increase or decrease the latency spikes then a tester could do a bunch of blind tests and come up with an average number for when people actually see the issue.

It's obvious to me that the goal should be completely consistent frame times from both vendors that are as low as possible, but if the actual worst case scenario before visible issues are easily apparent then both vendors have a number to shoot for and gamers win.

The "average" won't cut it...no one size fits all in human perception.

 

[VulgarDisplay]

The "average" won't cut it...no one size fits all in human perception.

That's why you get an average, because one person isn't enough to draw a conclusion...

They're looking for the point where NONE of those tested can sense anything strange in the motion. That's the point where they can say what is acceptable and what isn't.

 

[BrightCandle]

That's why you get an average, because one person isn't enough to draw a conclusion...

They're looking for the point where NONE of those tested can sense anything strange in the motion. That's the point where they can say what is acceptable and what isn't.

Exactly this. By no one we probably mean 99.999%. The data is likely to come out as a bell curve of some description but with some odd charateristics (huge spike around 17ms if its not more noticeable to most below that). Because for a product that sells millions to only have 10's to 100's of people see a problem is probably OK. More than that and you have a problem.

 

[BoFox]

ABT who? Never heard of them until the silly blowout here last year and I will look elsewhere for an honest review. I never even see ABT articles popping up in search engines.

Hmm, that's interesting - I always saw ABT articles popping up pretty high with some searches..

Like if I type this in google:
hd 7970 frame time

ABT's article comes up at #5, after 4 links to techreport articles.

:awe:

Honestly, what was wrong with ABT's article?

It didn't make HD 7970 look all that terrible - just that it performed abysmally in Hitman: Absolution - which can be clearly shown with youtube videos of this and GTX 680.

Hitman GTX 680
http://youtu.be/p8bqf3uMrG8 :good:

Hitman Radeon 7970
http://youtu.be/kiouVs1BpMM :shock:

I'm not rallying against AMD here - in fact, I'm pointing out that there's only ONE game where the 7970 really "sucked" at in ABT's article. There were also cases where GTX 680 performed worse, like in Resident Evil 5 and WiC:SA.

Hopefully ABT will test the rest of the ~30-game suite.. at two resolutions with frame time measurements. That would be kick-ass, and good for the gaming community to have (because fps is not the only metric that matters).

 

[Elfear]

Hmm, that's interesting - I always saw ABT articles popping up pretty high with some searches..

Like if I type this in google:
hd 7970 frame time

ABT's article comes up at #5, after 4 links to techreport articles.

:awe:

Honestly, what was wrong with ABT's article?

It didn't make HD 7970 look all that terrible - just that it performed abysmally in Hitman: Absolution - which can be clearly shown with youtube videos of this and GTX 680.

Hitman GTX 680
http://youtu.be/p8bqf3uMrG8 :good:

Hitman Radeon 7970
http://youtu.be/kiouVs1BpMM :shock:

I'm not rallying against AMD here - in fact, I'm pointing out that there's only ONE game where the 7970 really "sucked" at in ABT's article. There were also cases where GTX 680 performed worse, like in Resident Evil 5 and WiC:SA.

Hopefully ABT will test the rest of the ~30-game suite.. at two resolutions with frame time measurements. That would be kick-ass, and good for the gaming community to have (because fps is not the only metric that matters).

I don't think the issue was with that particular article. More just the general attitude at ABT and how trustworthy the reviews are.

 

[Lonbjerg]

I don't think the issue was with that particular article. More just the general attitude at ABT and how trustworthy the reviews are.

Have you read any threads here...a lot of people have issues with microsttuer:

 

[Elfear]

Have you read any threads here...a lot of people have issues with microsttuer:

Again, I don't think the people posting about ABT in this thread were specifically calling out the ms article. Just seeing the article was from ABT was what some posters questioned. In a similar vein as seeing an article from Geforce.com or AMDZone.com.

 

[3DVagabond]

Like I asked earlier, who do we blind test? The general public? Gamers? Enthusiasts? The guys that claim they can see it?

I'd argue that your average person won't see it, because they're ignorant of what they're looking for.

I would blind test the people who claim they see it first.

 

[VulgarDisplay]

I would blind test the people who claim they see it first.

If done right it really wouldn't matter who was tested. They need to come up with an animation of something like pong. Simple, objects moving across the screen, and have a way to increase/control frame latencies on this simulation.

Just tell the observer to watch the movement and once they find the movement to be no longer be perfectly smooth have them click a button or something to record the exact amount of frame latency that has been introduced.

It would actually probably be better to test people that have no idea what frame latency is whatsoever.

 

[Rikard]

If done right it really wouldn't matter who was tested. They need to come up with an animation of something like pong. Simple, objects moving across the screen, and have a way to increase/control frame latencies on this simulation.

Just tell the observer to watch the movement and once they find the movement to be no longer be perfectly smooth have them click a button or something to record the exact amount of frame latency that has been introduced.

It would actually probably be better to test people that have no idea what frame latency is whatsoever.

We need all types of people. Those who claim that they are ultra sensitive, average gamers, and average Joe as control group. It is also important that the degree of stuttering is not gradually changing but come in random order. Otherwise it is too easy to guess right.

 

[Rikard]

Exactly this. By no one we probably mean 99.999%. The data is likely to come out as a bell curve of some description but with some odd charateristics (huge spike around 17ms if its not more noticeable to most below that). Because for a product that sells millions to only have 10's to 100's of people see a problem is probably OK. More than that and you have a problem.

Isn't the typical RMA rate at percent level for computer components already? It is hard to claim that it is serious if it is much smaller than that.

 

[Lonbjerg]

I would blind test the people who claim they see it first.

Yeah, because no way we could see the fluctuations that the data shows.

Funnny thing:

This was first reported by people...before any data showed what was up.

This is really low damage control :|

 

[3DVagabond]

Yeah, because no way we could see the fluctuations that the data shows.

Funnny thing:

This was first reported by people...before any data showed what was up.

This is really low damage control :|

The question was asked, "Who would you test?" It makes sense to test people who say that they can see stuttering on cards with these recorded latency numbers. It would make no sense to test people who say they can't see a difference. Joe Blow off of the street might not have a clue what he/she is looking for. Who would you suggest we test?

Do you have a problem with sitting people down at gaming rigs with 7950's and 660ti's who say that they can see a difference and compare the 2 for smoothness without knowing which card they are using?

Why don't you explain where my logic is faulty, rather than making accusations of my motives?

 

[Lonbjerg]

The question was asked, "Who would you test?" It makes sense to test people who say that they can see stuttering on cards with these recorded latency numbers. It would make no sense to test people who say they can't see a difference. Joe Blow off of the street might not have a clue what he/she is looking for. Who would you suggest we test?

Do you have a problem with sitting people down at gaming rigs with 7950's and 660ti's who say that they can see a difference and compare the 2 for smoothness without knowing which card they are using?

Why don't you explain where my logic is faulty, rather than making accusations of my motives?

Your sentecne about people "claiming" to see it was loaded.

I see it...it's not a claim, it's a fact.

 

[3DVagabond]

Your sentecne about people "claiming" to see it was loaded.

I see it...it's not a claim, it's a fact.

The whole point is to verify that it's visible during normal gameplay. Until it's verified it's not a fact.

 

[Lonbjerg]

The whole point is to verify that it's visible during normal gameplay. Until it's verified it's not a fact.

Still in denail eh...

 

[3DVagabond]

Still in denail eh...

I'm not denying anything. I don't know what you're talking about.

Here's the scenario for you. If you did a double blind test like I described and can tell a difference then it proves there's a visible flaw in the reproduction. No problem, as I never said there wasn't one. Just that I'd like it tested under real world conditions without any chance of bias (which as humans we all have) dictating the outcome. Of course, if you couldn't tell which card was which with any statistical significance, then you are wrong. You are the only one between us who has predicted an outcome. It's no wonder you don't like the idea.

 

[Keysplayr]

I'm not denying anything. I don't know what you're talking about.

Here's the scenario for you. If you did a double blind test like I described and can tell a difference then it proves there's a visible flaw in the reproduction. No problem, as I never said there wasn't one. Just that I'd like it tested under real world conditions without any chance of bias (which as humans we all have) dictating the outcome. Of course, if you couldn't tell which card was which with any statistical significance, then you are wrong. You are the only one between us who has predicted an outcome. It's no wonder you don't like the idea.

So if the people used in the double blind test are people who cannot for whatever reason perceive microstutter or hitchy performance, than it doesn't exist?

You keep saying: "I never said there wasn't one."
But you won't say: "There is one"

 

[3DVagabond]

So if the people used in the double blind test are people who cannot for whatever reason perceive microstutter or hitchy performance, than it doesn't exist?

You keep saying: "I never said there wasn't one."
But you won't say: "There is one"

I'm saying to use people who say they can see it. Use more than one person. The more the better. It would be dumb to use people who say they can't see it.

As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other? Just benchmarks don't visually demonstrate anything. Neither does videos in slo-mo. Real world testing will demonstrate if it can be seen in regular gameplay. Testing with multiple people, multiple games, different settings, etc... The more thoroughly it's tested the better we can understand it.

 

[Keysplayr]

I'm saying to use people who say they can see it. Use more than one person. The more the better. It would be dumb to use people who say they can't see it.

As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other? Just benchmarks don't visually demonstrate anything. Neither does videos in slo-mo. Real world testing will demonstrate if it can be seen in regular gameplay. Testing with multiple people, multiple games, different settings, etc... The more thoroughly it's tested the better we can understand it.

Slo Mo is actually very good in seeing if you are getting what you expect. The Slo Mo was done by Scott Wasson, presumably because he got pummeled with complaints from those who could not see the stutter in real time. He did them a favor, although some didn't think so. Smooth gameplay and a real representation of the fps you are said to be getting in any given game. If you are told that you'll get a peak framerate in Hitman Absolution or Skyrim or Crysis2 and you buy a card on that premise (after looking at pure fps benches and reviews), but what you really get is a really uneven delivery of frames, then you aren't really getting what you paid for.

Lastly: "As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other?"
If AMD acknowledged the problem, I don't see any reason for you to deny the problem exists. AMD thinks it has been shown to be a problem. It's up to you now. Continue the pointless battle or appreciate that AMD is working to rectify the problem. Your call.

 

[3DVagabond]

Slo Mo is actually very good in seeing if you are getting what you expect. The Slo Mo was done by Scott Wasson, presumably because he got pummeled with complaints from those who could not see the stutter in real time. He did them a favor, although some didn't think so. Smooth gameplay and a real representation of the fps you are said to be getting in any given game. If you are told that you'll get a peak framerate in Hitman Absolution or Skyrim or Crysis2 and you buy a card on that premise (after looking at pure fps benches and reviews), but what you really get is a really uneven delivery of frames, then you aren't really getting what you paid for.

Lastly: "As far as me saying there is a visible problem, how can I if I don't think it's been shown one way or the other?"
If AMD acknowledged the problem, I don't see any reason for you to deny the problem exists. AMD thinks it has been shown to be a problem. It's up to you now. Continue the pointless battle or appreciate that AMD is working to rectify the problem. Your call.

If it has to be shown in slo-mo to be seen then what does it matter?

Nobody, not me, AMD, or anyone else is saying that the measured frame latencies don't exist. Of course AMD is going to be concerned. It's a benchmark that shows their card in a negative light compared to the competition. Please link to AMD saying there's a visible stuttering problem. They have said they are addressing the frame latencies, not stuttering. At least that's all I have seen. Maybe I missed it?

 

[Keysplayr]

If it has to be shown in slo-mo to be seen then what does it matter?

It doesn't have to be. It was done for those that could not perceive it or at least were arguing that it couldn't be perceived. It was done for people like you.

"Nobody, not me, AMD, or anyone else is saying that the measured frame latencies don't exist."

And AGAIN, you won't actually say it does. What is the problem here?


Of course AMD is going to be concerned. It's a benchmark that shows their card in a negative light compared to the competition. Please link to AMD saying there's a visible stuttering problem. They have said they are addressing the frame latencies, not stuttering. At least that's all I have seen. Maybe I missed it?

Look, I can see why you chose to take this line in the debate. If done well, it would provide a certain level of deniability of the issue if it can't be seen. Unfortunatly for you, you're not doing it well.
You've just claimed that because AMD did not claim to see it, it isn't there?
Your position is quite weak my friend. I don't know why you continue with this line. All I know is that you have been one busy beaver the last week.