[TECH Report] As the second turns: the web digests our game testing methods

Page 29 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

KingFatty

Diamond Member
Dec 29, 2010
3,034
1
81
See Zenon and his Grain of Millet paradox. There is no telling exactly how many grains is needed before you can hear them producing noise when hitting the ground.

Here are the solutions to the Grain of Millet "paradox". It's a nice analogy because just as our ears have a finite limit of hearing, I think our visual system has a finite limit to where you see the effect of low framerate, but get that framerate high enough and bam it's like butter/reality.

Actually, I wonder what the equivalent frames per second our eyes are capable of perceiving the real world?

Here are the two solutions to the paradox, and the second solution is more analogous I think because it relates to human perception of sound, instead of attacking the division fallacies of the paradox:

The Standard Solution to this interpretation of the paradox accuses Zeno of mistakenly assuming that there is no lower bound on the size of something that can make a sound. There is no problem, we now say, with parts having very different properties from the wholes that they constitute. The iterative rule is initially plausible but ultimately not trustworthy, and Zeno is committing both the fallacy of division and the fallacy of composition.

Some analysts interpret Zeno’s paradox a second way, as challenging our trust in our sense of hearing, as follows. When a bushel of millet grains crashes to the floor, it makes a sound. The bushel is composed of individual grains, so they, too, make an audible sound. But if you drop an individual millet grain or a small part of one or an even smaller part, then eventually your hearing detects no sound, even though there is one. Therefore, you cannot trust your sense of hearing.

This reasoning about our not detecting low amplitude sounds is similar to making the mistake of arguing that you cannot trust your thermometer because there are some ranges of temperature that it is not sensitive to. So, on this second interpretation, the paradox is also easy to solve. One reason given in the literature for believing that this second interpretation is not the one that Zeno had in mind is that Aristotle’s criticism given below applies to the first interpretation and not the second, and it is unlikely that Aristotle would have misinterpreted the paradox.

Taken from here:
http://www.iep.utm.edu/zeno-par/#SSH3ci
See section 3(c)(i)
 

f1sherman

Platinum Member
Apr 5, 2011
2,243
1
0
Sure you could do blind testing, but does anyone remember all the blind testing that was done for the industry to agree on 30 fps being the crossover between acceptable and unacceptable? I don't, it just happened. It's just universally agreed on for the most part (I think, maybe now it's bumped up with fancier cards?).

Exactly. More knowledge will come with more games and more data on them.
Not with blind-testing and extrapolating from there. That is how I see it.

Anyway, I'm just saying that we are all exploring the frontier right here, it's like a game changing thing right now, we are seeing the birth of a new metric to rate video cards. It's happening now, and that's kind of cool. There will be a way to easily quantify it, and you'll see cards rated like that (X frames per second, Y stutter).

Z total time in danger zone (>50ms) :)

It's not exactly straight forward to quantify new data with well developed tools of statistics. Now imagine trying to extract any general conclusions by varying this data across the range of test persons.
Too much effort for almost nothing. Why nothing? Because error/uncertainty will dwarf the result itself.

So before this ever happening, I can more easily see giant farms streaming GPU simulations right on my desktop :sneaky:
 

Jaydip

Diamond Member
Mar 29, 2010
3,691
21
81
What if the Swiss-made micrometer answers a different question than the stick? We are not talking or replacing one with the other mind you.


Wouldn't that be good progress? If a customer can understand what is a meaningful difference, and what is not, he/she will be able to make a better informed decision. As a result hardware manufacturers will not spend money on useless gimmicks but on actual improvements that matter to people. Nowadays we have almost universally agreed that playable is 30 FPS and good is 60 FPS, but we are not more stupid than we realize that this depends on person and situational conditions. My hope is that we will get to that point also for microstutter, tearing, artifacts and other graphical imperfections. Why are you so against it?

Why? You had the chance to deliver your ground breaking alternative method, but for some reason you avoided presenting it. You have a still the chance to do it you know...

I didn't because repeating the same thing over and over gain is tiring.Lets see we have performance monitoring tools for a long time now. PerfHUD allowed any NV cards from 8xxx to Fermi series to be monitored in real time.For Kepler we have Nsight for that.You can trace a single frame delivey time across its entire life cycle.There is a catch though, during production we essentially remove all the performance hooks to make the application run faster.To make use of these tools you need to have access to the codebase. This is applicable for all D3D apps not just for games.It probably works for OpenGL too but I never tried that.
 

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
Browsing this thread and found it odd that I'm being called a liar.

I saw the stutter. I went through the work to pull a latency log and from that data I was able to see the stutter.

I got a second person who saw the stutter.

But I guess we're just lying because 3DVagabond need someone else (possibly himself) to verify it for it to be factual?

Not even sure what you're trying to defend at this point but, keep up the good work, :rolleyes:


Nobody is calling you a liar. If somehow you are offended by what I've suggested, you want to have your feelings hurt, and that's not my fault. I am not trying to personally attack anyone.

You forgot one important aspect with your study, the blind testing. I've been involved with blind listening tests for audio equipment. It's really revealing how much difference people can perceive when they know which equipment they are listening to. When they then have to reproduce those perceptions and listening experiences when they don't know whether they are listening to amplifier A or amplifier B though, they can't. I'm not saying that with similar testing that you wouldn't be able still accurately assess which is "smoother". All I'm saying is that I've seen people who were absolutely convinced that amplifier A was far superior to amplifier B using all kinds of superlatives to describe the immensely superior musicality and then when they didn't know which one they were listening to couldn't identify one from the other with any more accuracy than if they flipped a coin. These people were not liars. These people were sincere that these differences existed.

Just in case there are "audiophiles" amongst us (I consider myself one) I'm not saying all amplifiers are equal or even sound alike. With a revealing enough speaker in a well designed listening environment differences can be heard and identified. But assuming two well engineered amplifiers of basically similar design (Not a 35watt tube amp compared to a Krell reference amplifier driving a fatally low impedance load) they sound very very similar.
 

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
I'm thinking along the lines of how this would be exploring characteristics of people.

I mean, most gamers would agree that falling below 30 fps is going to affect your enjoyment.

We just need to know how to, ah, come up with a way to quantify stutter. Once you do that, I think gamers would come to some unwritten agreement about what is acceptable and what is not.

Sure you could do blind testing, but does anyone remember all the blind testing that was done for the industry to agree on 30 fps being the crossover between acceptable and unacceptable? I don't, it just happened. It's just universally agreed on for the most part (I think, maybe now it's bumped up with fancier cards?).

Anyway, I'm just saying that we are all exploring the frontier right here, it's like a game changing thing right now, we are seeing the birth of a new metric to rate video cards. It's happening now, and that's kind of cool. There will be a way to easily quantify it, and you'll see cards rated like that (X frames per second, Y stutter).

The motion picture industry and animation professionals would have had a guideline on at what point FPS was smooth long before video cards or games were even a twinkle in their designer's eyes.
 

Final8ty

Golden Member
Jun 13, 2007
1,172
13
81
Well good luck to you.
Good luck with finding suitable control group,
GL with conducting controlled experiment with even semi-reproducible results(sorry you kinda need that :D),
and from there developing theory and algorithms that will pass 1/10 of scrutiny that TR has been subjugated to.

I am sure you'd want your findings to be reusable later, without the need to gather your control group again, amirite?
Then good luck with finding algorithm and values that will hold it's ground tested against different games, with different screen dynamics, across range of different gfx cards, FPS, details and resolutions.

Its not like its a medically critically study, so getting all anal about it is not necessary just like THD and other specs is not critical for the avg person, most don't even look at it, because they just don't notice the deference between a certain spectrum of values, it has to get very wide for them to notice.
"Oh but look this one has 002% and this 005%" don't worry about it you will not notice the difference, but its is a measurable statistic so it gets measures because the sum of all the parts in the chain from the source to the output is what counts and including the human factor.
http://www.bcae1.com/thd.htm

Just need to know out of 100 people at what level of Micro stuttering did the majority notice.
No need to get all flippant with good luck with that.

But its clear to me now that some people are not really interested in anything than the measurable statistic because they can slap im better on there favourite brand regardless if it noticeable or not and change what statistic matters most when there brand is better at it statistically and not also where it makes a real difference, the past is testament to that fact.

I'm done with this until i see new info.
 
Last edited:

Final8ty

Golden Member
Jun 13, 2007
1,172
13
81
The motion picture industry and animation professionals would have had a guideline on at what point FPS was smooth long before video cards or games were even a twinkle in their designer's eyes.

24 FPS was used because it was the minimum they could get away with because it was all about the cost of Film and Storage.
 

f1sherman

Platinum Member
Apr 5, 2011
2,243
1
0
Just need to know out of 100 people at what level of Micro stuttering did the majority notice.
No need to get all flippant with good luck with that.

But its clear to me now that some people are not really interested in anything than the measurable statistic because they can slap im better on there favourite brand regardless if it noticeable or not and change what statistic matters most when there brand is better at it statistically and not also where it makes a real difference, the past is testament to that fact.

Well you are the only one at this point mentioning and contemplating favorite b_____ :colbert:

By saying even before work has begun: "no need to be anal about this and that..."
By saying "Just need to know out of 100 people at what level of Micro stuttering did the majority notice"

therefore assuming that "level of Micro stuttering" is some handily controlled variable which is quite easy to isolate by test subjects,
and is NOT easily confused with frame-skipping, low fps and input lag...

tells me that the value of such study would be just that: To find out "at what level of Micro stuttering did the majority notice"
Nothing more, and perhaps somewhat less :D

That is, if you get around to define control variable you call "Micro stuttering".
I mean your suggestion is not even a sketch of a sketch.
Your key variable - "Micro stuttering"? We don't even know what exactly we are talking about here.

Sorry if I sound condescending. I tried to be funny, but it didn't work :D
nhf is what I;m saying
 
Last edited:

VulgarDisplay

Diamond Member
Apr 3, 2009
6,188
2
76
I fail to understand why people are so against trying to quantify the level of frame latency that is acceptable. This number needs to be found not to absolve AMD of their issues, but to find out where BOTH manufacturers need their hardware to run for the best possible experience.

Perhaps doing a test with zero frame latency and ramping up to levels similar to what nvidia enjoys right now could actually prove that even the amount they have is unacceptable and noticeable to the average viewer, or the testing could find that even AMD's current worst case frame latency is completely undetectable to the unsuspecting and unbiased viewer.

There is only one reason for being against finding that acceptable worst case scenario, and that is through blind loyalty to the manufacturer that already has the apparent lead in this benchmark. By outright trashing a way for reviewers to give their viewers a very meaningful and and 100% necessary metric to compare their frametime graphs against you destroy your own credibility as an unbiased participant on these forums.

This is a neat graph:
gw2.gif

Now imagine how useful it would be if there was a line drawn across it at (just tossing a number out) 20ms and anything above that line was a detectable stutter. In that case both manufacturers would have work to do. Why do we not want to know what the magic number is?

Now I'm sure many of you will say, "But it's not 20ms because we can't detect it on our Nvidia cards." Well, tell me what the number is then, so I can actually look at the graph and be able to draw a conclusion.

There was no burden on the reviewer to validate an FPS metric because it is a number that is entirely separate from the human element. Perceived smoothness requires a human observer, which is precisely why they need to blind test a group of human observers to find the worst case undetectable level of frame latency to let anyone reading their review understand the graph.
 

Black Octagon

Golden Member
Dec 10, 2012
1,410
2
81
I fail to understand why people are so against trying to quantify the level of frame latency that is acceptable. This number needs to be found not to absolve AMD of their issues, but to find out where BOTH manufacturers need their hardware to run for the best possible experience.

+1

Or, as I see it, the point should be to generate more and better data to help us understand specifically what is causing frame latency and work towards minimising (if not totally eliminating it) as much as possible in the future. Even if AMD/NVIDIA succeed in reducing it to a level at which 'the average user' no longer perceives an unacceptable reduction in smoothness, that would not be a reason to suddenly halt all efforts. Smoothness is not a binary thing. Even something we judge to be 'smooth' can be made smoother, and that is something to aim for in and of itself.
 

Final8ty

Golden Member
Jun 13, 2007
1,172
13
81
I fail to understand why people are so against trying to quantify the level of frame latency that is acceptable. This number needs to be found not to absolve AMD of their issues, but to find out where BOTH manufacturers need their hardware to run for the best possible experience.

Perhaps doing a test with zero frame latency and ramping up to levels similar to what nvidia enjoys right now could actually prove that even the amount they have is unacceptable and noticeable to the average viewer, or the testing could find that even AMD's current worst case frame latency is completely undetectable to the unsuspecting and unbiased viewer.

There is only one reason for being against finding that acceptable worst case scenario, and that is through blind loyalty to the manufacturer that already has the apparent lead in this benchmark. By outright trashing a way for reviewers to give their viewers a very meaningful and and 100% necessary metric to compare their frametime graphs against you destroy your own credibility as an unbiased participant on these forums.

This is a neat graph:
gw2.gif

Now imagine how useful it would be if there was a line drawn across it at (just tossing a number out) 20ms and anything above that line was a detectable stutter. In that case both manufacturers would have work to do. Why do we not want to know what the magic number is?

Now I'm sure many of you will say, "But it's not 20ms because we can't detect it on our Nvidia cards." Well, tell me what the number is then, so I can actually look at the graph and be able to draw a conclusion.

There was no burden on the reviewer to validate an FPS metric because it is a number that is entirely separate from the human element. Perceived smoothness requires a human observer, which is precisely why they need to blind test a group of human observers to find the worst case undetectable level of frame latency to let anyone reading their review understand the graph.

+1

Or, as I see it, the point should be to generate more and better data to help us understand specifically what is causing frame latency and work towards minimising (if not totally eliminating it) as much as possible in the future. Even if AMD/NVIDIA succeed in reducing it to a level at which 'the average user' no longer perceives an unacceptable reduction in smoothness, that would not be a reason to suddenly halt all efforts. Smoothness is not a binary thing. Even something we judge to be 'smooth' can be made smoother, and that is something to aim for in and of itself.
Well put both of you.
 

Whitestar127

Senior member
Dec 2, 2011
397
24
81
Now imagine how useful it would be if there was a line drawn across it at (just tossing a number out) 20ms and anything above that line was a detectable stutter. In that case both manufacturers would have work to do. Why do we not want to know what the magic number is?

Now I'm sure many of you will say, "But it's not 20ms because we can't detect it on our Nvidia cards." Well, tell me what the number is then, so I can actually look at the graph and be able to draw a conclusion.

There was no burden on the reviewer to validate an FPS metric because it is a number that is entirely separate from the human element. Perceived smoothness requires a human observer, which is precisely why they need to blind test a group of human observers to find the worst case undetectable level of frame latency to let anyone reading their review understand the graph.

This all sounds sensible to me. Finding that sweet/safe spot is what this should all be about.

On a side note: Yesterday I just had to just check that I actually CAN see ms. :) I played Skyrim on my rig (Crossfire mind you, not single-GPU). I stood outside Whiterun and side-strafed. My FRAPS reported 73-75 fps, which should be *silky* smooth on my CRT. But sure enough, it actually looked more like say 45-50. It's very easy to see that it's not silky smooth on a CRT. I could actually also see the unevenness (is that a word? It is now :)) of the motion/frame-distribution.
Funny thing is, if I just turned around and looked somewhere else then suddenly fps was 85 (with v-sync) and movement was silky smooth. It's very arbitrary.

So was it very distracting? Honestly, no. Not to me. But it's there none the less.
 

Rikard

Senior member
Apr 25, 2012
428
0
0
Finding that sweet/safe spot is what this should all be about.
I agree. Just like with everything else in VC&G (FPS, AA etc). But for some reason I cannot grasp there are posters here that prefer ignorance when they are offered knowledge.

Anyway, previously in this thread I posted some graphs together with my personal experience from them. Those cannot be taken as general guidelines since we would need to test more people to see if they agree. But anyway I reiterate:
  • t= 8.8+-1.5 ms -> No visible stuttering. Probably because even with variation the frametimes are well below 16 ms. (High FPS saves the day)
  • t=16.7+-0.8 ms -> No visible stuttering. Probably because variation is not large enough (Too small MS)
  • t=15.9+-2.9 ms -> Acceptable microstuttering. A noticeable fraction of frames are rather slow due to the wider spread than in the previous case, but it is small enough to not be disturbing.
  • t=18.6+-4.4 ms -> Unacceptable microstuttering. Average frame time is large in combination with a very large spread in frametimes, giving a large fraction of frames with large frame times. At this level it is gives a disturbing effect.
One reason I have not pushed that further is that they involve different games, and different light levels, different movement patterns etc can interfere with the conclusions. I would rather use BrightCandles simulation where conditions can be reproduced systematically before taking this to the next level. But if reviewers could post measurements like this (or something even better) I could buy my next card based on my own, subjective, sweet spot.
 

Keysplayr

Elite Member
Jan 16, 2003
21,219
55
91
I agree. Just like with everything else in VC&G (FPS, AA etc). But for some reason I cannot grasp there are posters here that prefer ignorance when they are offered knowledge.

Anyway, previously in this thread I posted some graphs together with my personal experience from them. Those cannot be taken as general guidelines since we would need to test more people to see if they agree. But anyway I reiterate:
  • t= 8.8+-1.5 ms -> No visible stuttering. Probably because even with variation the frametimes are well below 16 ms. (High FPS saves the day)
  • t=16.7+-0.8 ms -> No visible stuttering. Probably because variation is not large enough (Too small MS)
  • t=15.9+-2.9 ms -> Acceptable microstuttering. A noticeable fraction of frames are rather slow due to the wider spread than in the previous case, but it is small enough to not be disturbing.
  • t=18.6+-4.4 ms -> Unacceptable microstuttering. Average frame time is large in combination with a very large spread in frametimes, giving a large fraction of frames with large frame times. At this level it is gives a disturbing effect.
One reason I have not pushed that further is that they involve different games, and different light levels, different movement patterns etc can interfere with the conclusions. I would rather use BrightCandles simulation where conditions can be reproduced systematically before taking this to the next level. But if reviewers could post measurements like this (or something even better) I could buy my next card based on my own, subjective, sweet spot.

Knowledge? What is the definitive "answer" you are looking for in these blind tests? When the said tests are all done and gone, what is the final answer you wish to have? More importantly, what is the question you want answered? I've read the thread and yes I have seen the reasons given for the testing, but what will it mean after the tests?

1. If the majority cannot see or detect frame latency (hitching, roughness) than is isn't a concern for gamers.
2. If the majority can see or detect frame latency (hitching, roughness) than don't worry, it will be fixed soon with newer drivers.

Look, we know how this forum works and I'm anticipating why this test is so vehemently requested. Because in all actuality, it ISNT scientific. No two persons eyes, or brain, is alike. Some will see it, some will not as is evident by most AMD users here. They can't see a thing. :D
Seriously though, TR, ABT, PCPER, all have the right idea. Scientifically reproducible results.
 
Last edited:

Leadbox

Senior member
Oct 25, 2010
744
63
91
Knowledge? What is the definitive "answer" you are looking for in these blind tests? When the said tests are all done and gone, what is the final answer you wish to have? More importantly, what is the question you want answered? I've read the thread and yes I have seen the reasons given for the testing, but what will it mean after the tests?

1. If the majority cannot see or detect frame latency (hitching, roughness) than is isn't a concern for gamers.
2. If the majority can see or detect frame latency (hitching, roughness) than don't worry, it will be fixed soon with newer drivers.

Look, we know how this forum works and I'm anticipating why this test is so vehemently requested. Because in all actuality, it ISNT scientific. No two persons eyes, or brain, is alike. Some will see it, some will not as is evident by most AMD users here. They can't see a thing. :D
Seriously though, TR, ABT, PCPER, all have the right idea. Scientifically reproducible results.

It would be a bit silly to blindtest just 2 people:colbert:
They're never going to be perfectly smooth or be as smooth as each other but they can be just as playable so some sort of cut-off point needs to be defined
 

Keysplayr

Elite Member
Jan 16, 2003
21,219
55
91
It would be a bit silly to blindtest just 2 people:colbert:
They're never going to be perfectly smooth or be as smooth as each other but they can be just as playable so some sort of cut-off point needs to be defined

erm.. I didn't say 2 people. I don't even think blind test is relevant. Too many variables are present.
 

Leadbox

Senior member
Oct 25, 2010
744
63
91
erm.. I didn't say 2 people. I don't even think blind test is relevant. Too many variables are present.

If the blindtest group is big enough these variables will sort themselves out
There is a point below which A being smoother than B wont matter, it would be nice to see it plotted on that frametime gragh don't you think?
 

Rikard

Senior member
Apr 25, 2012
428
0
0
When the said tests are all done and gone, what is the final answer you wish to have?
Given condition A, X people will notice the effect, out of which Y people perceive it as having a negative impact on their gameplay experience.
If that can be repeated in groups like pro players it would be interesting to see the differences in the results IMHO.
Look, we know how this forum works and I'm anticipating why this test is so vehemently requested.
Easy on the paranoia bro. I am not one of those guys. I am genuinely interested in this phenomenon.
Because in all actuality, it ISNT scientific. No two persons eyes, or brain, is alike. Some will see it, some will not
I never said everybody was the same. If I thought so, I would just have said, nope the problem does not exist in Skyrim. But I don't because I acknowledge that we are all different. But just because our brains are not alike does not make it unscientific. As an example, if you measure length of grown men you might find that the average is 175 cm (made up number). This has a standard deviation of 20 cm. There are correlations such that Dutch men are 10 cm longer than Americans. Just because we have these variations does not mean that we must give up. It still tells us a lot. Without any measurement (as if you had never before seen a human) you would not know if me we 10 cm tall or 10 m tall on average.
 

Imouto

Golden Member
Jul 6, 2011
1,241
2
81
@Rikard

Anything below 33.3 ms should be ditched for 60Hz displays. If your game is running at anything behind 60 fps you'll get a duped frame every second leading into a 33.3 ms frame even if it's not shown in any graph.
 

HurleyBird

Platinum Member
Apr 22, 2003
2,812
1,550
136
erm.. I didn't say 2 people. I don't even think blind test is relevant. Too many variables are present.

Indeed. I'd be willing to bet most people on this forum would fail a proper double blind ABX test between 50 FPS and 60 FPS. That doesn't mean that 60 FPS isn't better and smoother, or that the difference can't be felt on some level even by those who are unable to determine the 60 FPS sample in a statistically relevant manner.

If the blind testing crowd was logically consistent they would swear off benchmarking entirely in favor of controlled double blind ABX testing. Not likely though.
 

Rikard

Senior member
Apr 25, 2012
428
0
0
@Rikard

Anything below 33.3 ms should be ditched for 60Hz displays. If your game is running at anything behind 60 fps you'll get a duped frame every second leading into a 33.3 ms frame even if it's not shown in any graph.
Sorry can you explain more please? If my screen frequency is 60 Hz it refreshes every 16.7 ms. How would 59 average FPS lead to one 33.3 ms frame per second? Why would it not be included in the graphs?
 

Imouto

Golden Member
Jul 6, 2011
1,241
2
81
Sorry can you explain more please? If my screen frequency is 60 Hz it refreshes every 16.7 ms. How would 59 average FPS lead to one 33.3 ms frame per second? Why would it not be included in the graphs?

Maybe cuz at 59 fps you leave 1 refresh empty every second repeating the previous frame hence lasting 33.3 ms.

It won't be shown in the graphs cuz it's totally monitor side and something that only a human (or high speed camera) can see.
 

Arzachel

Senior member
Apr 7, 2011
903
76
91
Indeed. I'd be willing to bet most people on this forum would fail a proper double blind ABX test between 50 FPS and 60 FPS. That doesn't mean that 60 FPS isn't better and smoother, or that the difference can't be felt on some level even by those who are unable to determine the 60 FPS sample in a statistically relevant manner.

If the blind testing crowd was logically consistent they would swear off benchmarking entirely in favor of controlled double blind ABX testing. Not likely though.

Tell me, if for example I couldn't tell the difference between 50 and 60 FPS in a double blind test, tell me why would 60 fps for me be "better" and "smoother". Because the number's higher? That's the whole point of double blind testing, if you could on some level feel the difference, you would be able to discern it, even if subconsciously. That's what statistical significance is in very simple terms - separating random guesses from actual impact.

Look, we know how this forum works and I'm anticipating why this test is so vehemently requested. Because in all actuality, it ISNT scientific. No two persons eyes, or brain, is alike. Some will see it, some will not as is evident by most AMD users here. They can't see a thing. :D
Seriously though, TR, ABT, PCPER, all have the right idea. Scientifically reproducible results.

So double blind testing isn't scientific now? Wanna buy some magical Nvidia case stickers that make frames more tight and FXAA more crisp? :p What double blind testing would do is establish a common baseline for both manufacturers to strive for. Just like 60 fps, it wouldn't be a be all end all figure, but an useful standard nevertheless. On a more personal note, a different double blind test could be used to find out your frame latency perception limits, which would allow you to get more useful info out of benchmarks.
 

KingFatty

Diamond Member
Dec 29, 2010
3,034
1
81
1. If the majority cannot see or detect frame latency (hitching, roughness) than is isn't a concern for gamers.
2. If the majority can see or detect frame latency (hitching, roughness) than don't worry, it will be fixed soon with newer drivers.

Also a third option:
3. Where is the "danger zone" for hitching roughness, where it starts to become noticeable and then annoying for most gamers?

I think we are generally agreed about average frames per second, we know there can be a danger zone there for most gamers.

But it's still being ironed out where most gamers feel about what level of hitching roughness is acceptable. So maybe it's an exploration into the minds of gamers, see what gamers are OK with.

You want your video card to definitely be above 30 FPS, and maybe you'd be comfortable at 45 FPS but prefer 60 FPS.

For hitching roughness, let's do the same thing. You want your video card to definitely be less than +-20% hitching, but would be OK with +-10% etc. (or whatever metric will be used to quantify smoothness).

When review sites start using smoothness as a metric, whatever they call it, then people who read reviews will use that to buy a card.

If one card gets 50 FPS and 20% hitching, maybe I'd prefer to get the card that is 45 FPS and 5% hitching?