[TECH Report] As the second turns: the web digests our game testing methods

[BrightCandle]

Stop there.

Now you're not saying that you have a super-vision and you can tell apart differences at 6 ms level.

You're saying that you can tell apart differences in frame times that your monitor can't even show.

I told you about the emperors clothes back in page 7 but looks like we didn't move an inch in 23 pages.

Talk about brand suggestion.

Thanks for that, it brought tears to my eyes I laughed so hard. Its so hard to believe this is anything but brilliant sarcasm, bravo sir.

 

[ocre]

why frame times? Because every person who has sight can process information faster than second hand turns on a clock. We dont see in averages across a second=fps. It is really that simple.

So measuring frame times gives us information on every single frame that is generated and the time it takes between them. In real life, real motion we dont have any breaks at all. The information is coming as fast as it can be processed......faster even. With animation we are trying to give the appearance of true life like motion. It is possible to have an illusion of lifelike motion in animation but in order to do so each picture must be projected in steady increments with animation linked to each frames time. This is how it is done and why with GPUs frames need to be consistent across the entire second. An average of the second is pretty useless in determining smoothness. It is incapable because we see faster than in seconds.

Think of a race car going 80 miles per hour exactly, with perfect accuracy from perfect autocruise. The track is exactly 80 miles in a loop so in one hour the car travels 80 miles and is back at the flag completing one lap. Now imagine another car which is on the track at exact same moment at the start of the hour. This second car is trying to record the experience of the 1st car and driver for a documentary but is having issues keeping it at exactly 80mph. So half the time the 2nd car is running 120mph and the other half it downs it to 40mph. Over the time span of 1 hr both cars meet at the same flag pole and start another lap. Although they both get to the flag at the same moment, watching from above you will notice both are very different. One is more erratic speeding up and slowing down while the other is much more steady. Any observer would notice and wonder what is going on with the 2nd car. It is unusual and will get noticed.

Needless to say, the 2nd car does a poor job in recording the event it was intended to. Instead of capturing the 1st car and driver just as if someone was right beside them, you get a video that races past the subject and then watches the subject race back zooming past the camera. This happens over and over. This is like the animation which cannot remain smooth if things speed up and slow down.

 

[3DVagabond]

Hurt my feelings? Really? Interesting you think you could hurt my feelings. Haha. Actually rather amusing, but let's get to point of my comment since you didn't even take the time to answer.

When you were asked who would you blind test, your response was:



When asked why your response was:



What isn't a fact? That people saw it during gameplay? So, are you implying they are lying? The content of these two posts has absolutely nothing to do with a comparative of which is smoother. It is a out right questioning those that saw it.

Clearly they are lying until you can verify it. By your own wording.

Perhaps you got hurt feelings, I dunno, not even sure why you are questioning the validity of those who have seen it. Unless of course you are trying to claim what we don't see anything.

And because I'm bored:



What study? I gave the perspective of two individuals. We saw it. You claim we didn't until it can be verified - by whom, well I dunno since you're the one calling our experiences into question.



Interesting, you clearly point out the faults of blind testing yet continually push for blind testing.

So if a person percieves the stutter at 17MS and a person doesn't at 34MS, where are we? Or do we aim for the majority who can not see it at 24MS? Sucks for that guy with the 17MS hawk eyes, cuz instead of pushing for no latency issues, bunch of you are crowd surfing for a status quo.

You are not comprehending anything I'm saying. I'm afraid don't know how else to explain it. Sorry.

 

[Will Robinson]

Thanks for that, it brought tears to my eyes I laughed so hard. Its so hard to believe this is anything but brilliant sarcasm, bravo sir.

Sometimes the truth just looks like sarcasm...:whiste:

 

[railven]

You are not comprehending anything I'm saying. I'm afraid don't know how else to explain it. Sorry.

Right back at ya. Leave it at that, not enough time to waste anyways

 

[railven]

Any NV users here that play SW:TOR? I just redownloaded it recently and it stutters bad on my rig. Feeling lazy to download the 15+GBs to test on the GTX 680 system so figured I'd ask here.

Stutters REALLY bad even if my FPS is telling me 60FPS. Not sure if it is a game engine flaw or le GCN pulling a fast one. Don't remember it being this bad when I originally played with my HD 5870.

EDIT: Will pull a Fraps log tomorrow.

 

[Granseth]

Ummm of course it is. Microstutter has nothing to do with vsync and when it happens to the screen. Its to do with the game world "moment" being taken at uneven points. Games use the system time to determine how far the world has moved since the last frame so they can update your view based on your inputs and the animations. Its that which we are measuring as unsmooth and not the frames delivered to the monitor, which is happily happening at 16ms regardless of what is in them or if they are the same one after the other.

(...)

OK, that sounds reasonable. But is it really the GPU that determines the position, or is it just that it ask the game about positions at uneven points? It sort of sounds like an easy fix, but I guess it's not since the games and GPUs have a problem with getting this right.

 

[BFG10K]

OK, that sounds reasonable. But is it really the GPU that determines the position, or is it just that it ask the game about positions at uneven points? It sort of sounds like an easy fix, but I guess it's not since the games and GPUs have a problem with getting this right.

Games generally use a fixed tick for internal timing (e.g. physics, animations, etc). Assuming a variable framerate, the frames are an interpolated display of this internal timer.

 

[Imouto]

OK, that sounds reasonable. But is it really the GPU that determines the position, or is it just that it ask the game about positions at uneven points? It sort of sounds like an easy fix, but I guess it's not since the games and GPUs have a problem with getting this right.

Those two paragraphs are just a deflection not even related to my quote that answer nothing.

 

[parvadomus]

Games generally use a fixed tick for internal timing (e.g. physics, animations, etc). Assuming a variable framerate, the frames are an interpolated display of this internal timer.

This is true, but variable framerate is not an interpolated display of this timer. Generally when frame rate is faster that this fixed tick (gametick) the displayed frame is the same as the last calculated gametick, however some things that dont affect actual game state can be calculated at that precise time, like camera position, particle positions, maybe some effects, animations, etc. but not the positions of the things that "matters" like enemies, etc.
When frame rate is slower, then some gameticks may be calculated before a frame is displayed on screen, this allows the game to run in a timely manner independently of the gpu speed.

 

[Jaydip]

Any NV users here that play SW:TOR? I just redownloaded it recently and it stutters bad on my rig. Feeling lazy to download the 15+GBs to test on the GTX 680 system so figured I'd ask here.

Stutters REALLY bad even if my FPS is telling me 60FPS. Not sure if it is a game engine flaw or le GCN pulling a fast one. Don't remember it being this bad when I originally played with my HD 5870.

EDIT: Will pull a Fraps log tomorrow.

Dude you should have asked couple of days ago :biggrin:

Anyways it was fine most of the time save some spots.I am not very susceptible to ms so ymmv.

 

[Final8ty]

Probably because for the last 23 pages, people have been trying to think up new ways to discredit, delay, destroy, anything to do with the TR testing. Even you promised to go back to lurking, and here you are. The latest attempt is to say that it's impossible to see because the ms latency is to fast for the monitors refresh to show.
Did you not see the comparison video?

I don't see much of that, all im seeing is besides the graphs people want more and talk about how best it could be done.

 

[Keysplayr]

I don't see much of that, all im seeing is besides the graphs people want more and talk about how best it could be done.

But you do see some of that then? So talking about how best it could be done is not all you're seeing? I'm confused.

 

[Final8ty]

But you do see some of that then? So talking about how best it could be done is not all you're seeing? I'm confused.

I don't see discrediting TR in the last few pages, i see people trying to discredit each other.
There has been some good input regardless.

 

[SirPauly]

Any NV users here that play SW:TOR? I just redownloaded it recently and it stutters bad on my rig. Feeling lazy to download the 15+GBs to test on the GTX 680 system so figured I'd ask here.

Stutters REALLY bad even if my FPS is telling me 60FPS. Not sure if it is a game engine flaw or le GCN pulling a fast one. Don't remember it being this bad when I originally played with my HD 5870.

EDIT: Will pull a Fraps log tomorrow.

Indeed! Did use a frame limiter to solve the problem.

 

[Keysplayr]

I don't see discrediting TR in the last few pages, i see people trying to discredit each other.
There has been some good input regardless.

We said 23 pages, so I guess that would include pages from now back 23 pages. And that would probably include TR's findings as well as the concept of benchmarking frametimes on the whole from now on in reviews.


Anywho, are you another proponent of blind testing?

 

[Final8ty]

We said 23 pages, so I guess that would include pages from now back 23 pages. And that would probably include TR's findings as well as the concept of benchmarking frametimes on the whole from now on in reviews.


Anywho, are you another proponent of blind testing?

1) How many pages is irrelevant, the only thing i see is how you want to interpret things as trying discredit TR.

2)Sound like another good idea to expand upon on the graphs.

 

[VulgarDisplay]

But you do see some of that then? So talking about how best it could be done is not all you're seeing? I'm confused.

I have seena few posters participating in an interesting discussion about how to improve tr's testing methods. The only discrediting going on is from people who think the testing method is good enough because it supports their conclusion and they are nonsensically attacking anyone who wants to dig further into the topic.

If more can be done to improve or make sense of the tests the only reason looking into it further could generate the reaction we have seen in the last few pages is because those posters against improving the tests are worried their views could be wrong. People who want to expand and improve the benchmark don't seem to care that it could actually further strengthen the troll's views and weaken their own. Its sad that these trolls feel the need to ruin what was actually turning into the best discussion I have seen in vc&g in a long time.

 

[railven]

Indeed! Did use a frame limiter to solve the problem.

I don't think I've ever used a frame limiter, however, this fixed my stuttering issue on SW:TOR!

Kudos Sirpauly!

 

[f1sherman]

No I am not. Why would I take a very granular trace and then average it all out?! No what I am saying is you take the derivative of the frame time graph, draw lines at +2.5 and -2.5 and anything that swings above then below those two points is microstuttering noticeably.

Derivative of frametimes with respect to time is dimensionless.
You are talking about difference between neighboring frametimes, right?
Which is how it should be done - first locally, and then later stepping back and looking overall.

If you want to process the graph to determine how badly then I suggest you absolute all the values and then sort them, and graph that. In that case its 2.5 that is the threshold but it doesn't directly show that microstuttering is happening, the fact one frame was +2.5 and the next is -2.5 is what matters and not the absolute number of +2.5 and -2.5 frame deviations. A string of -2.5's in a row is not stuttering or noticeably a problem at all.

I don't think you have to worry to which side it's swinging (-2.5 or +2.5).
because the graph will look almost symmetrical with respect to abscissa
else FPS would quickly shoot up/down to unreasonable numbers.

So, just threshold and count(per unit of time) should be sufficient condition.

 

[BrightCandle]

Derivative of frametimes with respect to time is dimensionless.
You are talking about difference between neighboring frametimes, right?
Which is how it should be done - first locally, and then later stepping back and looking overall.

Its the gradient of the frame time graph so derivative is the right term, its just a numerical derivative not formula based. But all it is is just the difference between the frame times. H or distance in X is always 1 as its in regards to the frame number so the calculation is always ((x(n) - x(n-1)) /1) which simplifies to x(n) - x(n-1).

I don't think you have to worry to which side it's swinging (-2.5 or +2.5).
because the graph will look almost symmetrical with respect to abscissa
else FPS would quickly shoot up/down to unreasonable numbers.

So, just threshold and count(per unit of time) should be sufficient condition.

Infact I realise its more than that. The calculation itself takes into account its neighbour already, if the neighbour is not wildly different it doesn't show in the gradient graph. So abs and sorting it will have no impact on the amount of stutter other than to show how severe it actually is. The ideal gradient graph will be all 0's if there is no stutters, jitters or microstutter present. What it can't do is tell you if the actual frame times are delivered at an acceptable rate, only how smoothly they are generated. Its a good secondary graph and I am going to take to plotting both from now on.

I am astonished actually how much microstutter we see in the average game on any hardware, NVidia or AMD. Its not uncommon for the frame time on a good trace to swing by 30% and be on the verge of being perceptible, which likely means there is a panacea of smoothness which we rarely see.

 

[f1sherman]

BTW are you on Windows 8. I'm pretty sure that at least for NV, that's one fine OS
AMD and Win8 is mixed bag, if I recall TR correctly.

 

[BrightCandle]

I need a volenteer who has an NVidia 600 series (680 ideally) card with a 120Hz monitor to test BF3. There is a member in the gaming forum who is seeing severe microstutter in that game even on the lowest settings at 1080p@120. I am at a loss as to the potential way to reduce the problem he is seeing, its mighty persistent against all the usual messing about to remove it. Its remarkably consistent infact:

BF3, 680, 1080p@120, lowest settings on a 3770k@ 4.4Ghz

Thread is here.

Can someone test the game and capture a trace and provide it or graph it yourself and just confirm its not a something happening more widely in the latest set of drivers? Would appreciate it.

 

[BoFox]

Recently posted at ABT forum ( http://alienbabeltech.com/abt/viewtopic.php?f=6&t=27699&view=unread#p81461 )

This only took a few minutes ... here is Battlefield 3 single player with the GTX 680 at 2560x1600 (Ultra plus 4xAA/maxed settings)

Here is the HD 7970 GHz at 2560x1600

Both jitter and they look bad on the charts. Now check 1920x1080 for the GTX 680

(All of these benches are from HDD and the refresh rate is at 60hz as it is from the 'battle of the WHQLs' evaluation; it's still jittery)
Now the HD 7970 GHz at 1920x1080 (60Hz) ... which cleaned up the jitters

i'd say we have to go on a game-by-game basis and also look at the resolution and the detail settings
:hello:

 

[f1sherman]

Bright, dude. Go easy will ya

PM's are to be used for exact that type of content what I sent you.
Sorry, I had no idea that's not acceptable behavior.
In fact it seemed like a right thing to do, instead of boring entire audience with math definitions.

Call that thing derivative, differential, gradient, instead of simply difference. Lets confuse more ppl. What do i care

"unacceptable behavior". Jesus...