[TECH Report] As the second turns: the web digests our game testing methods

[KentState]

I'm confused. Are you just substracting the time between adjacent cells in the frametimes.csv files? Are you saying that frames should be around 5ms apart? In BL2, I'm seeing an average around 15ms with jumps over 25ms every 5-6 frames.

 

[BrightCandle]

Everyone I hope remembers their teenage years in maths? Good.

The first conversion we do for the fraps data everyone knows, you have an ever climbing game time in milliseconds and if we take the difference of those you get the frame times. What you may not realise is you just took a derivative of the game time graph, you determined its slope at all its points. Its not an estimate like you used to do when taking two points in secondary school because the data in this case is discrete, its a completely accurate derivative.

What I have done here is taken the slope of the frame time graph, I have taken the next derivative. The reason is that the in the frame time graph you can see a lot of movement, a lot of noise and its not the absolute value its moving around that is the problem in this case its how far any two frames move away from the ideal that determines the stutter. The noisiness in the graph is the issue and its that which I wanted to plot. I have tried Fourier transformations as well but nothing has struck me as something that shows this problem well until I sorted the abs of this graph.

So the exact transformation is the difference of the frame times from each other, then absoluted so that they are all positive and then I graph the sorted points. If you don't sort it and don't abs the results you also produce two interesting graphs in their own right, the raw differences oscillating around the 0 is quite informative but still too noisy for the purpose I had in mind. But when I started looking for a better way to determine magnitude of frame time I wanted a smooth graph where you could read off the number of milliseconds of microstutter. I have found sorting the absolute values (as it doesn't matter if a frame is long or short only how much it differs from its predecessor) results showed the magnitude of the microstutter quite well and so this is what I am showing.

Alas because it takes a little while to explain what it is I have done I doubt it'll catch on, but it is actually an incredibly useful graph for a trace because you can easily see the amount of microstutter it has, how big that microstutter is and how often it occurs.

Now what I need to do is work out how much microstutter people can see, at what magnitude and frequency and we have a reliable way to turn frame time data into "will I notice this or not". I have a way of capturing the second part too I just haven't finished it yet.

 

[Whitestar127]

Now what I need to do is work out how much microstutter people can see, at what magnitude and frequency and we have a reliable way to turn frame time data into "will I notice this or not". I have a way of capturing the second part too I just haven't finished it yet.

This should be very interesting. Thanks for doing it.

 

[Rikard]

I spotted a mistake on my part. The traces I gave you are just fine, but the plot I attached saying it was for Hitman, was actually for Battlefield 3... All the other plots and traces were correctly labelled. For clarity, here are the plots with correct labeling:

Skyrim wilderness scene:

16.7+-0.8 ms
No visible microstuttering

Battlefield 3, multiplayer:

8.8+-1.5 ms
No visible microstuttering
csv file

Hitman, no FXAA:

15.9+-2.8 ms (Gaussian values, but distribution is not very Gaussian)
Visible microstuttering, but at acceptable level

I think the first case has not visible microstuttering since the 0.8 ms is not large enough. The second case has no visible microstuttering since the FPS is so high (mean frametime is <9 ms) so the variation from frame to frame does not give large instantaneous frame times anyway. The third case has rather large variance (standard deviation is 2.8 ms), and in combination with an FPS near 60 (16 ms) there is some detectable microstuttering. I would be interested to hear your take on this.

Can I have a trace of that as well?
Stuttering verses a real mess is also useful.

Yes, but I want to reproduce it to be 100% sure I give you the correct file! I did not intend to use that trace for anything so I did not book keep it. I wont have time for a couple of day so please be patient.

The absolute variance graph is quite dramatically less severe in this.

I appreciate your way of presenting the results but I think you need another name for it since "variance" is already a well defined statistical term. (In short it is the square of the standard deviation.) Having more than one definition will lead to confusion.

 

[railven]

Sorry was a tad busy, but I'll try to get some Data today after work on my WoW adventures.

I'm running 1440p w/2xSSAA Ultra Settings + triple buffering + v-sync. There are frame drops (ie entering a town, turning the camera and facing a forest), of course, but the microstuttering/stuttering I see is when my FPS meter is telling me 60 and I'm just flying around or walking through a zone.

I'll try to get that data posted today.

 

[f1sherman]

I appreciate your way of presenting the results but I think you need another name for it

uhmm :hmm: ... DELTA!

 

[BrightCandle]

http://benchmark3d.com/microstutter-case-study-skyrim-wtf-edition

 

[Imouto]

http://benchmark3d.com/microstutter-case-study-skyrim-wtf-edition

Mad respect, seriously.

 

[KingFatty]

So he fixed the microstutter by messing around with the CPU, disabling features of the CPU?

 

[Hitman928]

So he fixed the microstutter by messing around with the CPU, disabling features of the CPU?

He forced the game to run on 1 core only. It seems (if others can validate) that the game is extremely buggy (already known) and that by using the multithreaded code paths, it causes a lot of microstutter. Forcing the game to use a single core smoothed out the stutter, though some people in the comments said that only a single core might not be strong enough for a solid 60fps if that is your goal. Either way it all comes back to, there is no one configuration/issue/card/system/driver/etc. that causes microstutter. It's a complex problem with many roots.

 

[Elfear]

http://benchmark3d.com/microstutter-case-study-skyrim-wtf-edition

Very interesting.

Rikard, Brightcandle, Railven - You guys up for testing this?

 

[VulgarDisplay]

I think that the fixes for memory usage and and working with devs will go a long way to fixing this problem. Perhaps with more awareness about how poor the ports are made amd and nvidia can use their influence to nip these problems in the bud early in the Dev cycle.

 

[3DVagabond]

http://benchmark3d.com/microstutter-case-study-skyrim-wtf-edition

Props for posting that. :thumbsup:

 

[Final8ty]

http://benchmark3d.com/microstutter-case-study-skyrim-wtf-edition

:thumbsup:

 

[SPBHM]

when I had issues with stuttering on Skyrim two things worked well, one was using DXtory to lock the framerate to something like 40-50FPS, or underclocking the CPU...
alternatively lowering graphics details would also give me good results...

 

[Final8ty]

http://www.overclock.net/t/1348515/benchmark3d-microstutter-case-study-skyrim#post_19031693

http://www.overclock.net/t/1348515/benchmark3d-microstutter-case-study-skyrim/10#post_19031950

http://www.overclock.net/t/1348515/benchmark3d-microstutter-case-study-skyrim/10#post_19032286

 

[MrK6]

Quick test of Skyrim with 4, 3, 2, and only 1 core affinity. The big problem is that lowering the active cores increases the actual stuttering. You can see more frequent and longer pauses in the single core graph than any other. There might be an argument for setting two core affinity, but overall I'm not seeing the microstuttering being awful. And this is in one of the most demanding areas of the game (open tundra at sunrise) with ENB, SSAO, 6GB+ texture mods, etc. to increase load. In dungeons where I'm pegged at 60FPS I don't see the issue. I'm sure Skyrim's engine could use more work, but that's always been the case.

 

[railven]

Very interesting.

Rikard, Brightcandle, Railven - You guys up for testing this?

Unfortunately I don't have Skyrim.



I did however do that csv file using the instructions Bright provided. While doing it I asked my GF to look over and she too saw the stuttering.

Here is the csv file:
https://docs.google.com/spreadsheet/ccc?key=0AhY9sOddohlTdGdsbHUyb1dfUm9RWnBlTFJKWnljQkE

I uploaded that info the URL provided and got the following charts:

Honestly not sure how to read it, but I can vouch the two of us saw the stutter.

Info on the test:
WoW 64-bit client
All ingame settings on Ultra
CCC Force 2xSSAA
V-synch on

I flew back and forth facing some rocks in the Vale, I saw the stutter, called the GF over, she saw the stutter and then ended the FRAPs.

Fraps and ingame FPS meter fluctuated between 59 and 60.

Do the logs show the stutter?


EDIT:

I did the proptip suggestion and zoomed in on a portion of the frames:

That looks more like what was described for stuttering, no?

 

[f1sherman]

Honestly not sure how to read it, but I can vouch the two of us saw the stutter.

Perhaps dropped frames, similar like this?

https://docs.google.com/open?id=0B11hajGAJWuVdXYwV3p2WVlBQmM

 

[railven]

Perhaps dropped frames, similar like this?

Not sure what you mean, if its dropping frames and that creates the stutter, well that should be easy to fix and is perhaps a setting on my end?

However, I'm using the same settings I used throughout my HD 5870 days and never noticed stuttering on that card.

Well now my resolution is higher and I use less SSAA, with HD 5870 I got away with 1080p with 4xSSAA. Now I'm using 1440p with 2xSSAA.

 

[f1sherman]

I have no idea, on who's end is this. It could be on server's end, and this is not something very unlikely.

But if the frame generation is smooth, fps is good, and you are seeing "stuttering",
that means animation is culprit.

Hence dropped frames. Or perhaps issues with game camera.

 

[railven]

I have no idea, on who's end is this. It could be on server's end, and this is not something very unlikely.

But if the frame generation is smooth, fps is good, and you are seeing "stuttering",
that means animation is culprit.

Hence dropped frames. Or perhaps issues with game camera.

So I assume based on my CSV info, there is no stuttering?

I'm going to do the exact same test on the GF's rig, she has a GTX 680, I'll post my info in a minute.

There is visible stuttering, whether it is a server issue, or something, I dunno. Outside of filming it, I don't know how else to show it.

EDIT: To capture my file all I did was face a mountain, and strafed left and right on my mount. I chose this since it wouldn't tax the GPU, maintain 60 FPS, and still stutter.

 

[Final8ty]

Skyrim frametimes(max everything ingame), Labyrinthian Thoroughfare Dungeon.

Rig in the sig all@stock for the test(7970 MSI OC, CrossFireX disabled) 12.8 drivers.

My Skyrim isn't vanilla, it has quite a few high res mods etc.

http://forums.overclockers.co.uk/showpost.php?p=23549481&postcount=33

 

[railven]

Okay, already saw a difference however - I also saw stutter on the GTX 680, just not as bad haha. I'll post the CSV info on an edit, but for now here is the actual graph (I zoomed in.)

I tried to normalize the two runs as best I could, both rigs are set to ingame ultra, with 8xAA, disabled any outside optimizations (CCC to default, NV panel to default), and both at 1080p full screen
EDIT for more info:

I used the same account, so its the same character, same gear, same mount, yada. I disabled all mods, so its all WoW stuff.
Resolution was set to 1080p, full screen, 8xAA, all ingame options Ultra.
CCC/NV panel reset for the run.
I found a two landmarks easy to identify and faced the wall, then strafed between the two. CSV file for both runs was similar in size.

I noticed stutter on both rigs, but far more on mine (the HD 7970.)

GTX 680 Results:

HD 7970 Results:

I will update my CSV file with the two runs, clearly there is a difference.
EDIT #2: Posted the info in my CSV files:
https://docs.google.com/spreadsheet/ccc?key=0AhY9sOddohlTdGdsbHUyb1dfUm9RWnBlTFJKWnljQkE#gid=0

 

[Fx1]

Quick test of Skyrim with 4, 3, 2, and only 1 core affinity. The big problem is that lowering the active cores increases the actual stuttering. You can see more frequent and longer pauses in the single core graph than any other. There might be an argument for setting two core affinity, but overall I'm not seeing the microstuttering being awful. And this is in one of the most demanding areas of the game (open tundra at sunrise) with ENB, SSAO, 6GB+ texture mods, etc. to increase load. In dungeons where I'm pegged at 60FPS I don't see the issue. I'm sure Skyrim's engine could use more work, but that's always been the case.

i just got a 7970 and the stuttering in skyrim kills me. some people say its vsync issues and others say its mouse acceleration.

It sucks.