Microstutter: The Poll

[Hitman928]

Ok, so out of curiosity, I put together a little sample of gameplay from Max Payne 3. I recorded a run through of each section and then ran through each section again to benchmark with fraps.

As I don't have the full version of fraps, I split each run into two sections so that I could get a bigger sample. There is no correlation in the order of the videos to the order of the fraps results, they have been jumbled so that you don't know which fraps graph belongs to which video, only I do. All of the frame time graphs are scaled the same (each grid line represents 5 ms), however, the axis labels have been removed and not all have an origin of zero, so you cannot determine the frame rate by the frame times and cannot compare the frame rate between graphs.

Each run represents a change in settings/hardware (not saying which run is which until the end). With each run I tried to duplicate the same pattern each time. Each run is not exactly identical but are as close as I could make them. The videos were recorded at 1/2 size but 60 fps, so I recommend actually downloading the source videos and not watching at full screen to compare. This is not a perfect setup, but seeing as how I don't have a high speed camera, it's the best I could do. This is NOT meant to be conclusive or to thoroughly prove anything. I just thought with all the microstutter debate going on, this could be another angle to look at things. Mostly it's an idea that I hope others who can perform a more thorough test here could pick up on.

So, please watch all the videos, vote honestly in the poll, and then, post which videos you think line up with which frame times. If you can't tell, or only want to guess on a particular video, that's fine too. So for instance, if I thought the 'A' videos lined up with the frame times 1-1 and 1-2 graphs and the 'B' videos lined up with the 2-1 and 2-2 frame times, I would post A with 1, B with 2, etc.

Thanks for playing

Videos:

A1 - https://docs.google.com/open?id=0B11hajGAJWuVVmZmNVFxT3VVZ2M
A2 - https://docs.google.com/open?id=0B11hajGAJWuVV0Z5b3dGUDRqOVE
B1 - https://docs.google.com/open?id=0B11hajGAJWuVSV9TV0thcXB1d2s
B2 - https://docs.google.com/open?id=0B11hajGAJWuVRU51eHRfa2dZc2M
C1 - https://docs.google.com/open?id=0B11hajGAJWuVQlRpYlN1VDdyVFE
C2 - https://docs.google.com/open?id=0B11hajGAJWuVVHplcEQ4WGxOeWM
D1 - https://docs.google.com/open?id=0B11hajGAJWuVN2wtNi0ybVE1N1k
D2 - https://docs.google.com/open?id=0B11hajGAJWuVdXYwV3p2WVlBQmM

 

[Hitman928]

Frame times:

 

[VulgarDisplay]

This is how these frametime reviews should be conducted. Blind studies with many people providing the results. I'm giddy to see what happens.

 

[Hitman928]

This is how these frametime reviews should be conducted. Blind studies with many people providing the results. I'm giddy to see what happens.

Thanks for the encouragement, I figured, if we're really gonna do this, we might as well do it right. This definitely isn't the end game, just trying to get the ball rolling in the right direction. Plus, I'd love to see how people vote, if they are willing to put themselves out there

 

[boxleitnerb]

Good idea in principle, but I will not vote because:

To watch is one thing, to actually play and feel the feedback and smoothness (or lack thereof) is another altogether, especially when differences are not massive and blatantly obvious. I therefore cannot guarantee that my judgement would be accurate enough and in line with the gameplay experience.

 

[Final8ty]

They all looked bad to me, sorry.

 

[VulgarDisplay]

Are you playing with a controller? It's making my eyes bleed.

 

[Hitman928]

Are you playing with a controller? It's making my eyes bleed.

Lol, nope kb/m. I tend to be a bit jerky with a mouse though, sorry about that. I tired to be as smooth as I could, but it's just how I play. I think it's more pronounced because of the third person perspective, but there's plenty of slow mo and game controlled camera sequences to compare as well, that's why I picked MP3. I could do it agin with a fps, it would probably be a lot less influenced by input "jerks". I also found that the transitions in and out of slo mo in this game can be a bit jarring, no matter what the setting.

Make sure you're dl'ing them as well because the streaming quality sucks.

 

[thilanliyan]

Thanks for your effort. When I get a chance I will participate...although I don't think I would be able to notice many differences especially since looking at the frametime graphs, there are no consistent spikes.

 

[VulgarDisplay]

Thanks for your effort. When I get a chance I will participate...although I don't think I would be able to notice many differences especially since looking at the frametime graphs, there are no consistent spikes.

Yeah, there are hardly any spikes above 30ms which is I believe what is being said to be the threshold for noticeable stutter.

One thing I've decided is that google docs is slow. 500kb/s is prehistoric.

I honestly can't tell the difference. They all have jerky motion, but I think it's just character animations glitching for the most part. It doesn't look like a smooth game in general, but I haven't played it myself so I can't comment.

 

[Hitman928]

Yeah, there are hardly any spikes above 30ms which is I believe what is being said to be the threshold for noticeable stutter.

One thing I've decided is that google docs is slow. 500kb/s is prehistoric.

I honestly can't tell the difference. They all have jerky motion, but I think it's just character animations glitching for the most part. It doesn't look like a smooth game in general, but I haven't played it myself so I can't comment.

Yeah, the animations are a bit jerky in general, probably a generation behind most games today, watch the trailer and you'll probably see the same thing you're seeing in the video. It's unfortunate because everything else in the game looks great. Maybe I'll pick a different game, a smoother game in general and repeat this. Could be easier to spot differences. Any suggestions?

edit: yeah, next run, switching to something else than google, I figured I would try it since I had it so easily available, but there are other, better options.

 

[Ben90]

After playing some 125FPS Quake Live and watching the second clip of each series here is my (edited) response:

A2: Stutter
B2: Stutter
C2: Stutter
D2: Stutter

I voted "Yes, they were all about the same"

 

[Hitman928]

After playing some 125FPS Quake Live and watching the second clip of each series here is my response:

A2: Jesus Christ that is choppy as hell, definitely the one with microstutter.
B2: Oh Jesus, this one has the stutter as well.
C2: Holy Jesus, can this guy's computer run the game?
D2: Oh Jesus Christ, look at that guy chop around.

I voted "Yes, they were all about the same"

Just out of curiosity, did you download them or just watch them in browser?

 

[Elfear]

After playing some 125FPS Quake Live and watching the second clip of each series here is my response:

A2: Jesus Christ that is choppy as hell, definitely the one with microstutter.
B2: Oh Jesus, this one has the stutter as well.
C2: Holy Jesus, can this guy's computer run the game?
D2: Oh Jesus Christ, look at that guy chop around.

I voted "Yes, they were all about the same"

You mind posting without the expletives next time?


OP - Thanks for taking the time to record all the videos. I'll vote when I get home.

 

[WaitingForNehalem]

You mind posting without the expletives next time?


OP - Thanks for taking the time to record all the videos. I'll vote when I get home.

Agreed, please refrain from using the Lord's name in vain. It is almost as if you are purposely trying to be disrespectful since you used it in all four scenarios.

 

[ICDP]

Thanks for posting these Hitman. Video A looked the worst for me, though it was hard to tell as all looked to have low FPS IMHO. Honestly it doesn't look like microstutter is being observed from your graphs, might just be just low FPS.

 

[ICDP]

Agreed, please refrain from using the Lord's name in vain. It is almost as if you are purposely trying to be disrespectful since you used it in all four scenarios.

I consider having a passage from a religious book for your signature line to be disrespectful. This is an international forum and is not frequented exclusively by Christians. I agree he or she should be more respectful of other peoples cultures and beliefs, so should you. It is hypocritical of someone with a religious message in his/her sig to moan about being disrespectful IMHO.

Please don't say "this is an American site", there is no such thing when it comes to an internet site that is aimed at an international audience.

Enough off topic though.

 

[Hitman928]

Thanks for posting these Hitman. Video A looked the worst for me, though it was hard to tell as all looked to have low FPS IMHO. Honestly it doesn't look like microstutter is being observed from your graphs, might just be just low FPS.

I'll let you know, it's not low fps. I think I'm going to put this on hold for now though and redo this with a game with better animation and tweak a few things in my process. Looking at the game itself and the videos, there's just a lot of jerkiness in a lot of areas that don't go away whatever the settings. This game is known for really stiff animations though. I'll work on a new set of videos today and should have somthing better by late tonight or tomorrow. Go ahead and view/vote if you want in the mean time.

 

[ICDP]

I'll let you know, it's not low fps. I think I'm going to put this on hold for now though and redo this with a game with better animation and tweak a few things in my process. Looking at the game itself and the videos, there's just a lot of jerkiness in a lot of areas that don't go away whatever the settings. This game is known for really stiff animations though. I'll work on a new set of videos today and should have somthing better by late tonight or tomorrow. Go ahead and view/vote if you want in the mean time.

Great work and thanks for doing this. I know it is a very labour intensive process and do appreciate it. Good idea doing posting the iharts without GPU details, it makes the test blind.

 

[WaitingForNehalem]

I consider having a passage from a religious book for your signature line to be disrespectful. This is an international forum and is not frequented exclusively by Christians. I agree he or she should be more respectful of other peoples cultures and beliefs, so should you. It is hypocritical of someone with a religious message in his/her sig to moan about being disrespectful IMHO.

Please don't say "this is an American site", there is no such thing when it comes to an internet site that is aimed at an international audience.

Enough off topic though.

How is having a Bible verse disrespectful? I'm not sure how that could be taken maliciously. Everyone is free to believe in what they choose and express their beliefs in a respectful, peaceful way. I don't see anything wrong with that.

But I agree, enough digression, back to the topic at hand.

 

[ICDP]

How is having a Bible verse disrespectful?

Do you not see the irony in this statement on an international forum? I will make it simple for you, because not everyone on this planet is a Christian and believes the Bible. There are Muslim, Hindu, Atheist, Buddhist etc. I won't tell you what beliefs I have and I certainly wouldn't put it in a sig on an international forum, because that would be disrespecting other peoples beliefs.

 

[lakedude]

Are you playing with a controller? It's making my eyes bleed.

This^

The input is way too herky jerky to really get a feel for microstutter.

 

[Greenlepricon]

Maybe my eyesight really is pretty bad but they all look decent to me. I do notice some of the stutter but I tend to play jerky too and am not that picky as long as it's relatively playable by my standards. Also I'm on my laptop which doesn't help the quality right now. I'll check again on my desktop and see if my opinion changes at all.

 

[blastingcap]

Good idea in principle, but I will not vote because:

To watch is one thing, to actually play and feel the feedback and smoothness (or lack thereof) is another altogether, especially when differences are not massive and blatantly obvious. I therefore cannot guarantee that my judgement would be accurate enough and in line with the gameplay experience.

Not to mention that FRAPS records things in the middle of the graphics chain, not at the end like a high-speed video camera would. So all this frametime talk is useless without a high speed camera to back it up.

More info here about why FRAPS is insufficient:

http://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking/11

 

[Hitman928]

Not to mention that FRAPS records things in the middle of the graphics chain, not at the end like a high-speed video camera would. So all this frametime talk is useless without a high speed camera to back it up.

More info here about why FRAPS is insufficient:

http://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking/11

Yes, a high speed camera would be best . . . unfortunately I don't have one. Part of this experiment is to see just how useful, if at all, fraps is to track microstutter.