FCAT: The Evolution of Frame Interval Benchmarking

[Grooveriding]

http://www.anandtech.com/show/6862/fcat-the-evolution-of-frame-interval-benchmarking-part-1

So AT is going to start using this new tool to measure frame intervals. No data yet. From the screenshots it looks like what the pcper website was using and releasing data from.

Nice to see transparency from AT, whereas pcper was releasing data without disclosing they were using tools from nvidia and keeping that information hidden. Now that we have one more than one site using this tool (are there more than just AT and pcper now?), there will be a proper level of corroboration and confidence brought by comparing results attained.

FCAT, the Frame Capture Analysis Tool, is NVIDIA’s take on what the evolution of frame interval benchmarking should look like. By moving the measurements of frame intervals from the start of the rendering pipeline to the end of the pipeline, FCAT evolves the state of benchmarking by giving reviewers and consumers alike a new way to measure frame intervals. A year and a half ago the use of FRAPS brought a revolution to the 3D game benchmarking scene, and today NVIDIA seeks to bring about that revolution all over again.



The solution to this is in the first-half of FCAT, the overlay tool. The overlay tool at its most basic level is a utility that color-codes each frame entering the rendering pipeline. By tagging frames with color bars, it is possible to tell apart individual frames by looking at the color bars. Regardless of the action on the screen (or lack thereof), the color bars will change with each successive frame, making each frame clear and obvious.

​

On a technical level, the FCAT overlay tool ends up working almost identically to video game overlays as we see with FRAPS, MSI Afterburner, and other tools that insert basic overlays into games. In all of these cases, these tools are attaching themselves to the start of the rendering pipeline, intercepting the Present call, adding their own draw commands for their overlay, and then finally passing on the Present call. The end result is that much like how FRAPS is able to quickly and simply monitor framerates and draw overlays, the FCAT overlay tool is able to quickly insert the necessary color bars, and to do so without ever touching the GPU or video drivers.


With the frames suitably tagged, the other half of the FCAT solution comes into play, the extractor tool. By using a PC capture card, the entre run of a benchmark can be captured and recorded to vide for analysis. The extractor tool in turn is what’s responsible for looking at the color bars the overlay tool inserts, parsing the data from a video file to find the individual frames and calculate the frame intervals. Though not the easiest thing to code, conceptually this process is easy; the tool is merely loading a frame, analyzing each line of the color bar, finding the points where the color bar changes, and then recording those instances.

 

[sontin]

It's the same:
http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Testin

 

[AdamK47]

Congratulations to Nvidia for successfully shifting the focus of average frames per second to frame time.

 

[csbin]

Congratulations to Nvidia for successfully shifting the focus of average frames per second to frame time.

:sneaky:

 

[ElFenix]

Congratulations to Nvidia for successfully shifting the focus of average frames per second to frame time.

which is instantaneous fps

 

[blastingcap]

*In Tychus Findlay voice*

Hell... it's about time.

Looking forward to AT's take on this.

Frames per second is not good enough of a measure when the human eye can see faster than 1/60 fps. Many users have complained about microstutter over the years, and high speed footage has shown that microstutter is real.

 

[railven]

This be an NV tool, correct?

Let's hope it works and AMD fix their shit! Perhaps NV can send some of their tech ops over there to get the ball rolling

 

[SPBHM]

crossfire is useless if this is the result

Far Cry 3 testing is pretty good, I always noticed that something was really wrong with that game, and stopped playing because of the amount of stuttering.

but overall single GPU looks good for AMD.

http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Testin

 

[notty22]

TH also released a article today announced they are also using a FCAT system .Challenging FPS: Testing SLI And CrossFire Using Video Capture

Today, we're exploring another tool that's going to help us dig into the performance of graphics cards (particularly multi-GPU configurations): Nvidia's Frame Capture Analysis Tool, or FCAT.

 

[railven]

crossfire is useless if this is the result

<snip>

<snip>


Far Cry 3 testing is pretty good, I always noticed that something was really wrong with that game, and stopped playing because of the amount of stuttering.


<snip>


but overall single GPU looks good for AMD.

http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Testin

If you read this page:

http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Tes-11

Seems V-sync fixes a lot of the issues. Explains why all the CFX users kept saying to use a frame limiter.

Perhaps AMD needs to revist their AFR method.

 

[ICDP]

If you read this page:

http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Tes-11

Seems V-sync fixes a lot of the issues. Explains why all the CFX users kept saying to use a frame limiter.

Perhaps AMD needs to revist their AFR method.

This is true. Using RadeonPro I can eliminate or vastly reduce Crossfire microstutter/frametime problems. Vsync, Triple Buffering, Flip queue sizes, FPS caps can all be used to work around the problem. Even if you do prefer to use no vsync or FPS caps it is possible to work around the problem using flip queue adjustments.

It does mean CF can be a case of trial and error to get working well, but it can be done. I would still recommend SLI if you were starting from scratch, but if you have an AMD 7xx0 series card and are prepared to tinker, going CF does still give a massive performance boost.

 

[GaiaHunter]

I've been watching the pc perspective videos - outside of the skyrim video the only thing I've noticed is the screen tearing.

 

[SPBHM]

If you read this page:

http://www.pcper.com/reviews/Graphi...ils-Capture-based-Graphics-Performance-Tes-11

Seems V-sync fixes a lot of the issues. Explains why all the CFX users kept saying to use a frame limiter.

Perhaps AMD needs to revist their AFR method.

"It does appear that enabling Vsync will help alleviate the runts issue seen with AMD Radeon cards in CrossFire but at the cost of much more frame variance and stuttered animation on games that previously didn&#8217;t exhibit that problem. "

"owever, enabling Vsync creates a whole host of other potential issues that gamers have to deal with. Even though the goal of removing visual tearing is met with the option turned on, you do add latency to the gameplay experience, as much as 60ms in some cases, from input to display. "

doesn't sound like a solution, but it would be interesting to see some testing with a limited max framerate without vsync.

I've been watching the pc perspective videos - outside of the skyrim video the only thing I've noticed is the screen tearing.

look carefully there is a clear difference

https://www.youtube.com/watch?v=nvS9JBzUzMM

https://www.youtube.com/watch?v=lY0iFojEKRU

 

[GaiaHunter]

"It does appear that enabling Vsync will help alleviate the runts issue seen with AMD Radeon cards in CrossFire but at the cost of much more frame variance and stuttered animation on games that previously didn’t exhibit that problem. "

"owever, enabling Vsync creates a whole host of other potential issues that gamers have to deal with. Even though the goal of removing visual tearing is met with the option turned on, you do add latency to the gameplay experience, as much as 60ms in some cases, from input to display. "

doesn't sound like a solution, but it would be interesting to see some testing with a limited max framerate without vsync.

Yeah, but have you noticed the screen tearing of the videos?
Is stuttering worse than see part of your screen just act on its own?

 

[VulgarDisplay]

Can anyone else spot the problems with this picture?

That is not how frames are drawn on the screen at all.

 

[GaiaHunter]

"It does appear that enabling Vsync will help alleviate the runts issue seen with AMD Radeon cards in CrossFire but at the cost of much more frame variance and stuttered animation on games that previously didn&#8217;t exhibit that problem. "

"owever, enabling Vsync creates a whole host of other potential issues that gamers have to deal with. Even though the goal of removing visual tearing is met with the option turned on, you do add latency to the gameplay experience, as much as 60ms in some cases, from input to display. "

doesn't sound like a solution, but it would be interesting to see some testing with a limited max framerate without vsync.



look carefully there is a clear difference

https://www.youtube.com/watch?v=nvS9JBzUzMM

https://www.youtube.com/watch?v=lY0iFojEKRU

In Far Cry 3 I don't see much difference.
In Sleeping Dogs I see a bunch of screen tearing in the 7950 and outside of that I actually prefer the 7950 image - looking at the steps it seems the 7950 is running at higher IQ.

I doubt if there wasn't monikers there people would be able to accurately pick sli from cf.

And of course these tests don't include latency in the first place.

 

[SPBHM]

the 660 TI SLI side looks a lot smoother here, specially when he is moving the mouse.

 

[railven]

"It does appear that enabling Vsync will help alleviate the runts issue seen with AMD Radeon cards in CrossFire but at the cost of much more frame variance and stuttered animation on games that previously didn&#8217;t exhibit that problem. "

"owever, enabling Vsync creates a whole host of other potential issues that gamers have to deal with. Even though the goal of removing visual tearing is met with the option turned on, you do add latency to the gameplay experience, as much as 60ms in some cases, from input to display. "

doesn't sound like a solution, but it would be interesting to see some testing with a limited max framerate without vsync.

Tearing/Input Lag/Microstutter is all subjective. There is really no fix-all solution but more so options. I can deal with some input lag but absolutely can't handle tearing (it ruins a game for me.)

If v-sync can solve Tearing/Microstutter at the expense of a little input lag - that is a solution to me.

 

[railven]

Okay, so there are a few threads about this issue (seems a lot of sites dropped their articles on the same date), but, dunno where to post anymore haha.

Either way from the PCPer article based on all the hooplah that occured here, i was not honestly expecting this conclusion:

Finally, one of the newest games to our test suite, Sleeping Dogs, the AMD Radeon HD 7970 holds a sizeable advantage across the board of the three tested resolutions. The margins are 34% at 1920x1080, 37% at 2560x1440 and 23% when using triple displays.

While some people might have assumed that this new testing methodology would paint a prettier picture of NVIDIA’s current GPU lineup across the board (due to its involvement in some tools), with single card configurations nothing much is changing in how we view these comparisons. The Radeon HD 7970 GHz Edition and its 3GB frame buffer is still a faster graphics card than a stock GeForce GTX 680 2GB GPU. In my testing there was only a couple of instances in which the experience on the GTX 680 was faster or smoother than the HD 7970 at 1920x1080, 2560x1440 or even 5760x1080.

So much for AMD cheating with the Never Settle driver package. People were waiting for this article to set things straight, and I hope they accept this as their answer.

 

[Rikard]

"owever, enabling Vsync creates a whole host of other potential issues that gamers have to deal with. Even though the goal of removing visual tearing is met with the option turned on, you do add latency to the gameplay experience, as much as 60ms in some cases, from input to display. "

It would be really nice if they would actually measure it instead of just winging it. That would be useful information to consumers.

 

[raghu78]

Tearing/Input Lag/Microstutter is all subjective. There is really no fix-all solution but more so options. I can deal with some input lag but absolutely can't handle tearing (it ruins a game for me.)

If v-sync can solve Tearing/Microstutter at the expense of a little input lag - that is a solution to me.

yeah i play with vsync on. i cannot stand tearing at all.

 

[GaiaHunter]

Okay, so there are a few threads about this issue (seems a lot of sites dropped their articles on the same date), but, dunno where to post anymore haha.

Either way from the PCPer article based on all the hooplah that occured here, i was not honestly expecting this conclusion:



So much for AMD cheating with the Never Settle driver package. People were waiting for this article to set things straight, and I hope they accept this as their answer.

If you read tomshardware you will get a similar feeling.

Pretty graphics need to actually translate into something the end user can see.

We've all experienced games, even the same style of game, where despite having frame rates of 50+ they still don't feel fluid, while games with much lower FPS actually feel fluid.

I'm still not a fan of slowing down videos by half or one third to show me something.

I rather have the testers also add their subjective input at the same time they use different measurement tools, lest we forget the objective is to have an enjoyable gaming experience rather than just having a number contest for the sake of it.

 

[Homeles]

So much for AMD cheating with the Never Settle driver package. People were waiting for this article to set things straight, and I hope they accept this as their answer.

Was that ever a serious argument? Also, AMD still has plenty of single card stutter issues to address. Keep in mind, this review is using newer drivers.

 

[njdevilsfan87]

Congratulations to Nvidia for successfully shifting the focus of average frames per second to frame time.

Rightfully so because I've tried out CF before and it's frame rate numbers felt like such a bad lie. And this was well over a year ago. Maybe two. The sad thing is it looks like it hasn't improved at all.

 

[GaiaHunter]

The interesting thing is

Frame time variance is relatively high for both graphics setups on this game. The GeForce cards seemed choppier, based on my experience, but that's a subjective call.

The spikiness from AMD's Radeon cards can be demonstrated by looking at frame time variance. At the 95th percentile, we can see that frames aren't being delivered as consistently. I did notice the game felt a little laggier, too.