Should reviews that don't use FCAT be ignored?

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Erenhardt

Diamond Member
Dec 1, 2012
3,251
105
101
Who decided that 20 lines is a runt frame but 21 lines is a complete frame? nVidia, did. What would happen if it was set at 100 lines, still only ~10% of a frame, or 200 lines? Would the results comparing the 2 brands still be the same? When one company sets the parameters how do we know that they aren't designed to make their products appear superior?

I'm not saying to disregard FCAT. What I'm saying is don't disregard your eyes either.

You just don't know you want it!

People, don't look into eyes of this beast or you will be petrified for ever! I mean, if you play on Xfire your eyes will roll 180 degrees and go out of your butt. And don't say I didn't warn you!
 

Zanovar

Diamond Member
Jan 21, 2011
3,446
232
106
You just don't know you want it!

People, don't look into eyes of this beast or you will be petrified for ever! I mean, if you play on Xfire your eyes will roll 180 degrees and go out of your butt. And don't say I didn't warn you!

erm
 

Lepton87

Platinum Member
Jul 28, 2009
2,544
9
81
Who decided that 20 lines is a runt frame but 21 lines is a complete frame? nVidia, did. What would happen if it was set at 100 lines, still only ~10% of a frame, or 200 lines? Would the results comparing the 2 brands still be the same? When one company sets the parameters how do we know that they aren't designed to make their products appear superior?

I'm not saying to disregard FCAT. What I'm saying is don't disregard your eyes either.

Our subjective experience is the most important I already said that. And I only meant we can trust objective numbers, in your example we should only compare lines and disregard what they call runt frames altogether.
 

Keysplayr

Elite Member
Jan 16, 2003
21,219
55
91
Our subjective experience is the most important I already said that. And I only meant we can trust objective numbers, in your example we should only compare lines and disregard what they call runt frames altogether.

It was the experiences of a few reviewers that led to these investigations in the first place. What did you think happened? Did you think these reviewers just decided to pursue frame latency investigations out of boredom? No. They were prompted by seeing it and feeling it and wondered why.
So, you are absolutely right in saying our opinions are important. Especially if we don't lie to ourselves to the extent the lie becomes the truth.
 
Last edited:

Erenhardt

Diamond Member
Dec 1, 2012
3,251
105
101
It was the subjective experiences of a few reviewers that led to these investigations in the first place. What did you think happened? Did you think these reviewers just decided to pursue frame latency investigations out of boredom? No. They were prompted by seeing it and feeling it and wondering why.
So, you are absolutely right in saying our subjective opinions are important. Especially if we don't lie to ourselves to the extent the lie becomes the truth.

Yep! Just like you did with NV focus group! It is such a great company, why not promote them on the web for free*?

*free hardware
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
I had no idea Key's was part of any program for nVidia, did they change their disclosure policy? Last I checked AMD wasn't requiring it for their "Advocates".
 

bystander36

Diamond Member
Apr 1, 2013
5,154
132
106
FCAT is a tool to take something subjective and make it objective. At least a couple of the sites doing FCAT tests have done subjective tests along side it, and subjectively, the results are the same.

http://www.tomshardware.com/reviews/radeon-hd-7990-review-benchmark,3486-13.html

Pcper has some videos to watch as well.

While it seems like we may not notice the stuttering, when compared side by side, we do notice a difference. We just don't always realize that we are seeing it until you compare it to something without it. That is why we have review sites who can compare them for us.

While there are other things that affect stuttering, the GPU drivers and hardware don't have control over those things, and they should affect both equally. Until things change, FCAT is one of the best tools we have for measuring multi-GPU configurations in regards to stuttering.

It is not the only useful tool, but I would not buy into crossfire or SLI without first seeing some results. It might change when both companies get their drivers in order, but until then, it matters.
 

Lepton87

Platinum Member
Jul 28, 2009
2,544
9
81
Unfortunately CF is still very immature, I had many problems with getting it to work at all
hence I downgraded my QuadFire set-up to a single card. Multi-GPU are more optimized to work with just two card, NV completely discarded 4-way with Titan, saying it supports up to Three-SLI. People having 4 gpus is such a small percent that they don't bother to make it good. ;( But when it worked right it delivered a of FPS. With V-Synch enabled and when fps stayed at 60 fps the experience was comparable to Titan but when fps dropped to 30-40 it was a stutter fest. 30-40fps feel better now.
 

chimaxi83

Diamond Member
May 18, 2003
5,457
63
101
I think this should be viewed on a person by person basis. For example, look at post #3 in this thread for a glance at a radical position on the subject. Where the person believes the whole thing is a sham and only important to some people because it makes their favorite company look better than another. I wonder how long this will be perpetuated. That persons argument, "Yes, ignore them all.... until AMD is better at it than Nvidia"
also applies to the poster himself. What will his tune be if and when AMD surpasses Nvidia at the gameplay experience level? Contrary to his accusations, I can tell you that mine won't change. I believe this method for testing is very important regardless who is doing better in the gameplay experience.
We've turned a corner in the way things are done, this isn't a "trendy" thing that will go away anytime soon BECAUSE this gives the impression of the actual experience instead of simply looking at fps due to the discovery in no small part by H, TR, PCPer and others that the actual gaming experience is indeed more important than raw FPS.
I couldn't agree more. FCAT is what is used, in part, now for this sort of testing. Other tools will emerge that improve upon or expand these ideas.
Should sites that don't use FCAT be ignored? This solely depends on the individual reading the reviews. Third poster in this thread is a good example of how NOT to embrace new technologies or methodologies for benchmark testing.

A whole lot of fluffy dramatic crap in your post. Me saying to ignore the site if you don't like their methodology is a radical position? Talk about a dramatic response. You basically said the same thing I did:

I think this should be viewed on a person by person basis

is basically the same as ignore it if you don't like it. Useful to some people, meaningless to others, ignore it if useless. Show me where I said anything was a sham, please. Oh, and this little gem:

Where the person believes the whole thing is a sham and only important to some people because it makes their favorite company look better than another. I wonder how long this will be perpetuated.

Don't play the ignorant innocence bs. Everyone here (except maybe your biggest supporters) know without a doubt that you wouldn't have attempted to make this the massive deal that you did if your favorite company wasn't the one performing better in the "smoothness" aspect (in the test they developed, no less).

What has always been, and will continue to be, most important to my gameplay experience is ME, as it should be with everyone else. With the exception of one game, my hardware performs great, but people like you would have everyone believe it doesn't matter how it feels to the actual gamer, what matters is what review sites say. Ok :rolleyes:

That persons argument, "Yes, ignore them all.... until AMD is better at it than Nvidia"

Was sarcasm directed at the troll of an OP that I probably could've left out.
 

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
The important "capture" part is open source, MSI AfterBurner currently employs it.

FCAT itself has no idea where the output comes from, be it AMD or nVidia.

FCAT does not necessarily operate properly on AMD hardware. Many reviews which report "actual vs. FRAPS" framerates are wrong. FCAT claims to compare "this is what FRAPS would say" to "this is what you got" but FRAPS is never actually involved, the FRAPS number is just the "actual" FPS + the number of dropped/runt frames... and FCAT has a bug which reports groups of 15 dropped frames on AMD cards (when in fact no frames were dropped). This appears to be due to a colour discrepancy. Look at these results. Notice all of those identical, 15FPS spikes in "FRAPS reported FPS"? Yeah, those are all bullshit. FCAT can paint a much worse picture for Crossfire than is actually going on.
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
FCAT doesn't operate on either companies hardware.

FCAT doesn't claim anything.

Why don't you link to the credible person who says that's what happening instead of linking to a review that shows it yet makes no mention of what you said?

Also the minor spikes in "reported" aren't making CF look bad, it's the "Actual" FPS that are.


Here is what the reviewer said...

In the charts below, we have taken the raw framerate from FRAPS (listed as Reported) and then used FCAT to subtract the invisible runt and dropped frames in order to give a Actual Framerate.

What you're saying makes no sense.
 
Last edited:

Lepton87

Platinum Member
Jul 28, 2009
2,544
9
81
FCAT does not necessarily operate properly on AMD hardware. Many reviews which report "actual vs. FRAPS" framerates are wrong. FCAT claims to compare "this is what FRAPS would say" to "this is what you got" but FRAPS is never actually involved, the FRAPS number is just the "actual" FPS + the number of dropped/runt frames... and FCAT has a bug which reports groups of 15 dropped frames on AMD cards (when in fact no frames were dropped). This appears to be due to a colour discrepancy. Look at these results. Notice all of those identical, 15FPS spikes in "FRAPS reported FPS"? Yeah, those are all bullshit. FCAT can paint a much worse picture for Crossfire than is actually going on.

If true this makes it useless for AMD's cards, they better fix that bug. I wouldn't put it past them that they did it intentionally.
 

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
FCAT doesn't operate on either companies hardware.

FCAT doesn't claim anything.

Why don't you link to the credible person who says that's what happening instead of linking to a review that shows it yet makes no mention of what you said?

Also the minor spikes in "reported" aren't making CF look bad, it's the "Actual" FPS that are.


Here is what the reviewer said...


What you're saying makes no sense.

Have you ever used FCAT? You need to look at the source code. FCAT has NOOOOTHING to do with FRAPS. It never, ever talks or uses FRAPS at all (it can't, as the video is captured on a seperate computer).

FCAT is a tool made by Nvidia, who doesn't necessarily "guarantee" how it works on AMD hardware. AMD may handle the DX colour bar draw calls differently, AMD may have different colour rendering, AMD may do many things. FCAT doesn't necessarily account for that because it's not really in Nvidia's best interest to account for that.

Yes, the reported FPS are generally accurate, but in games like Assassains Creed or Crysis it's not painting an accurate picture. It's claiming that AMD is dropping frames but they aren't, the tool is just reporting groups of 15 missed frames due to 1 or 2 lines of colour that aren't as expected.

The reviewer doesn't know how FCAT works if he thinks FRAPS has anything to do with it. He's using FCAT supplied numbers (the tool literally makes that graph for you) and those numbers are painting a worse picture for AMD than what is actually happening. This is especially bad for reviews which use "frametime delta" values or anything like that, because these dropped frame groups are considered "0ms" frametimes and lead to spikes to 0 on the frametime graph, which is considered a massive delta. All of these values are wrong and can't really be taken seriously until FCAT is fixed and properly coded.

I can go into further detail but I don't want to have to give you a background lesson on how FCAT works.
 
Last edited:

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Have you ever used FCAT? I've read through the code, I've spent days testing using a setup. FCAT has NOOOOTHING to do with FRAPS. It never, ever talks or uses FRAPS at all (it can't, as the video is captured on a seperate computer).

FCAT is a tool made by Nvidia, who doesn't necessarily "guarantee" how it works on AMD hardware. AMD may handle the DX colour bar draw calls differently, AMD may have different colour rendering, AMD may do many things. FCAT doesn't necessarily account for that because it's not really in Nvidia's best interest to account for that.

Yes, the reported FPS are generally accurate, but in games like Assassains Creed or Crysis it's not painting an accurate picture. It's claiming that AMD is dropping frames but they aren't, the tool is just reporting groups of 15 missed frames due to 1 or 2 lines of colour that aren't as expected.

The reviewer doesn't know how FCAT works if he thinks FRAPS has anything to do with it. He's using FCAT supplied numbers (the tool literally makes that graph for you) and those numbers are painting a worse picture for AMD than what is actually happening. This is especially bad for reviews which use "frametime delta" values or anything like that, because these dropped frame groups are considered "0ms" frametimes and lead to spikes to 0 on the frametime graph, which is considered a massive delta. All of these values are wrong and can't really be taken seriously until FCAT is fixed and properly coded. https://twitter.com/CatalystCreator

I can go into further detail but I don't want to have to give you a background lesson on how FCAT works.

Why do you feel the need to make things up?

This will actually give this review a beginning, middle and end so to speak as we go through the motions from FRAPS to FCAT.

You clearly have no idea what you're talking about.

Why do you think it's a good idea to post a bunch of worthless trash, then create links that serve no purpose and have no relation to anything you're saying?

http://www.whitehouse.gov/
 

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
Why do you feel the need to make things up?

You clearly have no idea what you're talking about.

Why do you think it's a good idea to post a bunch of worthless trash, then create links that serve no purpose and have no relation to anything you're saying?

http://www.whitehouse.gov/

Making stuff up? You just don't know how the tool works :rolleyes:

There is no way to communicate between the testing rig and the capturing rig. FRAPS can't communicate between the two. Even if they did, those 15 FPS spikes are NOT recorded by FRAPS, they don't "exist", it's made up from the FCAT tool. FCAT displays a 16 colour pattern, it sees a 1 bar line that doesn't match up to what Nvidia cards show, it figures "well I guess I missed all of the other colours" and makes 15 dropped frames (plus it considers that 1 line a runt frame). So that's 16 "FRAPS frames" that FRAPS never would have reported (as FRAPS relies on the DX Present() call, which doesn't suddenly get 16 extra calls for no reason).

So, unless you can come up with a proper reason (besides "You're wrong! Because you are!") then please stop telling me I'm wrong, because I know how the tool works and you clearly don't even understand how an FCAT testing rig is set up (let alone how FRAPS works).

I didn't post any link? There's a twitter link in your quote but none in my post. Adbot or something, get some antivirus? D:
 
Last edited:

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Prove it then.

Link to something that actually says that, should be well covered by now if it's real.

You have no idea how the tool works, it doesn't capture anything and it doesn't "run" during any games.

Hardware Canucks used FRAPS for FRAPS not FCAT, FCAT doesn't even have that functionality to begin with.

You're wrong because you're making stuff up, have no proof backed by any reviewer, and link to random things like the CCC twitter page which doesn't reference FCAT once using the find function.


It is called FCAT, short for Frame Capture Analysis Tool. It is a set of tools that derives from the NVIDIA performance laboratory. Now please don't throw objectivity and subjectivity concerns at us as yes, this methodology is coming from NVIDIA. Let me state upfront that pretty much all software and scripts can be read out as source code and the simple truth is that the FCAT benchmark method can't see the difference in-between AMD and NVIDIA graphics cards as we look at rendered frames, we are not measuring inside the graphics card(s).

http://www.guru3d.com/articles_pages/fcat_benchmarking_review,1.html
 
Last edited:

SKYMTL

Junior Member
Jun 3, 2013
2
0
0
It's unfortunate my first post here on the AT forums has to be this but here it goes.

Have you ever used FCAT? I've read through the code, I've spent days testing using a setup. FCAT has NOOOOTHING to do with FRAPS. It never, ever talks or uses FRAPS at all (it can't, as the video is captured on a seperate computer).

FCAT is a tool made by Nvidia, who doesn't necessarily "guarantee" how it works on AMD hardware. AMD may handle the DX colour bar draw calls differently, AMD may have different colour rendering, AMD may do many things. FCAT doesn't necessarily account for that because it's not really in Nvidia's best interest to account for that.

Yes, the reported FPS are generally accurate, but in games like Assassains Creed or Crysis it's not painting an accurate picture. It's claiming that AMD is dropping frames but they aren't, the tool is just reporting groups of 15 missed frames due to 1 or 2 lines of colour that aren't as expected.

The reviewer doesn't know how FCAT works if he thinks FRAPS has anything to do with it. He's using FCAT supplied numbers (the tool literally makes that graph for you) and those numbers are painting a worse picture for AMD than what is actually happening. This is especially bad for reviews which use "frametime delta" values or anything like that, because these dropped frame groups are considered "0ms" frametimes and lead to spikes to 0 on the frametime graph, which is considered a massive delta. All of these values are wrong and can't really be taken seriously until FCAT is fixed and properly coded.

I can go into further detail but I don't want to have to give you a background lesson on how FCAT works.

As the writer of the article you mention, I think you need to be corrected on a number of your misconceptions about FCAT, how it works and the general direction in which the GPU review industry is heading.

First and foremost, this is a vendor agnostic solution. It accomplishes its goals by sitting apart from the general display driver. Even the "colored bars" as you call them are very much driver-agnostic. As I am sure you saw when reading the code (not sure how you could miss this), they use variations of CMYK colors processed through a standardized routine and don't have any driver hooks installed. The graphics card may render the bars but that doesn't have an impact on the manner by which they are displayed or the order in which they are meant to appear.

As for your supposition that FRAPS has nothing to do with FCAT, you are right and wrong. While FCAT captures a display output and FRAPS measurements happen earlier in the pipeline, they can still be directly compared. This comparison is what is represented by the "Reported" versus "Actual"; it compares what FRAPS sees versus what is actually displayed on the screen. The FCAT software itself actually has an output value that measures both, essentially taking the measurements from FRAPS' position. Once again, this is quite evident when looking at FCAT's code and its various outputs.

The issue between FRAPS and FCAT is that FRAPS isn't actually tracking what you, the gamer, will see. If you want the ACTUAL framerate, dropped frames MUST be subtracted as they don't visually add anything to a given output. How can a solution claim to hit 100FPS when (for example) 20 frames every second are dropped and or only partially rendered? Simply put, it can't and FCAT has called that bluff.

Is FCAT perfect? Absolutely not. A reviewer has to know exactly what to look for in order to identify any faulty results. It also requires a ton of storage resources. However, as an unbiased tool that shows what the end user will experience, it is far superior to FRAPS.
 
Last edited:

NIGELG

Senior member
Nov 4, 2009
852
31
91
I had no idea Key's was part of any program for nVidia, did they change their disclosure policy?
I find this totally unbelievable that you don't know he's part of the ''Nvidia Focus Group''.This group seems to me to be an AMD bashing group more than an Nvidia focus group.
 

Jaydip

Diamond Member
Mar 29, 2010
3,691
21
81
It's unfortunate my first post here on the AT forums has to be this but here it goes.



As the writer of the article you mention, I think you need to be corrected on a number of your misconceptions about FCAT, how it works and the general direction in which the GPU review industry is heading.

First and foremost, this is a vendor agnostic solution. It accomplishes its goals by sitting apart from the general display driver. Even the "colored bars" as you call them are very much driver-agnostic. As I am sure you saw when reading the code (not sure how you could miss this), they use variations of CMYK colors processed through a standardized routine and don't have any driver hooks installed. The graphics card may render the bars but that doesn't have an impact on the manner by which they are displayed or the order in which they are meant to appear.

As for your supposition that FRAPS has nothing to do with FCAT, you are right and wrong. While FCAT captures a display output and FRAPS measurements happen earlier in the pipeline, they can still be directly compared. This comparison is what is represented by the "Reported" versus "Actual"; it compares what FRAPS sees versus what is actually displayed on the screen. The FCAT software itself actually has an output value that measures both, essentially taking the measurements from FRAPS' position. Once again, this is quite evident when looking at FCAT's code and its various outputs.

The issue between FRAPS and FCAT is that FRAPS isn't actually tracking what you, the gamer, will see. If you want the ACTUAL framerate, dropped frames MUST be subtracted as they don't visually add anything to a given output. How can a solution claim to hit 100FPS when (for example) 20 frames every second are dropped and or only partially rendered? Simply put, it can't and FCAT has called that bluff.

Is FCAT perfect? Absolutely not. A reviewer has to know exactly what to look for in order to identify any faulty results. It also requires a ton of storage resources. However, as an unbiased tool that shows what the end user will experience, it is far superior to FRAPS.

Welcome SKYMTL :)
 

Jaydip

Diamond Member
Mar 29, 2010
3,691
21
81
@ SKYMTL I believe you can detect screen tearing with FCAT as well? how are the frames displayed in such a case?
 

NIGELG

Senior member
Nov 4, 2009
852
31
91
No thanks -- rather have tools.
I rather play games that look and feel smooth to ME!!!

I don't care what tools say as long as I am having fun playing my games.
I am starting to believe that many here are only concerned about numbers and E peen.
 

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
Prove it then.

Link to something that actually says that, should be well covered by now if it's real.

You have no idea how the tool works, it doesn't capture anything and it doesn't "run" during any games.

Hardware Canucks used FRAPS for FRAPS not FCAT, FCAT doesn't even have that functionality to begin with.

You're wrong because you're making stuff up, have no proof backed by any reviewer, and link to random things like the CCC twitter page which doesn't reference FCAT once using the find function.


http://www.guru3d.com/articles_pages/fcat_benchmarking_review,1.html

Nobody knows about it because nobody who does FCAT testing cares really. And they kind of shouldn't, with the current drivers it's unarguably broken, but this issue also occurs with the prototype drivers.

The tool is 2 parts: A DX colour overlay, and a captured video using a DVI splitter which is analyzed by Nvidia's extractor and FCAT tool. FCAT is a perl script which analyses the .scv file outputted by the extractor, and then generates FPS information based on the analysed extractor file (which is just scanline data).

Again, you cannot use FRAPS with FCAT. They are totally seperate things. As SKY confirms, FCAT is run on a seperate computer and simply "claims" a FRAPS number (which has NOTHING to do with FRAPS, as FCAT can't see DX Present() calls, but instead infers them). When you use FCAT, it outputs a graph with "here's what FRAPS says, here's what you dropped, here's the runts" as well as a frametime graph.

Again, I didn't link to anything, there is no link in my post, but only in your quote. I think you have an adword virus or something along those lines.

First and foremost, this is a vendor agnostic solution. It accomplishes its goals by sitting apart from the general display driver. Even the "colored bars" as you call them are very much driver-agnostic. As I am sure you saw when reading the code (not sure how you could miss this), they use variations of CMYK colors processed through a standardized routine and don't have any driver hooks installed. The graphics card may render the bars but that doesn't have an impact on the manner by which they are displayed or the order in which they are meant to appear.

As for your supposition that FRAPS has nothing to do with FCAT, you are right and wrong. While FCAT captures a display output and FRAPS measurements happen earlier in the pipeline, they can still be directly compared. This comparison is what is represented by the "Reported" versus "Actual"; it compares what FRAPS sees versus what is actually displayed on the screen. The FCAT software itself actually has an output value that measures both, essentially taking the measurements from FRAPS' position. Once again, this is quite evident when looking at FCAT's code and its various outputs.

The issue between FRAPS and FCAT is that FRAPS isn't actually tracking what you, the gamer, will see. If you want the ACTUAL framerate, dropped frames MUST be subtracted as they don't visually add anything to a given output. How can a solution claim to hit 100FPS when (for example) 20 frames every second are dropped and or only partially rendered? Simply put, it can't and FCAT has called that bluff.

Is FCAT perfect? Absolutely not. A reviewer has to know exactly what to look for in order to identify any faulty results. It also requires a ton of storage resources. However, as an unbiased tool that shows what the end user will experience, it is far superior to FRAPS.

I can't look through the driver overlay code because as far as I know Nvidia hasn't made the source code public. All I can look at is the perl script that analyses the output. The issue with the coloured bars appears to be A. frame compression of some sort that leaves minor changes in colour, I'm assuming in order to fit large frames across the PCI bridge (which doesn't make sense, considering the bridge is much more massive than the frames) or B. slight differences in colour rendering between each GPU (eg. card 1 renders 0x575757, card 2 renders 0x565656 for some reason).

I am right in that FRAPS has nothing to do with FCAT. That is the unarguable truth. FCAT claiming to know what FRAPS would report is OK, except they inflate the results when errors occur, hence making FRAPS look bad when FCAT is actually at fault.

Dropped frames don't really even exist with either AMD or Nvidia cards. The only time dropped frames occur is with this error. Runt frames are an issue, dropped frames just don't really happen from what I have tested.

FCAT is not unbiased because it doesn't produce identical results on AMD and Nvidia cards. In fact, someone at Nvidia knows about the issue but never fixed it (I can tell you what line in the code there is evidence of this). They also borked the formatting of the run.sps file, making the groupings of 15 dropped frames almost impossible to spot until you fix the formatting....

My point isn't necessarily that FCAT is biased, however, it's that FCAT is a badly coded tool and shouldn't be taken as gospel.
 
Last edited: