Should reviews that don't use FCAT be ignored?

Page 4 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
Do you have any examples of sites using the FRAPS number from FCAT or are you assuming this?

It's the only way to get the data, it's the output that FCAT gives you (actual and "FRAPS claimed" FPS). SKYMTL just confirmed this on the previous page (he's the HWC reviewer).
 

bystander36

Diamond Member
Apr 1, 2013
5,154
132
106
It's the only way to get the data, it's the output that FCAT gives you (actual and "FRAPS claimed" FPS). SKYMTL just confirmed this on the previous page (he's the HWC reviewer).

It is not the only way. It is not how Tom's Hardware said they got the data. The author (cleeve in the comment section) said that they made separate runs for their FCAT and fraps data.
 

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
It is not the only way. It is not how Tom's Hardware said they got the data. The author (cleeve in the comment section) said that they made separate runs for their FCAT and fraps data.

Tom's hardware ran a different test than most, notice how they said their results weren't accurate. None of you have ever even seen FCAT output, you don't realize it gives you an .sps file with raw/FRAPS and real/actual FPS for each second of the benchmark.
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
It's not giving FRAPS info, it's giving FPS based on how many frames are displayed, including RUNT frames.


Why are you so confused? FRAPS and FCAT are two different things and take place on opposite ends of the pipeline, FRAPS reads the front end and FCAT reads off the backend at the display.


Use your words better, talk in a way us simple folk can understand what only you out of so many have come to know.
 
Last edited:

sushiwarrior

Senior member
Mar 17, 2010
738
0
71
Where is that located? All I get from the extractor is an .xls file with these categories:

[frame] [scanlines] [time (ms)] [fps] [frame start (s)] [screen refresh] [color]

Oh my god you actually have to run the FCAT tool o_O that's just the extractor. Use run_doall.bat and run_fcat.bat, they output a folder under Analysis/NV/"game"/"card" with a frametimes.sps and run.sps file, run.sps holds the "FRAPS" value and "actual" value.
 

Jaydip

Diamond Member
Mar 29, 2010
3,691
21
81
You're just deflecting the fact that you don't know how FCAT works or you wouldn't ask such a question.

Any game were you are panning left and right the most has got to be the answer surely :confused:

It's an issue in any game without Vsync on.

Deflection? lol. Have you guys played Skyrim? it exhibits more screen tear than any game I have played with vsync off.My question remains how do we know that from FCAT?
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Your claim is that FCAT can make CF look worse than it actually is, because it's reporting higher reported FPS than what are being reported, thus the loss of FPS looks greater with the removal of RUNT and dropped frames because the actual displayed with runt/dropped frames is actually 15 fps lower than what FCAT is reporting.

Is that correct?

Why does this not effect Nvidia, is it because they don't have the RUNT problem that AMD has?

Then to wrap this up, your feeling is that because you believe there is an error in the script for output, the rest of the results are invalid correct?

My contention is that the actual FPS is the problem, not how many frames they lost due to RUNT and dropped.


Going off Techreport, which I know for a fact used both FRAPS and FCAT I'm going to have to dismiss your idea as there isn't evidence to support Nvidia's FCAT is giving FPS to CF where FRAPS does not.

http://techreport.com/review/24553/inside-the-second-with-nvidia-frame-capture-tools/9

In fact to the contrary, both Nvidia and AMD score higher FPS with FRAPS than they do with FCAT. It's only AMD that suffers the huge loss of performance once the RUNT frames are removed. Every reviewer I can recall has used FRAPS for their FRAPS output and FCAT for their FCAT output. No reviewer that I know of uses FCAT results and calls them FRAPS results.

Until reviewers pick up on what you've claimed it's just a claim, you've offered no real proof to your claim you are simply making one. If you really feel you've discovered something you should send it to some of the major reviewers who do use FCAT. I'm not against it being wrong, only your assertion.
 
Last edited:

bystander36

Diamond Member
Apr 1, 2013
5,154
132
106
Tom's hardware ran a different test than most, notice how they said their results weren't accurate. None of you have ever even seen FCAT output, you don't realize it gives you an .sps file with raw/FRAPS and real/actual FPS for each second of the benchmark.

Just because they give that info, does not mean everyone uses that info. You are assuming.

According to Pcper.com, they also ran separate runs: http://www.pcper.com/reviews/Graphi...art-3-First-Results-New-GPU-Performance-Tools
Here are these results and our discussion. I decided to use the most popular game out today, Battlefield 3 and please keep in mind this is NOT the worst case scenario for AMD CrossFire in any way. I tested the Radeon HD 7970 GHz Edition in single and CrossFire configurations as well as the GeForce GTX 680 and SLI. To gather results I used two processes:

  1. Run FRAPS while running through a repeatable section and record frame rates and frame times for 60 seconds
  2. Run our Frame Rating capture system with a special overlay that allows us to measure frame rates and frame times with post processing.
 

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
I don't know where all of the confusion is coming from?

Sushiwarrior says: The FRAPS output, like what was used by HWC (he never said nobody ever used the benchmark differently), is generated by FCAT itself and he's saying that it's not an accurate reading to what the actual FRAPS tool would generate. Toms and PCPer have tried it different and run 2 separate benches. If anything, these sites going through the effort to make separate a run through as identical as possible (this seems impossible to me) does more to back up what sushiwarrior is saying. If FCAT generates an accurate FRAPS output, why would they bother to attempt separate benches? That's twice the work for no reason.
 

BrightCandle

Diamond Member
Mar 15, 2007
4,762
0
76
Pcper.com on their podcast was talking about a second problem AMD had which showed up in FCAT where the output frame would sometimes interleave for a couple of lines. So you would normally expect blue then yellow, but actually it would be blue, then yellow for 2 lines, then back again blue for a couple of lines and finally yellow for a whole frame.

The reason is that at certain resolutions AMDs compositor can't do it in hardware, so its done in software. There is a limitation in what the software can do that means sometimes you get a few scanlines bleeding into each other. Its really bad, this is quite a serious bug that makes even vsync tear!

Without the context in the code I cant be sure but I think that is what the code listed does, it removes these errant few lines as they aren't in the right place as this would mess up the fps conversion that expects frames to come one after the other. Since its using scanlines to some extent ignoring the lines is showing increased variation than the actual displayed, however its a bug and these interleaved tears are not easy to reason about and ignoring them is a reasonable strategy.

Its not 15-30 frames though, that would be half second spikes and I have seen no traces with that at all, that is absolutely untrue. We are talking about 0.1% due to an additional bug in AMDs composting when in eyefinity resolutions at the worst and normally its 0.
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Because FCAT isn't reading the front end like FRAPS does, which is why FCAT reports slightly lower FPS before RUNT and DROPPED frames are removed than FRAPS because they're on opposite ends.

As to why they do both, well it's simple, they're two different measuring points. Two different sets of DATA. What we learn from FRAPS is what occurs before the GPU, what we learn from FCAT is what makes it to the display.

Knowing the front and back is key to understanding the entire scope of what is taking place, they're doing more work because it's work that needs to be done.

Pcper.com on their podcast was talking about a second problem AMD had which showed up in FCAT where the output frame would sometimes interleave for a couple of lines. So you would normally expect blue then yellow, but actually it would be blue, then yellow for 2 lines, then back again blue for a couple of lines and finally yellow for a whole frame.

The reason is that at certain resolutions AMDs compositor can't do it in hardware, so its done in software. There is a limitation in what the software can do that means sometimes you get a few scanlines bleeding into each other. Its really bad, this is quite a serious bug that makes even vsync tear!

Without the context in the code I cant be sure but I think that is what the code listed does, it removes these errant few lines as they aren't in the right place as this would mess up the fps conversion that expects frames to come one after the other. Since its using scanlines to some extent ignoring the lines is showing increased variation than the actual displayed, however its a bug and these interleaved tears are not easy to reason about and ignoring them is a reasonable strategy.

Its not 15-30 frames though, that would be half second spikes and I have seen no traces with that at all, that is absolutely untrue. We are talking about 0.1% due to an additional bug in AMDs composting when in eyefinity resolutions at the worst and normally its 0.

I had forgotten about this, thanks for reminding me. It seemed so trivial at the time compared to the greater implications. :thumbsup:
 
Last edited:

wand3r3r

Diamond Member
May 16, 2008
3,180
0
0
I don't agree or disagree, but I have no doubt it can easily be misleading (intentional or not). Anytime you have a company sponsoring/developing a tool that shows their product in a good light and the competition in a bad light there is valid reason to question the results. Conflict of interest much? We don't need a tool to demonstrate crossfire has issues. Funny to see fanbois defending / arguing about an app they have no clue about (the internals). Neither do most reviewers, they don't make software, they just use it. As if they are going to solve the intricate details of software code, what a ridiculous assumption.
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Of course nVidia wanted to show off their years of developing fixes for AFR. It was win/win for everyone (well except AMD, but it will end up being good for them since the exposure should help their drivers). It showcased all their hard worked while showing AMD in a poor light and it gave consumable data to "feelings" that most had already accepted and often had completely dismissed mGPU based on their limited experience with the worse offender. The RUNT frames are only one of the problems, there is also the undeniable stutter.

It also showed me there was a away around it, I could accept input lag and enable my second 7950. I've used this information in practice, where without vsync I couldn't mentally cope with the uneven frame delivery whereas with vsync I can enjoy my second card, while not having to push my other one nearly as hard.

Can't be any more ridiculous than people who add nothing and discuss nothing but simply enter threads to call people names like fanboi.
 
Last edited:

wand3r3r

Diamond Member
May 16, 2008
3,180
0
0
Kind of like entering threads and arguing that the NVIDIA code works as described without even basic understanding of software? :p Lot's of value in that I guess. I'm sure NV wanted to "show off", it doesn't mean the software doesn't cheat or even works as implied. I entered the thread to point out the irony of the situation. I'd say it was worth posting considering the situation of the thread.
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
Kind of like entering threads and arguing that the NVIDIA code works as described without even basic understanding of software? :p Lot's of value in that I guess. I'm sure NV wanted to "show off", it doesn't mean the software doesn't cheat or even works as implied.

Who did that exactly?

And why are you accusing people of doing it when you yourself don't even understand the software, how could you possibly judge if another did or not?

RUNT frames are shown in the overlay, which is independent of nVidia at this point.

Read what BrightCandle said...

Do you even have CrossFire?
 
Last edited:

wand3r3r

Diamond Member
May 16, 2008
3,180
0
0
I haven't read the code, nor do I have to, I'm not arguing if it works or not. I am merely pointing out the irony of your defense of the software when you clearly cannot read it and understand it but you try to quickly shut the door on the guy that is questioning it.

I find it ironic that NV sponsored it, nobody knows if it works properly, and you (appear to) stand behind it (until AT or another reviewer disproves it which they are mostly incapable of doing).

As for crossfire, this isn't about crossfire, it's if the tool even works or is entirely misleading if not outright lying. I don't know, but I wouldn't leave the possibility out that it doesn't. I already said crossfire clearly has issues, but who knows if this tool is correct in the least.

So to summarize, I don't know if it works but I don't entirely dismiss that it may or may not.

BrightCandle and SushiWarrior both appear to have considerable more knowledge about it than most people, but who knows if either are correct. I'm not trying claim either one is correct but they could be. The point is there is nothing wrong with questioning a clearly business motivated tool that portrays their own product better and is questionable about the integrity of the data for their competitors product.
 
Last edited:

BrightCandle

Diamond Member
Mar 15, 2007
4,762
0
76
Actually I am pretty certain the guys at pcper.com do understand the code. They have been developing statistics programs of their own around the data and have extensively tested it. While it comes from NVidia about the only thing pcper.com use nvidia tools for is to get the number of scan lines for a frame within a frame, from there its all custom software from them. The scanline count is easy to validate by looking at the frames, and the process showed this odd problem with AMD cards like I mentioned before.

So if pcper with their entirely custom statistics are getting the same results as people using all of the FCAT tooling then it suggests that there is no intended NVidia bias in the tool, no bad counting etc, its purely the origin of the tool that people have issue with. Who made it doesn't matter, its a proven to provide decent data for both sets of cards at this point, there just isn't any bias in the tooling nor is there really space for it. The tool is working on images, there is no way for it to know which output is NVidia and which is AMD produced.

So I keep hearing this bias argument and I am looking at how the tool works and keep asking myself how anyone can make that argument with a straight face, its simply impossible for it to show bias based on how it works. What is disturbing is the number of bugs in AMD's output this tool shows up, but then presumably NVidia didn't expect that either as it was a tool they used internally for their own purposes.

The other thing I always want to mention about this is that fraps still matters. We still need to measure backpressure because it defines the contents of the frames whereas FCAT only measures the smoothness of the frames, not what is in them,
 

BallaTheFeared

Diamond Member
Nov 15, 2010
8,115
0
71
I haven't read the code, nor do I have to, I'm not arguing if it works or not. I am merely pointing out the irony of your defense of the software when you clearly cannot read it and understand it but you try to quickly shut the door on the guy that is questioning it.

I fully understand the software and what it does. I didn't go over every line of code, I had no reason to. Everyone who matters has seen the code, one guy on the net says he found the line that targets AMD hardware? Please, on this forum I'd sooner believe I was adopted than anything people claimed without a slightest bit of proof.

Besides I didn't slam any doors on him we merely debated the aspect of the code he was questioning in relation to the results and what is and isn't actually a FRAPS readout, but given your lack of engagement in much of anything but name calling it doesn't surprise me you weren't able to keep up, nor does it surprise me you took the side of someone you thought was "defending" AMD.

I find it ironic that NV sponsored it, nobody knows if it works properly, and you (appear to) stand behind it (until AT or another reviewer disproves it which they are mostly incapable of doing).

It tags a frame, gives it a color, and another program decides if the frame is large enough to be considered a real frame to the end user. You can see the frames in question, they're little slivers in the screen shots.

There is no deep mystery on how it works, or what it does, or how it does it... All the code is open for viewing by anyone, ignorant fear mongering isn't the defense AMD deserves.

As for crossfire, this isn't about crossfire, it's if the tool even works or is entirely misleading if not outright lying. I don't know, but I wouldn't leave the possibility out that it doesn't. I already said crossfire clearly has issues, but who knows if this tool is correct in the least.

It's all just black magic and bogeymen to you, isn't it? lol sad. It's all about crossfire at this point, but the thread is more an extension of "Are reviews that don't use frame time graphs worth reading" from months prior, try to keep up at least if you're going to comment in a thread instead of blindly taking AMD's side on everything.

So to summarize, I don't know if it works but I don't entirely dismiss that it may or may not.

BrightCandle and SushiWarrior both appear to have considerable more knowledge about it than most people, but who knows if either are correct. I'm not trying claim either one is correct but they could be. The point is there is nothing wrong with questioning a clearly business motivated tool that portrays their own product better and is questionable about the integrity of the data for their competitors product.

When you're dealing with simple to follow, understand, and viewable code... There is something wrong with it. It's not some closed program that churned out results from nVidia HQ, what you're basically saying is nVidia provided a tool for everyone to examine right down to it's source code and pulled a fast one on everyone in the industry, get a grip on yourself.

They must really be getting at AMD with their lies, AMD is even going through the trouble of fixing a make-believe issue, much like they did with your make believe frame time single gpu issue.
 
Last edited:

PrincessFrosty

Platinum Member
Feb 13, 2008
2,300
68
91
www.frostyhacks.blogspot.com
The most important tool is your eyes (or the reviewer's eyes). If the site doesn't actually play the games and report on game play, then their results are of lesser importance than those who do. This is where [H], for one, is invaluable.

Using FCAT on a 30sec. section doesn't tell you anything. Unless you are only going to play that one 30sec. interval of the game. Benching different parts of the game will give you different results.

The last I checked the [H] reviews targeted 30fps and since they go for a highly subjective "feel" with their games their reviews are often next to useless. What about people who need 60fps constant, or 120fps?

Their review method has always been superiour for the fraction of users it's suitable for, but it doesn't give the wide array of information that other, more in depth, reviews give out. What they consider to be "max settings" is also subjective, when they have to drop a specific setting to achieve their playable frame rate, the settings they pick to lower affect the outcome, and since we don't all agree on what should go first when we have to sacrifice IQ it just becomes one massive mess.

I drifted away from their method of reviewing ages ago, I think their 30fps baseline is laughable, I've always expected 60fps and now with a 120hz monitor I tend to expect closer to 120fps, their reviews don't tell me what hardware is capable of that, how frame rate scales at high end.
 

Erenhardt

Diamond Member
Dec 1, 2012
3,251
105
101
Do you have any examples of sites using the FRAPS number from FCAT or are you assuming this?

Every single review putting FCAT and FRAPS on the same graph. It is impossible to do this simultaneously and putting 2 graphs from two different playsessions will be misleading due to changes in the scene. If it is a timed demo, "AKA benchmark" it is even more useless because the results are far from actual gameplay.

IMHO this smells like P4 is back again.
 
Last edited:

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
The last I checked the [H] reviews targeted 30fps and since they go for a highly subjective "feel" with their games their reviews are often next to useless. What about people who need 60fps constant, or 120fps?

Their review method has always been superiour for the fraction of users it's suitable for, but it doesn't give the wide array of information that other, more in depth, reviews give out. What they consider to be "max settings" is also subjective, when they have to drop a specific setting to achieve their playable frame rate, the settings they pick to lower affect the outcome, and since we don't all agree on what should go first when we have to sacrifice IQ it just becomes one massive mess.

I drifted away from their method of reviewing ages ago, I think their 30fps baseline is laughable, I've always expected 60fps and now with a 120hz monitor I tend to expect closer to 120fps, their reviews don't tell me what hardware is capable of that, how frame rate scales at high end.

What sites test playing games maintaining 120fps? Or, even 60fps?
 

Leadbox

Senior member
Oct 25, 2010
744
63
91
FCAT inserts the colours on the frames at the same spot where FRAPS gets its readings so i'm guessing this is how fcat "infers" fraps data (think I read this on pcper)
Some games don't play nice with both fcat and fraps running so two seperate runs are required and thats from techreport
 

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
Spoiler: None

I don't know of any. I'd love to have a site play "Max settings with vsync maintaining 60fps". What are the odds of that though? Sure wouldn't help the manufacturers because they couldn't sell more expensive cards because they run 66fps instead of 62fps with the same settings. Let's face it folks, the reason they send this hardware to the reviewers is to promote it.