Should reviews that don't use FCAT be ignored?

[Siberian]

Now that we have a way of accurately benchmarking gameplay performance with FCAT, reviews that do not use it are incomplete. Nobody watches a FPS meter while they play, but they sure notice stutter while playing. This really needs to be included in every review.

 

[bystander36]

I'm not sure I'd ignore them, but I'd take them with a grain of salt. For the most part, single GPU games don't have much issues anymore, at least with Nvidia and AMD discrete cards. If looking at SLI or CF, I'd definitely want to see FCAT results.

 

[chimaxi83]

Now that we have a way of accurately benchmarking gameplay performance with FCAT, reviews that do not use it are incomplete. Nobody watches a FPS meter while they play, but they sure notice stutter while playing. This really needs to be included in every review.

Yes, ignore them all.... until AMD is better at it than Nvidia

/sarcasm

No. Some sites do it, other sites don't. Don't like the review method of a particular site? Don't give them traffic. See how easy that was? There are things that can be appreciated about each individual review.

What should be ignored are trolls.

 

[Enigmoid]

For single gpu FCAT = FRAPS.

Multi GPU is different.

 

[bystander36]

Yes, ignore them all.... until AMD is better at it than Nvidia

/sarcasm

No. Some sites do it, other sites don't. Don't like the review method of a particular site? Don't give them traffic. See how easy that was? There are things that can be appreciated about each individual review.

What should be ignored are trolls.

Isn't that the point?

 

[gorobei]

FCAT isnt the end all be all. you need a honking big ssd drive array to store all the raw frames in order to truly analyze the card output and some additional tools to actually sort out what is being delivered. it can help in figuring out how to get rid of runt frames and other issues, but a single number metric of frame interval isnt necessarily better than fps.
most review sites are never going to be able to afford the ssd array or the time it takes to process all the frames to get full measure.

in order to know whether you are getting any real benefit or better performance, you need to get into the game engine (animation cycle and IO). those tools dont exist yet.

the guys at techreport have been going over this in their podcasts, and they indicate that we are really only at the beginning of going beyond fps or frame latency.

 

[3DVagabond]

The most important tool is your eyes (or the reviewer's eyes). If the site doesn't actually play the games and report on game play, then their results are of lesser importance than those who do. This is where [H], for one, is invaluable.

Using FCAT on a 30sec. section doesn't tell you anything. Unless you are only going to play that one 30sec. interval of the game. Benching different parts of the game will give you different results.

 

[AnandThenMan]

I was about to post the same thing, you can stare at data all day and learn nothing. From some reviews I've read (never mind posts in the various forums), my gaming experience should be a miserable, even unbearable.

But in reality it's downright fantastic. And I have to try really hard to find meaningful differences between my AMD and Nvidia rigs. Sometimes one will handle a particular game better, others the reverse is true. But rarely (honestly I don't think ever) have I said, this is just no good I can't take the stutter or runt frames or whatever nonsense jargon I see flying around.

And I swear some people never actually play games at all, I see some usernames all over the Internet going on about the same talking points. Why you no actually enjoy your expensive hardware? You're using it to type on a keyboard all day, is that really the best use of the computing power you have?

 

[Final8ty]

I was about to post the same thing, you can stare at data all day and learn nothing. From some reviews I've read (never mind posts in the various forums), my gaming experience should be a miserable, even unbearable.

But in reality it's downright fantastic. And I have to try really hard to find meaningful differences between my AMD and Nvidia rigs. Sometimes one will handle a particular game better, others the reverse is true. But rarely (honestly I don't think ever) have I said, this is just no good I can't take the stutter or runt frames or whatever nonsense jargon I see flying around.

And I swear some people never actually play games at all, I see some usernames all over the Internet going on about the same talking points. Why you no actually enjoy your expensive hardware? You're using it to type on a keyboard all day, is that really the best use of the computing power you have?

+1

 

[blastingcap]

They need to measure input lag, too, because if the input lag sucks due to some sort of software metering, that isn't as good as if they have some other way of smoothing frametimes, perhaps on the hardware.

 

[ocre]

Kyle at Hardocp has been trying to give his experience for a long long time. I actually didnt like his reviews long ago (mostly because of the very limited data in game sets) but over the past few years i have started growing fond of his style.

In his reviews that he hasnt used FCAT, would you ignore them?

kyle does things that really are rare these days, like actually playing the game instead of just canned benchmarks. I mean really, what good is a FCAT review full of canned benchmarks? I know their are a few reviews out their that play repeatable scenes if some cases here and there but kyle has been trying to give us a more detailed review of the experience.

I wouldnt disregard hardocp because they dont use FCAT. Reallym like others have said, read the reviews and if you feel like they are bringing you valuable information then cool..... those that dont, dont read them.

 

[Saffron]

We've trusted review sites without it before, why shouldn't we now? All because of FCAT? If one review is saying a particular card has bad stutter then it most likely IS NOT true, but if all reviews on this particular card are saying it then it's probably true. That's all there is to it. I consider FCAT a side dish, nothing too important but nice to have.

 

[BrightCandle]

FCAT shows up a particular problem with a particular configuration of cards. Other than that its not a particularly useful measure. We didn't know before this was released that AMD's crossfire had problems pacing the frames evenly, we just thought it very likely from the fraps frame times. It turned out what fraps was showing was basically what was happening and in reality the impact was to effectively halve the reported frames per secod. Nowadays we can determine that purely from the fraps data.

So I wouldn't argue that FCAT is a necessary measure, in the end it just validates the GPU is working as its meant to, but it doesn't prove that the experience is smooth. Because it tests frame delivery rather than frame contents it doesn't actually measure smoothness of the game, it measures smoothness of the GPU delivering frames. The game could still be giving the GPU unevenly spaced frames, and its fraps that helps to measure that (and my new tool yet to be released does it much better). So I wouldn't ignore a site that doesn't use FCAT but if they aren't using frame times, they are using fps measured continuously its a bit of worry, and a single FPS average for an entire run is really bad.

But its not really a review in isolation, if you read all the reviews around the internet you can take the FCAT/FRAPS data and combine it with the FPS over time and FPS averages and be fairly certain that some combination of kit is likely to work well without problems based on all the different types and styles of review. I think that is more important than it has ever been.

 

[Imouto]

You should ignore all reviews altogether. As for me I just read interesting reviews and use graphs to prove points. Graphs by themselves are a pretty useless metric when you don't give them any meaningful context. I find FCAT just another way to show the reader useless data. It's really amusing how some web sites wear out the latest trendy metric at hand to prove a point that the user won't notice at all.

And now seriously, folks go gaga over graphs with 1 ms or less variance and call that a win or lose for their favorite/hated brand. That's way worse than calling a 1 or 2 fps average advantage a win or lose and so goes one with other metrics like power consumption.

Review graphs are stupid. That's why Kyle is doing a way better job than other reviewers like Wizzard at TPU that don't give a flying duck about his articles. Wizzard's data is useful to argue over the interwebs but if you're in the market for a new card it's pretty much useless, a damn void ~32 pages article. Same goes for PCPer or TR that don't give a thing about the actual gameplay, they will slow videos down and show you all kind of useless data and metrics. Play? Come again?

What's good for page hits ain't good an actual purchase. I like reviewers that actually play (that's what graphics cards are for... mainly!) with their gear. Gaming isn't rocket science, really.

It's like the weather news really.

- The weather man over the TV is telling you what is going on with a pretty map behind and some numbers you don't care about.
- The weather man over the radio is telling you what is going on without any visual aid and overwhelming you with numbers. You actually get it.
- You in the middle of the street can tell what's going on without a thermometer, barometer. "Damn it's hot!", "Man, I'm freezing!" or "What a pleasant weather" are usually enough.

A damn graphics reviewer would tell you how much 0.1 Cº matters or why a single cloud ruins your blue sky.

 

[Lepton87]

The most important tool is your eyes (or the reviewer's eyes). If the site doesn't actually play the games and report on game play, then their results are of lesser importance than those who do. This is where [H], for one, is invaluable.

Using FCAT on a 30sec. section doesn't tell you anything. Unless you are only going to play that one 30sec. interval of the game. Benching different parts of the game will give you different results.

I agree with this, if you enjoy the game and it feels smooth to YOUR eyes what do you care if the frames are unevenly paced, that there it some MS and a couple of runt frames etc. The problem with this approach is that you have test the hardware yourself. As for [H] what they deem playable is a slide-show to me, maybe I exaggerated a bit, but I'd rather sacrifice some IQ that during game play I don't even notice to have 60fps and not 40-30 and H finds that range perfectly fine so to me their subjective feelings are nearly useless. Maybe If I capped fps at 30 I could get used to it like consoles players do but I doubt that it would be as good of an experience as at 60fps.

 

[3DVagabond]

I agree with this, if you enjoy the game and it feels smooth to YOUR eyes what do you care if the frames are unevenly paced, that there it some MS and a couple of runt frames etc. The problem with this approach is that you have test the hardware yourself. As for [H] what they deem playable is a slide-show to me, maybe I exaggerated a bit, but I'd rather sacrifice some IQ that during game play I don't even notice to have 60fps and not 40-30 and H finds that range perfectly fine so to me their subjective feelings are nearly useless. Maybe If I capped fps at 30 I could get used to it like consoles players do but I doubt that it would be as good of an experience as at 60fps.

I too prefer 60fps (vsync actually). [H] does give us a comparable metric with minimum playable rates, though. I know I'll have to reduce some settings, or will need more GPU power. My preferences are often different too. I'll turn shadows, or AO down before I reduce AA, for instance. If one card (or dual cards) needs higher fps to appear as smooth as another though, that tells me everything I need to know about frametimes, MS, etc. FCAT was designed as a diagnostic tool, it should be extremely valuable for that. Without it being open source, it's also difficult to know whether it's totally fair between the 2 brands.

 

[BrightCandle]

The point of all of these more detailed tests is to objectively measure the problem that our eyes detect. We are missing the right way to measure (micro)stutter simply and we are missing the actual measure of human perception for most people in that but it doesn't mean it isn't useful and the only that matters is our eyes, because just like FPS we are creating a proxy, a number that describes how the motion works.

The comparison is the fps number where we now know we need 30 fps before games feel smooth to most people. So we have the subjective measure and we have the human perception number.

Lets say the difference between frame times is the measure (one frame is 16ms, the next is 14ms, difference = 2ms) then the most humans perception figure is somewhere around 2-8ms. Nvidia said 2ms as a lower bound, the data from these forums suggests around 8ms, but its somewhere in there.

When it comes FCAT we have the scan line length, which we translate back to frame times and ultimately FPS, but for the scan line length we need a measure of variation between them that is acceptable. To call 20 lines a runt but 21 is fine probably isn't the right measure. We need a way to find the right number.

But these subjective measures are combined with the human perception of the actual motion to come to these conclusions. Ultimately what they allow us to do is look at graph and understand which product is better on some measure without having to see it ourselves, because we know that it achieves some minimum that everyone needs from it to feel comfortable with the motion.

These new measures are useful, its hard to argue against them considering what it is they do and what it is they tell us. Its just that the common understanding, the simplistic picture of it isn't formed fully yet, but these measures or derivatives of them will be how we measure all graphics performance in the future without a doubt. FPS is far from sufficient.

 

[Keysplayr]

I think this should be viewed on a person by person basis. For example, look at post #3 in this thread for a glance at a radical position on the subject. Where the person believes the whole thing is a sham and only important to some people because it makes their favorite company look better than another. I wonder how long this will be perpetuated. That persons argument, "Yes, ignore them all.... until AMD is better at it than Nvidia"
also applies to the poster himself. What will his tune be if and when AMD surpasses Nvidia at the gameplay experience level? Contrary to his accusations, I can tell you that mine won't change. I believe this method for testing is very important regardless who is doing better in the gameplay experience.
We've turned a corner in the way things are done, this isn't a "trendy" thing that will go away anytime soon BECAUSE this gives the impression of the actual experience instead of simply looking at fps due to the discovery in no small part by H, TR, PCPer and others that the actual gaming experience is indeed more important than raw FPS.
I couldn't agree more. FCAT is what is used, in part, now for this sort of testing. Other tools will emerge that improve upon or expand these ideas.
Should sites that don't use FCAT be ignored? This solely depends on the individual reading the reviews. Third poster in this thread is a good example of how NOT to embrace new technologies or methodologies for benchmark testing.

 

[Zanovar]

The more data the better imo,and if that data provides better gaming experiences for me.im all for it.keeps them on their toes*winks*.

 

[BallaTheFeared]

Without it being open source, it's also difficult to know whether it's totally fair between the 2 brands.

The important "capture" part is open source, MSI AfterBurner currently employs it.

FCAT itself has no idea where the output comes from, be it AMD or nVidia.

 

[Lepton87]

I don't know how this became about AMD vs NV. Isn't FCAT a program written by NV? It's not unusual for programs to prefer hardware, (cough ICC cough). So we should approach with caution. Before anyone accuses me of bias I'd say the exact same thing if it were an amd's program.
Update

The important "capture" part is open source, MSI AfterBurner currently employs it.

FCAT itself has no idea where the output comes from, be it AMD or nVidia.

That changes things, it means we can trust its numbers.

 

[Leadbox]

Anyone else notice how many times the Techpowerup benchies have been linked in the last couple of weeks? Does the op ignore those?

 

[BallaTheFeared]

I don't know how this became about AMD vs NV. Isn't FCAT a program written by NV? It's not unusual for programs to prefer hardware, (cough ICC cough). So we should approach with caution. Before anyone accuses me of bias I'd say the exact same thing if it were an amd's program.

Why don't you educate yourself on what FCAT actually is then get back to us?

"If you sail out too far, you'll fall off the edge of the world!"

 

[Lepton87]

Why don't you educate yourself on what FCAT actually is then get back to us?

"If you sail out too far, you'll fall off the edge of the world!"

I already edited my post, I don't have the time to learn about everything. What's wrong with learning something from the forum?

 

[3DVagabond]

That changes things, it means we can trust its numbers.

Who decided that 20 lines is a runt frame but 21 lines is a complete frame? nVidia, did. What would happen if it was set at 100 lines, still only ~10% of a frame, or 200 lines? Would the results comparing the 2 brands still be the same? When one company sets the parameters how do we know that they aren't designed to make their products appear superior?

I'm not saying to disregard FCAT. What I'm saying is don't disregard your eyes either.