[TECH Report] As the second turns: the web digests our game testing methods

[Lonbjerg]

You are saying the same thing AMD has been saying ever since Intel slaughtered them.

Hey folks, don't look at meaningless benchmarks.
Look at those average gamers? Lets blind test them, and use their observations instead of numbers.

http://amdfx.blogspot.com/2012/04/mobile-trinity-blind-test-amd-clear.html
http://legitreviews.com/article/1838/1/

Frankly I think that's a bunch of stuff, and PR at it's worst - doing damage control insteaf of fixing things.
Because numbers are meaningless only if you pull them out of your ass. Which is hardly the case here.

Frame times coupled with frame variations is what defines your gameplay experience.
In infinitely more objective way, than observations(or lack of) done by Joe, Marry and d3L74#w4rri0R.
And arguably in a much more complete way than FPS alone.

Average FPS(average latency) numbers, done by "computer output graph" has been fine for ages.
So why all of sudden these new numbers: FPS, FPS variations, latency spikes(>50ms) are somehow less worthy, more suspicious, and should be confirmed by "blind testers"?

You never asked for FPS blind test confirmations, or have you?

It's "damage control"...nothing to see here...move along...insufficient data..."I can't see it, so i's not there".

Didn't you get the memo

 

[Rikard]

So why all of sudden these new numbers: FPS, FPS variations, latency spikes(>50ms) are somehow less worthy, more suspicious, and should be confirmed by "blind testers"?

You still do not get it, do you? You cannot take the human out of the equation since the experience, not numbers, are what ultimately matters. There are a near infinite quantifiable differences between different hardware configurations that you could measure and report. That does not mean that they are all observable by a human being. For example, does it really matter if you have 500 FPS or 600 FPS in your favorite game? 25 compared to 30 FPS? The difference between those scenarios is whether they are observable by humans. I am all for improving numbers even when the improvements cannot be perceived, but it is wrong to attribute a large importance to such potential improvements.

 

[Jaydip]

Did Nvidia ever fix the major stuttering issues on the 5 series cards? Didn't you own one?

I owned a 580 and never experienced ms but if it exists now I want NV to fix it.Just because you can't see it doesn't invalidate the findings.My stance has been pretty clear.

 

[Jaydip]

Who is denying it?

Maybe my tinted glasses then :hmm:

 

[Jaydip]

No one is denying that a computer can output a graph that sees odd frame latencies.

IIRC that's the purpose of review in the first place.If blind testing would be the "one and only" metric for performance we can safely ignore review sites going forward.

 

[Rikard]

IIRC that's the purpose of review in the first place.If blind testing would be the "one and only" metric for performance we can safely ignore review sites going forward.

You keep missing the point. What BrightCandle and I did was to try to translate subjective observations of "No observable microstutter", "Acceptable microstutter" and "Not acceptable microstutter" into quantities derived from frame times. I have a pretty good idea of at least where I would draw the lines between those based on frame times. This is no different from those saying that FPS above 60 feels smooth, but individual sensitivity does come into play. If we could have review sites condensing the frame times into a simple number like they do for average FPS costumers can make informed decisions on whether a difference is meaningful or not. Or are you deliberately trying to not understand?

 

[Jaydip]

You keep missing the point. What BrightCandle and I did was to try to translate subjective observations of "No observable microstutter", "Acceptable microstutter" and "Not acceptable microstutter" into quantities derived from frame times. I have a pretty good idea of at least where I would draw the lines between those based on frame times. This is no different from those saying that FPS above 60 feels smooth, but individual sensitivity does come into play. If we could have review sites condensing the frame times into a simple number like they do for average FPS costumers can make informed decisions on whether a difference is meaningful or not. Or are you deliberately trying to not understand?

Lets see

1.The purpose of TR review was to bring a Quantitative approach to test ms.They didn't want to cloud subjective observations with their own "subjective" decisions.You present data as is and you should leave it to readers to interpret them.It is not your job to interpret that for readers.

2.Define "No observable microstutter", "Acceptable microstutter" and "Not acceptable microstutter".Don't create new terms to make it ever more Qualitative.

 

[Imouto]

2.Define "No observable microstutter", "Acceptable microstutter" and "Not acceptable microstutter".Don't create new terms to make it ever more Qualitative.

http://techreport.com/review/23981/radeon-hd-7950-vs-geforce-gtx-660-ti-revisited

4 benches = 50 ms threshold
2 benches = 33.3 ms threshold
1 bench = 16.7 ms threshold

 

[Rikard]

Lets see

1.The purpose of TR review was to bring a Quantitative approach to test ms.They didn't want to cloud subjective observations with their own "subjective" decisions.You present data as is and you should leave it to readers to interpret them.It is not your job to interpret that for readers.

They did well in their attempt at remaining objective. But since it is not a standard test that everybody can read the graph and be able to tell whether a result is going to affect their experience, a couple of blind tests are in order, don't you think?

2.Define "No observable microstutter", "Acceptable microstutter" and "Not acceptable microstutter". Don't create new terms to make it ever more Qualitative.

I am just trying to reach out to you by speaking a language you understand. These are not definitions I worked with, but since you ask so nicely:
"No observable microstutter" - No matter how much I stare at the screen, neither I nor other people that look at the same screen, can detect any unsmooth behaviour visually.
"Acceptable microstutter" - Some unsmooth behaviour can be observed if one deliberately looks for it, but it does not distract from gaming (or what ever you are doing)
"Not acceptable microstutter" - Unsmooth behaviour is disturbing and distracts from the general experience. Typically observed even when not deliberately looking for it.
Clearer now?

 

[Lonbjerg]

They did well in their attempt at remaining objective. But since it is not a standard test that everybody can read the graph and be able to tell whether a result is going to affect their experience, a couple of blind tests are in order, don't you think?


I am just trying to reach out to you by speaking a language you understand. These are not definitions I worked with, but since you ask so nicely:
"No observable microstutter" - No matter how much I stare at the screen, neither I nor other people that look at the same screen, can detect any unsmooth behaviour visually.
"Acceptable microstutter" - Some unsmooth behaviour can be observed if one deliberately looks for it, but it does not distract from gaming (or what ever you are doing)
"Not acceptable microstutter" - Unsmooth behaviour is disturbing and distracts from the general experience. Typically observed even when not deliberately looking for it.
Clearer now?

I am reminded of the CTR monitors days.

Some people had no problems with 60Hz...but most people did.
Most people were fine with 85HZ...I need 100Hz...or I can litterally see the screen flicker.

I am reminded of when shimmering was discovered.
Many people claimed they could not see it and it was all in the mind.
Today we know better.

I am remined of when people first started talking about microstutter in multi-GPU.
Many people dismissed that too.
Today we know better.

This is the same thing.
Some people (reasons left to reader to identify) now have the same stancece in reguards to single GPU microstutter.
Tehy either deny it, talk it down...or point to OTHER cases of microsttuer...ans if that makes it "go away" LOL

In the future these people will look even more silly than they do today.

 

[Rikard]

I am reminded of the CTR monitors days.

Some people had no problems with 60Hz...but most people did.
Most people were fine with 85HZ...I need 100Hz...or I can litterally see the screen flicker.

I am reminded of when shimmering was discovered.
Many people claimed they could not see it and it was all in the mind.
Today we know better.

I am remined of when people first started talking about microstutter in multi-GPU.
Many people dismissed that too.
Today we know better.

This is the same thing.
Some people (reasons left to reader to identify) now have the same stancece in reguards to single GPU microstutter.
Tehy either deny it, talk it down...or point to OTHER cases of microsttuer...ans if that makes it "go away" LOL

In the future these people will look even more silly than they do today.

Your message started so well, but you fell back into your trolling stance towards the end. A pity really.

With CRT I needed 85 Hz, which I think is pretty main stream. You needed 100 Hz. Like I said, we have different sensitivities, just like I cannot stand poor audio but some people are happy as long as there is sound coming out of the speakers. That is something we must accept, and is the reason for reviews listing objective, measurable numbers. What I have been trying to tell you guys is what such numbers mean to the user. I cannot tell what you will experience, but I can tell what I experience for all it is worth. You just have to do your own homework.

But then you go on to say that people are denying it. As with regards to other sources of microstuttering, yes you can produce stuttering by for example HDD activity, but I am not sure of microstuttering although there are reports of that too. And yes, in the only case I have perceivable microstuttering I can make it "go away" by reducing anti-aliasing. I suggest you read the thread again and you would not sound as silly. And with regards to talking it down, it should be talked down, since really it is not a big issue compared to many other things we have such as component life expectancy, resonances in the fan noise, RMA rates etc etc.

But who am I kidding, I am talking to a wall here, you have apparently already decided what the rest of us think.

 

[Homeles]

He's decided what everyone else thinks based on the very telling comments that have been made in this thread.

 

[Jaydip]

http://techreport.com/review/23981/radeon-hd-7950-vs-geforce-gtx-660-ti-revisited

4 benches = 50 ms threshold
2 benches = 33.3 ms threshold
1 bench = 16.7 ms threshold

So you defined that for yourself cool :hmm:

 

[MrK6]

I am reminded of the CTR monitors days.

Some people had no problems with 60Hz...but most people did.
Most people were fine with 85HZ...I need 100Hz...or I can litterally see the screen flicker.

I am reminded of when shimmering was discovered.
Many people claimed they could not see it and it was all in the mind.
Today we know better.

I am remined of when people first started talking about microstutter in multi-GPU.
Many people dismissed that too.
Today we know better.

This is the same thing.
Some people (reasons left to reader to identify) now have the same stancece in reguards to single GPU microstutter.
Tehy either deny it, talk it down...or point to OTHER cases of microsttuer...ans if that makes it "go away" LOL

In the future these people will look even more silly than they do today.

All you're doing is insisting your opinions are Gospel, which, news flash, they aren't. CRT monitor refresh rate is a taste preference, using bombastic rhetoric and saying "most people agree with me" shows just how poor of a point you're making (pseduo-argumentum ad populum, go look it up). Same with this "we know better" nonsense, it's your opinion, state it as so. If you can't make a point without calling in backup, it's a poor point. People's perceptions are different, and because someone isn't bothered by something or doesn't see it as big an issue as you do doesn't mean they're wrong. Try posting with information instead of hyperbole.

 

[Jaydip]

They did well in their attempt at remaining objective. But since it is not a standard test that everybody can read the graph and be able to tell whether a result is going to affect their experience, a couple of blind tests are in order, don't you think?


I am just trying to reach out to you by speaking a language you understand. These are not definitions I worked with, but since you ask so nicely:
"No observable microstutter" - No matter how much I stare at the screen, neither I nor other people that look at the same screen, can detect any unsmooth behaviour visually.
"Acceptable microstutter" - Some unsmooth behaviour can be observed if one deliberately looks for it, but it does not distract from gaming (or what ever you are doing)
"Not acceptable microstutter" - Unsmooth behaviour is disturbing and distracts from the general experience. Typically observed even when not deliberately looking for it.
Clearer now?

This is the stupidest thing I keep hearing in this thread.Blind tests what for? honestly I feel many member of this forum would do pretty well in medieval age.We have developed tools not to depend on this baseless ideas for a long time now.From your definition of ms it is pretty clear you don't understand that the world doesn't revolve around you.We don't need anymore mindless subjective definitions Clearer now?

 

[Homeles]

All you're doing is insisting your opinions are Gospel, which, news flash, they aren't. CRT monitor refresh rate is a taste preference, using bombastic rhetoric and saying "most people agree with me" shows just how poor of a point you're making (pseduo-argumentum ad populum, go look it up). Same with this "we know better" nonsense, it's your opinion, state it as so. If you can't make a point without calling in backup, it's a poor point. People's perceptions are different, and because someone isn't bothered by something or doesn't see it as big an issue as you do doesn't mean they're wrong. Try posting with information instead of hyperbole.

Way to ignore the good points that he brought up.

See: shimmering, multi-GPU microstutter. Things that were denied, despite being objectively true.

 

[Rikard]

This is the stupidest thing I keep hearing in this thread.Blind tests what for? honestly I feel many member of this forum would do pretty well in medieval age.We have developed tools not to depend on this baseless ideas for a long time now.From your definition of ms it is pretty clear you don't understand that the world doesn't revolve around you.We don't need anymore mindless subjective definitions Clearer now?

Fine, if your scientific method is so refined, please let us know how you want to account for the human factor?

 

[Homeles]

Fine, if your scientific method is so refined, please let us know how you want to account for the human factor?

Like I asked earlier, who do we blind test? The general public? Gamers? Enthusiasts? The guys that claim they can see it?

I'd argue that your average person won't see it, because they're ignorant of what they're looking for.

 

[MrK6]

Way to ignore the good points that he brought up.

See: shimmering, multi-GPU microstutter. Things that were denied, despite being objectively true.

It's still the same point. Shimmering bothered some people as our texturing got better, it was noticed, reviews highlighted it, AF algorithms were improved. Microstuttering was noticed in AFR multi-GPU setups, manufacturers started, and still are, working on it. There are people out there who never were, and still aren't, bothered by any of these.

However on the other side, there were idiots who went on and on about how CRT's below 85Hz ruined gaming for them and everyone on their block, everyone hates PC gamers, hardware companies suck, and the world was going to end. The same idiots came out when shimmering was noticed and also with microstutter, and you can sprinkle in a little fanboyism to make them more obnoxious. And here they are again with the same nonsense. As technology improves we will always find new ways to improve it, that's progress. Being a boorish and trite pseudo-elitist about it doesn't help anything.

 

[Jaydip]

Fine, if your scientific method is so refined, please let us know how you want to account for the human factor?

I already discussed that with blastingcap in "*Ryan Smith's*' thread.You can check it out.

 

[Homeles]

It's still the same point. Shimmering bothered some people as our texturing got better, it was noticed, reviews highlighted it, AF algorithms were improved. Microstuttering was noticed in AFR multi-GPU setups, manufacturers started, and still are, working on it. There are people out there who never were, and still aren't, bothered by any of these.

However on the other side, there were idiots who went on and on about how CRT's below 85Hz ruined gaming for them and everyone on their block, everyone hates PC gamers, hardware companies suck, and the world was going to end. The same idiots came out when shimmering was noticed and also with microstutter, and you can sprinkle in a little fanboyism to make them more obnoxious. And here they are again with the same nonsense. As technology improves we will always find new ways to improve it, that's progress. Being a boorish and trite pseudo-elitist about it doesn't help anything.

Neither does getting super defensive. You're even worse.

 

[Imouto]

So you defined that for yourself cool :hmm:

I'm just pointing out the low standards this huge fuss is all about. TR review was journalism at its finest.

- Face an OC card vs an almost stock card.
- Change OS without any single reason.
- Change all your bench suite except for 1 title.
- Change the place where the bench takes place in that title.
- Change thresholds just because you can. For the lulz.
- Change resolution whenever you want. For the shire.
- Pick an unplayable config in Skyrim (vsync messing with physics). Check.
- Pick an unplayable config in Sleeping Dogs for a staggering 3x average FPS. Sorry I can't hear you over it's smoothness.
- Dramatize differences till every single fangirl screams her soul out even if it is minimal like in Medal of Honor bench.
- Don't OC both cards just for the sake of science.

Drama^8

 

[Homeles]

I'm just pointing out the low standards this huge fuss is all about. TR review was journalism at its finest.

- Face an OC card vs an almost stock card.
- Change OS without any single reason.
- Change all your bench suite except for 1 title.
- Change the place where the bench takes place in that title.
- Change thresholds just because you can. For the lulz.
- Change resolution whenever you want. For the shire.
- Pick an unplayable config in Skyrim (vsync messing with physics). Check.
- Pick an unplayable config in Sleeping Dogs for an staggering 3x average FPS. Sorry I can't hear you over it's smoothness.
- Dramatize differences till every single fangirl screams her soul out even if it is minimal like in Medal of Honor bench.
- Don't OC both cards just for the sake of science.

Drama^8

Despite all that, they were still right. Argument nullified.

 

[Jaydip]

I'm just pointing out the low standards this huge fuss is all about. TR review was journalism at its finest.

- Face an OC card vs an almost stock card.
- Change OS without any single reason.
- Change all your bench suite except for 1 title.
- Change the place where the bench takes place in that title.
- Change thresholds just because you can. For the lulz.
- Change resolution whenever you want. For the shire.
- Pick an unplayable config in Skyrim (vsync messing with physics). Check.
- Pick an unplayable config in Sleeping Dogs for a staggering 3x average FPS. Sorry I can't hear you over it's smoothness.
- Dramatize differences till every single fangirl screams her soul out even if it is minimal like in Medal of Honor bench.
- Don't OC both cards just for the sake of science.

Drama^8

Dude I wasn't talking about TR's review methods rather than its findings.Despite that AMD considered their findings to be accurate.Anyway my point was not to cheer lead TR's review but honestly it seems they are only among few sites who can offer more insight with their method.No offence but at this moment AT is lacking in GPU review department compared to many other sites.

 

[Homeles]

No offence but at this moment AT is lacking in GPU review department compared to many other sites.

Definitely. Ryan Smith is a brilliant guy, who really knows his stuff when it comes to GPUs. I love reading about the nitty-grtty architectural details. But there are serious limitations to the usefulness of the data that they collect. It'd really help if they weren't using a closed test bed, for starters.