Asking Ryan Smith of AT if a special examination could be done?

[3DVagabond]

I'd like someone to do double blind testing of cards to find out if anyone can actually see a difference. Then, if they can, take measurements to see what it is that correlates to what is observed. Taking measurements and then trying to assign an effect to it is backwards and, with the way the human psyche works, will have people seeing differences where none exist.

Example, If you tell someone that card A measures smoother than card B, and then say, "Look at card A and B and tell me which is smoother." What do you suppose a majority of the subjects will see?

 

[boxleitnerb]

This should be investigated mainly for SLI/CF. For single GPU setups, it is not that relevant.

 

[Silverforce11]

Posted this in other thread, but since you guys are testing..

Radeon Pro testing, single card. Civilization V. One turn gameplay after initial map loading ~30s to stabilize.

vsync off ingame

vsync off ingame, with radeon pro dynamic frame rate control, set at 55 fps.

vsync on ingame

I tested further with vsync on 60, dynamic frame rate set at 59, and it was smooth as butter.

 

[Goatsecks]

Think this is a great idea. I'm not sure why there seems to be a bit of resistance to the idea in this thread?

My friends and I have been in agreement for a long time about how frame rate is not that sure an indicator as to a certain components performance. A high variance in frame rate can manifest itself in unpleasant ways. Input lag *is* a very real problem. Right back from the old CS days we would never limit our frame rates and still would never our frame rates (we now play bf3).

If anand can find quantitative ways of measuring these phenomena (which I am sure they can, this is hardware we are reviewing and not pieces of art) I would love to see the result (especially as I am doing a new build in the new year!).

 

[Keysplayr]

I'd like someone to do double blind testing of cards to find out if anyone can actually see a difference. Then, if they can, take measurements to see what it is that correlates to what is observed. Taking measurements and then trying to assign an effect to it is backwards and, with the way the human psyche works, will have people seeing differences where none exist.

Example, If you tell someone that card A measures smoother than card B, and then say, "Look at card A and B and tell me which is smoother." What do you suppose a majority of the subjects will see?

Excellent idea. A minimum of 10 people must be used though for a nice control.

 

[Keysplayr]

Keys, I like your posts and am happy you are part of these forums, Nvidia focus group or not. I think often you get flak when it is completely not deserved. But I can't help but feel like this is an attempt to get Ryan to write an article that will show Nvidia in a better light than the competition. Just look at it from my perspective, a poster associated with Nvidia trying to have an article written that more or less shows frame rates don't matter, Nvidia's smoothness matters at a time when Nvidia is producing lower frame rates at most every price point (we all know how the internet works, "zomg! Nvidia is smoother despite lower frame rates!" and people will think that this applies to single cards, too).

Maybe I'm 100% wrong here, it is an article I would read, but it seems odd that this request was made when there really isn't too big of a 'war' over this.

First of all, thank you for your comments. I think.
I am glad you are part of this form also.
Next, Isn't too big of a war? Have you seen the other thread?
http://forums.anandtech.com/showthread.php?t=2287359
The discussions over at H and TR? I do believe this is ruffling a bushel of feathers and I, like you, would love to read an article like this NO MATTER THE OUTCOME. If it shows Nvidia in a better light, why do you care? I don't care if it does, or turns out to all be rubbish. Either way, I'd like to know and that seems to be the overwhelming consensus in the posts I'm reading in this thread.
The ONLY reason I started this request was because of H and TR. I didn't just come out of the blue and request a smoothness test at a time when AMD was just ahead in framerates.

I am looking at it from your perspective. And I think this is the "I think often you get flak when it is completely not deserved." situation.
With only my preference for Nvidia and focus group participation, my motives for the thread come into question. I don't have any problem with that whatsoever. As long as it's done tastefully. The way you responded was admirable. But if you scroll back a bit and observe a post like Will Robinson's, you'd see those type of posts are useless and damaging to the forums.

Thanks SS.

 

[Keysplayr]

Oh and thank you for responding, Ryan.

 

[SirPauly]

Whats wrong with an article that shows NV in a better light than AMD if true?....Not like these are AMD forums.......is it?

Let's say an article is offered and adds more evidence of nVidia being smoother? The awareness and attention would probably improve smoothness for both nVidia and AMD moving forward -- as web-sites investigate this.

 

[omeds]

Yeh I'm glad the issue is recieving as much attention as it is in current reviews and message boards, hopefully it means both vendors will take more notice and try harder to improve the situation with MGPU.

 

[KingFatty]

Technical question/curiosity:

There has been mention of getting a high speed *VIDEO* camera to test/look for frames.

However, could there possibly be any way at all to construct a test methodology that produces at least some results, but based on a *STILL* camera (e.g., an SLR or other camera where you can dial in a specific shutter speed)?

Just a shot in the dark, but maybe there is a way that someone like me or others on these forums can use a camera and multiple monitors side-by-side to somehow probe a bit and see if the results are encouraging/going in the right direction? I remember setting up some monitors and taking photos before for testing response rate, it can be a fun exercise...

 

[KCfromNC]

I do believe this is ruffling a bushel of feathers

Could be the subject matter, could be that throwing around guesses and FUD as if it were fact is disruptive no matter what the subject being discussed.

 

[SirPauly]

Yeh I'm glad the issue is recieving as much attention as it is in current reviews and message boards, hopefully it means both vendors will take more notice and try harder to improve the situation with MGPU.

Demo, so right on --------- many are worried about perception but missing the point that awareness as a whole may improve gaming for single and multi-gpu smoothness.

 

[Jaydip]

Technical question/curiosity:

There has been mention of getting a high speed *VIDEO* camera to test/look for frames.

However, could there possibly be any way at all to construct a test methodology that produces at least some results, but based on a *STILL* camera (e.g., an SLR or other camera where you can dial in a specific shutter speed)?

Just a shot in the dark, but maybe there is a way that someone like me or others on these forums can use a camera and multiple monitors side-by-side to somehow probe a bit and see if the results are encouraging/going in the right direction? I remember setting up some monitors and taking photos before for testing response rate, it can be a fun exercise...

Not sure who thought about this stupid "camera" thing where human errors are bound to happen.NV has always offered tools regarding this, "PerfHUD" for cards up to Fermi and "Nsight" for Kepler.Not sure if something similar exists for AMD.Funny thing is no one ever used them except game developers.

 

[Keysplayr]

Could be the subject matter, could be that throwing around guesses and FUD as if it were fact is disruptive no matter what the subject being discussed.

Riddles then? Could be anything else but what it is? If it isn't that then maybe it could be the other if that actually what is being discussed.

 

[MrK6]

Hey look at that: AMD is faster than nvidia, looks like the focus group is out in full force.

 

[HurleyBird]

Let's say an article is offered and adds more evidence of nVidia being smoother?

It would be irrelevant without also measuring input lag. Perfectly smooth multi-GPU rendering is easy if you hold enough frames back and take the time to arrange them evenly before pushing them out to the display. Measuring just smoothness or just input lag gives you nothing. You need to measure both.

 

[96Firebird]

Hey look at that: AMD is faster than nvidia, looks like the focus group is out in full force.

Here, let me point you to a 100+ post thread that stemmed this request for Ryan.

http://forums.anandtech.com/showthread.php?t=2287359

Feel free to troll in that thread, and let the adults discuss the issue at hand here!

 

[MrK6]

Here, let me point you to a 100+ post thread that stemmed this request for Ryan.

http://forums.anandtech.com/showthread.php?t=2287359

Feel free to troll in that thread, and let the adults discuss the issue at hand here!

Hardware frame metering was covered during the release of the GTX 690 in April/May. Impact largely depends on the game: http://techreport.com/review/22890/nvidia-geforce-gtx-690-graphics-card/4

The true enthusiasts have already covered it thoroughly, as opposed to the shills now. Where was the focus group then? Oh right, nvidia wasn't getting hammered in value then.

Called you on your nonsense, but you and keysplayr can go back to consoling each other all "adult-like."

 

[cmdrdredd]

So now fps is all that matters? 4fps faster and "omg nvidia sux" is that it?

I see absolutely nothing wrong with showing that fps might not matter that much when it comes to actual gameplay some of the time and illustrating why this might be and what actual differences there are. Somehow though people construe this as some type of reassurance or marketing ploy. Totally ridiculous. Are you that afraid that it will show that AMD might not actually be a totally better experience 100% of the time? Or are you that against nvidia that there can be nothing positive written about them at all?

 

[blastingcap]

Technical question/curiosity:

There has been mention of getting a high speed *VIDEO* camera to test/look for frames.

However, could there possibly be any way at all to construct a test methodology that produces at least some results, but based on a *STILL* camera (e.g., an SLR or other camera where you can dial in a specific shutter speed)?

Just a shot in the dark, but maybe there is a way that someone like me or others on these forums can use a camera and multiple monitors side-by-side to somehow probe a bit and see if the results are encouraging/going in the right direction? I remember setting up some monitors and taking photos before for testing response rate, it can be a fun exercise...

I specifically mentioned the Nikon 1 and Casio which are cameras first, videocameras second. However the resolution goes to hell when you shoot at 1200 fps, though the resolution is only semi-bad at 400 fps. You can shine a light on the monitor or something to help with the low amount of light available at 1/1200 sec.

I also think that this is the truest measure of end user experience. You can try the interception method described by Ryan but it is possible that anything short of capturing the end user moment neglects something not contemplated.

Not sure who thought about this stupid "camera" thing where human errors are bound to happen.NV has always offered tools regarding this, "PerfHUD" for cards up to Fermi and "Nsight" for Kepler.Not sure if something similar exists for AMD.Funny thing is no one ever used them except game developers.

Nonsense. Everything has human error, including existing fps charts where people can mislabel the information, for instance.

My camera suggestion does not have horribad human error potential or anything. You start recording before the start of a run and stop after the stop of a run. Then you have thousands of frames of raw material to work with and can get actual end-user frame times to the closest 1/1200 sec or whatever you decide to use. If you use Vsync it'd be obvious when a frame switched, and even if not in Vsync, most frame should be closer to none or done; you can round up the half-frames in either direction, but those will be infrequent. It would be a painstaking process for sure, but would settle things for sure. Does your software function like FRAPS? If so, read the second post in this thread for why your "just use this other software" idea is insufficient when discussing actual end-user experiences. I suspect that your "brilliant" insight is suffers from the same or similar problem as FRAPs, because otherwise TechReport would have reported on it already.

You know what does have a lot more human error possibility is the suggestion of control group vs experimental in a blind taste test.

 

[MrK6]

So now fps is all that matters? 4fps faster and "omg nvidia sux" is that it?

I see absolutely nothing wrong with showing that fps might not matter that much when it comes to actual gameplay some of the time and illustrating why this might be and what actual differences there are. Somehow though people construe this as some type of reassurance or marketing ploy. Totally ridiculous. Are you that afraid that it will show that AMD might not actually be a totally better experience 100% of the time? Or are you that against nvidia that there can be nothing positive written about them at all?

Neither, and where did I ever say that? Take a look at my posting history and then take your foot out of your mouth. I rarely suggest multi-GPU and when I do, it's almost always SLI and not CF. Hardware frame metering is a step in the right direction, as it shows companies are addressing the issues of AFR, including microstutter. However, the experience is completely user-dependent and you'll have users of both SLI and CF technologies stating they have no problems/great experiences. However this was all covered MONTHS ago and the only reason it's being brought up again is because the green team needs something to throw at AMD's current superiority (which could change next month, but forget logic, right? )

Ryan Smith said it best:

In the meantime however I'm personally satisfied with both AMD and NVIDIA in this regard; in my own personal experience I don't believe either of them to be notably worse than the other when it comes to single-GPU configurations. At the same time this community has a terrible habit of making mountains out of molehills, so if there's a "big war" brewing then I fear you guys might be taking this whole subject a bit too seriously. Spend less time looking at charts and more time playing video games, it's not like there's a shortage of good action games this year.

And I completely agree. A few people here are trying to make a mountain out of a molehill for the sake of sour grapes.

Also I like that Mr. Smith is looking forward to further investigations with better recording technology in hand, as I'm a big proponent of doing great research with great tools. Good luck to the AT crew and I'm looking forward to whatever they venture to write-up.

 

[cmdrdredd]

Neither, and where did I ever say that? Take a look at my posting history and then take your foot out of your mouth. I rarely suggest multi-GPU and when I do, it's almost always SLI and not CF. Hardware frame metering is a step in the right direction, as it shows companies are addressing the issues of AFR, including microstutter. However, the experience is completely user-dependent and you'll have users of both SLI and CF technologies stating they have no problems/great experiences. However this was all covered MONTHS ago and the only reason it's being brought up again is because the green team needs something to throw at AMD's current superiority (which could change next month, but forget logic, right? )

Ryan Smith said it best:

And I completely agree. A few people here are trying to make a mountain out of a molehill for the sake of sour grapes.

Also I like that Mr. Smith is looking forward to further investigations with better recording technology in hand, as I'm a big proponent of doing great research with great tools. Good luck to the AT crew and I'm looking forward to whatever they venture to write-up.

So again you are saying this is reassurance since nvidia is 4 or 5 fps slower in whatever game. There are threads around that have a lot of conjecture and chest thumping and no concrete facts. To put this to the test would quiet some of this so there is no reason to be against testing this thoroughly.

You said this is only an issue because nvidia is slower. Your words dude.

 

[Jaydip]

I specifically mentioned the Nikon 1 and Casio which are cameras first, videocameras second. However the resolution goes to hell when you shoot at 1200 fps, though the resolution is only semi-bad at 400 fps. You can shine a light on the monitor or something to help with the low amount of light available at 1/1200 sec.

I also think that this is the truest measure of end user experience. You can try the interception method described by Ryan but it is possible that anything short of capturing the end user moment neglects something not contemplated.



Nonsense. Everything has human error, including existing fps charts where people can mislabel the information, for instance.

My camera suggestion does not have horribad human error potential or anything. You start recording before the start of a run and stop after the stop of a run. Then you have thousands of frames of raw material to work with and can get actual end-user frame times to the closest 1/1200 sec or whatever you decide to use. If you use Vsync it'd be obvious when a frame switched, and even if not in Vsync, most frame should be closer to none or done; you can round up the half-frames in either direction, but those will be infrequent. It would be a painstaking process for sure, but would settle things for sure. Does your software function like FRAPS? If so, read the second post in this thread for why your "just use this other software" idea is insufficient when discussing actual end-user experiences. I suspect that your "brilliant" insight is suffers from the same or similar problem as FRAPs, because otherwise TechReport would have reported on it already.

You know what does have a lot more human error possibility is the suggestion of control group vs experimental in a blind taste test.

Wrong that is one of the reasons we have automated tools to minimize that.Your process is clumsy and not repeatable at all.The software I mentioned isn't fraps, they are created for game developers.It is a real time performance analyzer.

 

[Lepton87]

Just get high speed camera an compare 2-way CF, 3-way CF and 4 -way CF with radeon pro and without. Do the same with NV cards. Doing it with only 2 cards would be a waste considering how much a high speed camera costs. I'm very interested if 4-way CF is indeed smother that 2-way. Maybe do it also on last generation cards, GTX590 and 6990 and 2xGTX590 and 2x6990 and see if they managed to improve smoothness. It would be very hilarious if they actually took a step back.

 

[blastingcap]

Wrong that is one of the reasons we have automated tools to minimize that.Your process is clumsy and not repeatable at all.The software I mentioned isn't fraps, they are created for game developers.It is a real time performance analyzer.

Wrong what? You can have automated tools to "minimize" one example I gave, but nothing is foolproof. Hell you could misinstall hardware or whatever as well.

How is it not repeatable, you can do the same run over and over again.

And if what you say is true, tell that to Ryan and TR. You are very smug about this but I suspect that you should not be... if it were that simple, why didn't NV mention this to TR when they talked to them about the insufficiency of using FRAPS data when discussing end-user experience.

Drop the smugness, it is not endearing and you are likely wrong about this. My suggestion may be painstaking as you advance frame by frame, and not automated, but it would get the job done.