[TECH Report] As the second turns: the web digests our game testing methods

[Hitman928]

If AMD releases a fix, would you say the fix would cause a noticeable difference in how the AMD performance appears?

Would that noticeable difference change the results of blind testing between AMD and NVidia?

Has NVidia said if they are going to do a parallel fix to make sure there is not noticeable difference between the two?

That's what I would like to see, actually, a blind test between pre and post 'fix' drivers on AMD. That I think would be the most interesting.

 

[SirPauly]

Rather see tangible data than subjective blind tests, with each individual with their own subjective tastes and tolerances, with some clamoring their subjective view is right, imho!

 

[f1sherman]

If AMD releases a fix, would you say the fix would cause a noticeable difference in how the AMD performance appears?

Would that noticeable difference change the results of blind testing between AMD and NVidia?

Has NVidia said if they are going to do a parallel fix to make sure there is not noticeable difference between the two?

7950 smoothness (lack of) may not be the problem per se. We're not all the same. Some cards are faster, some smoother, NV spends much more on driver team resources etc.

The thing is... Freshly installed OS with minimum background crapware, like the one in TR's benchmarking procedure is really the best case scenario.

What average Joe ends up with in his home machine, will only add to TR's rather academic case of stuttering.

 

[Hitman928]

Rather see tangible data than subjective blind tests, with each individual with their own subjective tastes and tolerances, with some clamoring their subjective view is right, imho!

I agree, but until I see tangible data sufficient enough to base conclusions on, blind tests are welcome information to add to the discussion. Being able to download source feeds of high speed camera tests would be even more valuable, though that's probably asking too much

 

[SirPauly]

What was refreshing is TechReport did bring some tangible data to go along with his subjective views in not only Windows 8 but Windows 7. There are plenty of sites that only offer frame-rate -- if this is the only metric that matters to a gamer -- there ya go - but it's wonderful to see discussions, reviews and investigations that go beyond just frame-rate.

Really believe this kind of awareness will only help improve products from nVidia and AMD.

 

[Leadbox]

If AMD releases a fix, would you say the fix would cause a noticeable difference in how the AMD performance appears?

Would that noticeable difference change the results of blind testing between AMD and NVidia?

Has NVidia said if they are going to do a parallel fix to make sure there is not noticeable difference between the two?

They're not gonna be as smooth as each other are they, in fact, right now if more titles or different sections of skyrim were tested you may well find instances were Amd is smoother.
Speaking personally, if it's going to take high speed camera footage slowed down or a frametime graph to tell me that one is smoother than the other I simply couldn't care less

 

[SirPauly]

Some dismiss frame time graphs but offers enough tangible data to see how single GPU's are smoother than dual multi-GPU; And one can see frame metering to some levels with SLi.

It may not be an ideal method but seems to be a good method 'till there are more ideal tools.

 

[Keysplayr]

Who did I call Asinine? Thats right, you won't find it. Thanks for playing Keys. Some people never change. How is your buddy Rollo doing?

BrightCandle's statement. Just as bad.
And the "idiot" comments? Just stop ok?
Actually people do change.
And Rollo is doing ok AFAIK.

 

[ArchAngel777]

BrightCandle's statement. Just as bad.
And the "idiot" comments? Just stop ok?
Actually people do change.
And Rollo is doing ok AFAIK.

You are acting pretty upset Keys. Lets take a look at my quote:

Originally Posted by ArchAngel777
Fair enough, there is at least one idiot. But, quite honestly, what I react to is the hyperbole being used around here. Most people are level headed around here, with a handful on each side acting like idiots.

Now, I am asking myself, why would you be upset by that statement? Are you taking it upon yourself to lump yourself into that group? Because if so, you did it, not me.

As for the first sentance in my quote, you will notice that I called a specific user on a different forum an idiot for saying that even though AMD acknowledged the problem, he persisted in denial. That, is an idiot who persists in willful ignorance.

Edit** By the way, I noticed you have a habit of taking people out of context. For example, you said "Actually, people do change" when I never said that people don't change. I said "SOME people never change". Please, try not to mispresent what people say. You did that to me about 4 times so far in this thread.

 

[Keysplayr]

You are acting pretty upset Keys. Lets take a look at my quote:



Now, I am asking myself, why would you be upset by that statement? Are you taking it upon yourself to lump yourself into that group? Because if so, you did it, not me.

As for the first sentance in my quote, you will notice that I called a specific user on a different forum an idiot for saying that even though AMD acknowledged the problem, he persisted in denial. That, is an idiot who persists in willful ignorance.

Edit** By the way, I noticed you have a habit of taking people out of context. For example, you said "Actually, people do change" when I never said that people don't change. I said "SOME people never change". Please, try not to mispresent what people say. You did that to me about 4 times so far in this thread.

You win. Can't talk reason with the unreasonable.
Apologies for the rest in the thread for the OT.

 

[HurleyBird]

Keys is many things, but he isn't an idiot

 

[blastingcap]

Fun to read you to.



There aren't any bumpy situations going on right now, is there? I thought I was being fair. At least fairer than someone bringing up a years old issue? Perhaps? :hmm:



I think that is quite enough from you today ArchAngel. So far there have been "idiots" and people who are "assinine". What is your deal? Let the name calling end here please.

Just saying that both companies have had issues and ignored some of them. I agree that AMD needed to fix this frametime issue. Hopefully it will not be of detriment to fps, but I suspect it will.

 

[Hitman928]

I was just glancing through the comments at TR and found some additional comments from Dave Baumann (AMD guy quoted in original article) that I think are insightful, especially in clearing up the situation as theories were running wild over there and possible starting to spill to other forums.

Curious, where did you read any of that?

First off, I mentioned that we have a memory management rewrite underway for orthogonal reasons and it was at a point where we could just try it and see if it makes any improvements; it transpires that in some cases it has helped improve things. This has raised the priority to trying to get this through earlier; that doesn't mean that we "weren't going to bother", otherwise what would have been the point of investing the development time in the first place? Again, the memory manager rewrite was being done for other reasons, and in fact there was no indications from anywhere (including TR's prior reviews) was this an area that needed to be addressed.

Secondly, the memory manager change is not "magic bullet". The quote Scott originally highlighted from me was that these types of things can occur from multiple different areas of the driver - for instance the BL2 improvement primarily comes from resizing a buffer, not related to generic the memory management at all.

This is not due to solely chasing average FPS. One of the software engineers looking into this contacted me to point out that for at least two of the titles where we've made improvements the average FPS should be improved, maybe by a reasonably large degree in some cases.

To be clear, we have not had a single customer complaint or issue arise directly related to what is portrayed here. Indeed, these type of frame render time graphing graphs have been used for many years at HardOCP and no comment have arisen about this for single GPU, most reviews other reviews also show min FPS, are an indicator to similar things (though you need to consider how min FPS is actually calculated); indeed, as far as I had seen, TR's prior reviews using this method didn't highlight anything particularly untoward.

Likewise, we all use our GPU's and, for instance, I've put may many hours in to Borderlands 2 (207 in fact :gasp:!) on other an HD 7970 Ghz Ed or a HD 7870 and I genuinely can't notice any affects on the gameplay related to this to have even flagged it up as something I can perceive. However, we take it seriously and this has occupied quite a lot of architectures and software engineers time over the past few weeks to see where this is coming from and whether it can be improved.

Yes, I think this is the area that needs further investigation. It is interesting that TR has does this type of analysis, and flagged an "issue" but to all intents and purposes I have no frame of reference that indicates that this is an "issue" that affected people in most of the cases; as mentioned nobody has really reported this an issue that affects their gameplay.

 

[Ferzerp]

Indeed, these type of frame render time graphing graphs have been used for many years at HardOCP

This sort of shows that the guy doesn't understand. The [H] graphs are not a per frame graph, but a roll up of a unit of time's worth of fps.

 

[Hitman928]

This sort of shows that the guy doesn't understand. The [H] graphs are not a per frame graph, but a roll up of a unit of time's worth of fps.

Someone already pointed that out to him and he acknowledged that TR's testing is different. He had a little slip up, no biggie.

 

[Ferzerp]

Someone already pointed that out to him and he acknowledged that TR's testing is different. He had a little slip up, no biggie.

It's such a fundamental error though that it shows he's not done his homework, or does not understand the concerns.

 

[BrightCandle]

Its interesting he says no customers have raised this problem because I have 7 support numbers that disagree with that statement. AMD and their lies....

 

[Hitman928]

Its interesting he says no customers have raised this problem because I have 7 support numbers that disagree with that statement. AMD and their lies....

Just curious, do you carry this same level of cynicism for every company you deal with or from which you buy a product?

 

[f1sherman]

PCPER ditching standard FPS measuring all together and instead capturing video output.

This is so much win. Possibly the only thing missing is recording input(mouse) lag.

http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-New-Graphics-Performance-Metric

 

[Keysplayr]

Looks like making the noise was worth the effort. More and more review sites will join in. I hope Anandtech doesn't miss the bus and keep to Ryan's "go play some games" plan.

 

[Hitman928]

PCPER ditching standard FPS measuring all together and instead capturing video output.

This is so much win. Possibly the only thing missing is recording input(mouse) lag.

http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-New-Graphics-Performance-Metric

Very cool, although after a [brief] skim, the only thing I would hope is that they still publish FPS numbers (just as a baseline of comparison). I also could see a problem that the capture card seems to top out at 60Hz (could go faster at lower res?). While this is great for most users, those with 120Hz monitors may find it insufficient. Very cool stuff their working on though. Man, that's gonna eat up some serious storage space and lots and lots of hours.

 

[Hitman928]

It's such a fundamental error though that it shows he's not done his homework, or does not understand the concerns.

That's a lot to infer from one little slip up. Especially given his background.

 

[Keysplayr]

Not sure if I trust that card if it can only do 60Hz. And the overhead that card might bring while capturing frames? I guess we'll need to see that method tested and verified.

 

[blastingcap]

PCPER ditching standard FPS measuring all together and instead capturing video output.

This is so much win. Possibly the only thing missing is recording input(mouse) lag.

http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-New-Graphics-Performance-Metric

Wow that is too awesome. Ryan are you paying attention? AT is using ancient pre-2012 video card analysis that doesn't even have graphs of fps over time, let alone finer-grained stuff like the stuff TR/PCPER are trying to do.

 

[Imouto]

PCPER ditching standard FPS measuring all together and instead capturing video output.

This is so much win. Possibly the only thing missing is recording input(mouse) lag.

http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-New-Graphics-Performance-Metric

Using VirtualDub is a good start but I don't think they are any good at AviSynth scripting since they didn't load the video through a script. AviSynth is a really powerful tool but you must be a real scripting master and a/v scholar to find and measure duplicates, tearing and other stuff.