[Toms] Blind trial of 690 verses 7990

[BrightCandle]

http://www.tomshardware.com/reviews/radeon-hd-7990-geforce-gtx-690-frame-pacing,3599.html

Tomshardware clearly wanted to verify the quantitative data with some qualitative blind testing for FCAT and crossfire. So they tested Catalyst 13.8 and a 7990 against a 690 on latest and saw what people thought on 4 different rigs. 2 with 5760x1080 (no frame pacing) and two with 2560x1600 with frame pacing.

What I think is interesting with these two results is that even after the frame pacing there is still a tendency towards the 690. However there is a problem in Metro last light and skews the numbers. If AMD could get it to work 100% right they would be on par with Nvidia's solution, but they don't yet have it working in all games properly yet. They have clearly however made some progress with frame pacing in the right direction.

 

[wand3r3r]

http://www.tomshardware.com/reviews/radeon-hd-7990-geforce-gtx-690-frame-pacing,3599.html

Tomshardware clearly wanted to verify the quantitative data with some qualitative blind testing for FCAT and crossfire. So they tested Catalyst 13.8 and a 7990 against a 690 on latest and saw what people thought on 4 different rigs. 2 with 5760x1080 (no frame pacing) and two with 2560x1600 with frame pacing.





What I think is interesting with these two results is that even after the frame pacing there is still a tendency towards the 690. However there is a problem in Metro last light and skews the numbers. If AMD could get it to work 100% right they would be on par with Nvidia's solution, but they don't yet have it working in all games properly yet. They have clearly however made some progress with frame pacing in the right direction.

The conclusion you are drawing is wrong (imo). They are essentially dead even 47 vs. 53% with the framepacing driver! That can be read that 50-50 choose other setups. There is not tendency here. (47% didn't find the NV solution smoother while 53 did find it smoother.)

The multi-monitor scenario was clear. There was a clear tendency there.

Anyways interesting to see blind testing.

 

[boxleitnerb]

Why do you count the even votes for AMD? That makes no sense.

 

[Haserath]

The conclusion you are drawing is wrong (imo). They are essentially dead even 47 vs. 53% with the framepacing driver! That can be read that 50-50 choose other setups. There is not tendency here. (47% didn't find the NV solution smoother while 53 did find it smoother.)

The multi-monitor scenario was clear. There was a clear tendency there.

Anyways interesting to see blind testing.

No...
AMD has 8 votes and Nvidia has 16 votes. Ties shouldn't count either way.

 

[wand3r3r]

Now that I think about it I guess that's true.

However half of the people still didn't 'prefer' NV which is surprising given all the hype and attention on the frame times in the last year. Clearly AMD has some work to do, especially on multi monitor.

BF3
2 vs. 2

Bioshock
1 vs. 4

Crysis
4 vs. 2

Grid 2
1 vs. 2

Metro
0 vs. 6

 

[Silverforce11]

No...
AMD has 8 votes and Nvidia has 16 votes. Ties shouldn't count either way.

Not bad considering they had titles in that list of games that did not work well with AMD frame pacing, I am surprised a large number of people felt it was better on AMD hardware at all...

 

[Erenhardt]

Why do you count the even votes for AMD? That makes no sense.

Yes it does. If you ask a question: "Show me which PC is Nvidia powered."
Then you have your answers:
-53% could identify which one was nvidia (supposedly times better than AMD's terrible, eye-scorching experience)
-20% couldn't see a difference
-27% thought AMD rig is powered by superior nvidia solution.
Therefore 53% choosen nvidia as better solution, and 47% didn't see a difference or had it all 'wrong'.

The group of 6 is barely enough for any scientific testing - that is when it comes to properties of an object. If we have possible human error and their subjective feelings, the number of "samples" should be increased... by much.

 

[boxleitnerb]

And if you ask the question the other way around, the result is different. So what?

 

[SiliconWars]

Looks like Nvidia's FUD campaign is in full swing before Hawaii is launched. I think they know what is coming.

This is a very flawed test due to the small sample size btw. There will be games out there that completely switch the Metro result and even it all out - that's the problem with doing this or ANY benchmark with such a small sample size. Can you imagine what the result would have been using Tomb Raider earlier in the year intead of Metro, for example?

 

[SirPauly]

It may be a small sampling but applaud the attempt! Good to see objective and subjective data, too! However, personally place more on objective data!

Good to see AMD improve here with DirectX 10 and 11 ---- and to bring more maturity to EyeFinity, OpenGL and DirectX 9 content moving forward.

 

[boxleitnerb]

Looks like Nvidia's FUD campaign is in full swing before Hawaii is launched. I think they know what is coming.

This is a very flawed test due to the small sample size btw. There will be games out there that completely switch the Metro result and even it all out - that's the problem with doing this or ANY benchmark with such a small sample size. Can you imagine what the result would have been using Tomb Raider earlier in the year intead of Metro, for example?

Nonsense. First, Hawaii is single-GPU, thus there is no reason to assume this blind trial has any repercussions for the battle Hawaii vs. GK110. Secondly, it is not guaranteed that a different game selection would result in different results. AMD still has a long way ahead of them. It seems like their frame pacing is still on a per-application basis.

 

[Leadbox]

Nonsense. First, Hawaii is single-GPU, thus there is no reason to assume this blind trial has any repercussions for the battle Hawaii vs. GK110. Secondly, it is not guaranteed that a different game selection would result in different results. AMD still has a long way ahead of them. It seems like their frame pacing is still on a per-application basis.

I'm pretty sure both solutions are a general framework that needs app specific tweaking

 

[SiliconWars]

Secondly, it is not guaranteed that a different game selection would result in different results. AMD still has a long way ahead of them. It seems like their frame pacing is still on a per-application basis.

It's also not guaranteed that a completely different set of games could have ended up with AMD scoring a 20-0 victory. But, it could happen, and is more likely to happen with a small selection of games.

Seems a bit strange that the latest game (Metro) is the one that performs clearly worst on AMD, and is also a TWIMTBP game. Well, not really strange actually.

 

[BrightCandle]

[/B]
I'm pretty sure both solutions are a general framework that needs app specific tweaking

NVidia claim theirs is hardware based which would preclude any possibility that it is tweaked by game.

Frame pacing from AMD is working, most people can't tell much difference between the two solutions. There is a slight preference still but that corresponds to a big difference in the actual game performance relatively speaking. They can tell if frame pacing is there or not and definitely prefer frame pacing.

Some of them are preferring AMD when clearly there is a problem in the motion. That corresponds with what we saw last year and early this year in our forums where a lot of AMD users said they didn't notice any problem and couldn't see it. Its definitely a problem for the majority based on the blind trial (although perhaps the sample is just too small here) but the majority of people do notice the problem.

I would like to see them get some pro gamers who use 144Hz screens all day long who can spot these sorts of problems and latency and such and do a blind trial with them as well, I suspect the results will be dramatically different. I think it was nice to see the quantitative validated with a qualitative measure as well which I know many people here were asking for and wouldn't believe FCAT without it.

 

[Leadbox]

NVidia claim theirs is hardware based which would preclude any possibility that it is tweaked by game.

I meant app specific driver tweaking by both nvidia and AMD to get their pacing/metering solution working optimally with given app. It's not just going to work just because its hardware based is it?

 

[Essence_of_War]

Did anybody glean from the article whether a control group was used at 2560x1600?

If I'm reading it correctly, they had everyone play at 5760x1080 with obvs no frame pacing, and then had them play again at 2560x1600 w/ frame pacing drivers.

Did the contestants know that the frame pacing was being added? If so, it seems like it would be hard to figure out what part of the effect size is a placebo without a control group getting the placebo.

 

[FalseChristian]

It looks good for AMD (ATI) to me. At least the new frame-pacing drivers are clearly working and will just get better.

It's all a mute point, though, as AMD is coming out with the Radeon HD 9970 soon before NVidia's Maxwell which should be faster than a Titan.

 

[Haserath]

Wait... I just got Metro LL yesterday, and I had terrible lag like Toms said. Turns out 'Advanced Physx' was the cause(thx Nvidia...). Should be turned off by default for AMD cards but no... I bet it runs on the CPU causing it to get capped at 20fps often.

So is crossfire really a problem in Metro or did they leave on Physx? The standings would change quite a bit.

AMD did just release new drivers today though. New crossfire profile for Metro.

 

[SiliconWars]

AMD did just release new drivers today though. New crossfire profile for Metro.

This is another thing I've noticed about Toms - the awesome timing they appear to have with these articles.

Who remembers the crossfire vs sli article they did the week before Barts was released? And they are always doing APU vs cheap dual + discrete articles when the discrete cards are at their cheapest, and the APU's are just released.

Though not the worst I do feel that some of their reviewers - Angelini and Woligrowski in particular - are a bit too close to Nvidia for comfort.

 

[VulgarDisplay]

NVidia claim theirs is hardware based which would preclude any possibility that it is tweaked by game.

Frame pacing from AMD is working, most people can't tell much difference between the two solutions. There is a slight preference still but that corresponds to a big difference in the actual game performance relatively speaking. They can tell if frame pacing is there or not and definitely prefer frame pacing.

Some of them are preferring AMD when clearly there is a problem in the motion. That corresponds with what we saw last year and early this year in our forums where a lot of AMD users said they didn't notice any problem and couldn't see it. Its definitely a problem for the majority based on the blind trial (although perhaps the sample is just too small here) but the majority of people do notice the problem.

I would like to see them get some pro gamers who use 144Hz screens all day long who can spot these sorts of problems and latency and such and do a blind trial with them as well, I suspect the results will be dramatically different. I think it was nice to see the quantitative validated with a qualitative measure as well which I know many people here were asking for and wouldn't believe FCAT without it.

High level players are not using multi gpu simply to avoid the problems in the first place. Also, there isn't really a pro gaming scene for anything other than RTS's and games like DOTA and LOL. Not really games where rendering smoothness matters.

 

[Zanovar]

Wow this thread is back*chuckles*someone needs to grow a pair...

 

[el etro]

This is another thing I've noticed about Toms - the awesome timing they appear to have with these articles.

Who remembers the crossfire vs sli article they did the week before Barts was released? And they are always doing APU vs cheap dual + discrete articles when the discrete cards are at their cheapest, and the APU's are just released.

Though not the worst I do feel that some of their reviewers - Angelini and Woligrowski in particular - are a bit too close to Nvidia for comfort.

I said this to ClubeDoHardware(Hardware Secrets brazilian edition) guys late in 2010. Is good to know i'm not the only one in the world who think this. The frametimes tests began right after the reviews showed Crossfire outperforming SLI after many years.

 

[cmdrdredd]

SO why is frame pacing disabled for multi monitor? Does it not work in that case?

 

[BrightCandle]

SO why is frame pacing disabled for multi monitor? Does it not work in that case?

AMD hasn't made it work in all cases yet. DX9 games and high resolutions such as 3 screens are both currently unsupported.

SO why is frame pacing disabled for multi monitor? Does it not work in that case?

Does it matter? Conspiracy theories asside there is a perfectly good simple explanation for that - we only just got the technology to test for it and find out the cause. Toms give their reasons in the test, they want to make sure the quantitative results were representative of how people experienced games, which they to an extent proved it is. But what they also showed was in the cases where NVidia has a slight advantage in stutter it doesn't show up in the users, so it starts to give us an idea of what matters and what doesn't.

Why it happened doesn't change the results, the result stand on their own regardless of the motives behind them.

 

[cmdrdredd]

Wow the quote button broke on that one lol.

I would have figured that multi-monitor would make no difference in terms of whather it works with a particular game or not. Didn't know it did.