PCPERDX12 GPU and CPU Performance Tested: Ashes of the Singularity Benchmark

[csbin]

http://www.pcper.com/reviews/Graphi...erformance-Tested-Ashes-Singularity-Benchmark

Finally, this graph attempts to show which card has an advantage on each processor with each setting and with each API. We are using the R9 390X as the baseline score, meaning that any positive percentage result means the GTX 980 is faster while a negative score means the R9 390X is faster.
The green bars are for DX11 performance and in every single instances the GTX 980 is faster, never by less than 15% and often stretching to over 50%! That is not a small advantage and though this benchmark is built to showcase DX12 potential, AMD users should at least by concerned about lost DX11 performance potential in AMD’s current hardware with current shipping games.
The blue bars of DX12 show a different story where AMD’s R9 390X is more often faster than the GTX 980. There are a few cases where NVIDIA’s performance is slightly ahead, but more often than not the gains that AMD sees moving from DX11 to DX12 not only completely make up for the DX11 performance deficit but also bump it ahead of the more expensive graphics card from NVIDIA.

 

[cmdrdredd]

So what conclusions can we draw currently?

It would seem that benchmark isn't heavily threaded as a 6700k beats a 5960x so clockspeed is king. Also seems that performance boosts from DX12(or at least in this test) don't appear to be as strong when you move the resolution and details to a level that removes the CPU limited scenarios. The 390x seems to have a performance advantage compared to a 980 in DX12 when you start stressing the GPU more heavily. Could be drivers, could just be the 980 isn't as strong in this test. Need more testing from various cards.

What I do see is that the 390x goes from unplayable in DX11 to faster than the 980 with DX12. The 980 sometimes loses a few FPS on a really fast CPU. Maybe what this is illustrating is that AMD has more overhead on DX11 than Nvidia and when you remove the driver overhead you get to see more of the GPU's potential. I'd be interested to see a 980ti in this test because it is a good bit faster than the vanilla 980. Also would be interested in SLI and crossfire results.

 

[.vodka]

The boost 390x is getting from DX12 is insane, at least in this particular game. AMD's DX11 driver is/was a mess!

I'd like to see what would happen to Fury X vs 980Ti now!

 

[linkgoron]

I've also seen this:

http://www.extremetech.com/gaming/2...he-singularity-amd-and-nvidia-go-head-to-head

which compares the fury x and 980ti

 

[Azix]

The boost 390x is getting from DX12 is insane, at least in this particular game. AMD's DX11 driver is/was a mess!

I'd like to see what would happen to Fury X vs 980Ti now!

It may not have been. It may have just been they stuck with the standards rather than going extreme with modifications/optimizations. So their cards run dx11 with most of its limitations unchanged.

Or it may have been a mess.

 

[cmdrdredd]

Found some more results. From extreme tech http://www.extremetech.com/gaming/2...he-singularity-amd-and-nvidia-go-head-to-head

980ti loses a lot of performance with MSAA. Nvidia claims there's some type of software bug when MSAA is applied in this benchmark. Probably because they don't have drivers to optimize performance yet and lose in raw performance.

DX11

DX12

 

[Glo.]

http://www.computerbase.de/2015-08/...diagramm-ashes-of-the-singularity-3840-x-2160

What is extremely consistent is that overtime you put highest possible details on Nvidia GPU performance of it goes down, compared to DirectX11.

On directX 12 we see regression of performance even on 980 Ti!

 

[selni]

Looks like some issue with the benchmark on NV there - DX11 results being better than DX12 in a bunch of cases suggests something odd is going on.

 

[.vodka]

980ti loses a lot of performance with MSAA. Nvidia claims there's some type of software bug when MSAA is applied in this benchmark. Probably because they don't have drivers to optimize performance yet and lose in raw performance.

Found this post in the pcper comments, not too far from the first one:

http://oxidegames.com/2015/08/16/the-birth-of-a-new-api/
Baker wrote a specific section just for you.



"All IHVs have had access to our source code for over year, and we can confirm that both Nvidia and AMD compile our very latest changes on a daily basis and have been running our application in their labs for months."
Then, farther on:



"Often we get asked about fairness, that is, usually if in regards to treating Nvidia and AMD equally? Are we working closer with one vendor then another? The answer is that we have an open access policy. Our goal is to make our game run as fast as possible on everyone’s machine, regardless of what hardware our players have.


To this end, we have made our source code available to Microsoft, Nvidia, AMD and Intel for over a year. We have received a huge amount of feedback. For example, when Nvidia noticed that a specific shader was taking a particularly long time on their hardware, they offered an optimized shader that made things faster which we integrated into our code.
We only have two requirements for implementing vendor optimizations: We require that it not be a loss for other hardware implementations, and we require that it doesn’t move the engine architecture backward (that is, we are not jeopardizing the future for the present)."



I believe that answers both your arguments.

Interesting. A year of access to the source code should have squashed any major bugs by this point, I believe.

Still, DX12 is too fresh and it's on its first steps. A lot to work on, on both sides of the fence.

 

[cmdrdredd]

What I meant was that Nvidia is saying there's some bug that the reviewer/tester doesn't see or notice (if it's even there). I think Nvidia says this because they know their MSAA performance isn't good in this test because they don't have a driver out for it. Nvidia performance is always sub par until they release one of their game ready drivers and then things get better.

So basically what I'm saying is Nvidia is trying to get people to ignore the MSAA results. They also said this isn't a good indication of overall DX12 gaming performance with no explanation as to why. Likely because they lose.

I wish we could download this benchmark. I'd be interested to see different user results.

 

[csbin]

http://www.computerbase.de/2015-08/...-nvidia/2/#abschnitt_benchmarks_in_1920__1080

 

[csbin]

What I meant was that Nvidia is saying there's some bug that the reviewer/tester doesn't see or notice (if it's even there). I think Nvidia says this because they know their MSAA performance isn't good in this test because they don't have a driver out for it. Nvidia performance is always sub par until they release one of their game ready drivers and then things get better.

So basically what I'm saying is Nvidia is trying to get people to ignore the MSAA results. They also said this isn't a good indication of overall DX12 gaming performance with no explanation as to why. Likely because they lose.

I wish we could download this benchmark. I'd be interested to see different user results.

 

[Silverforce11]

Why is NV getting slower results in DX12 mode at 4K, what's going on, drivers not ready?

Also, there is something very wrong with the other NV GPUs in the lineup.

Note the almost no gains in DX12 or even slower. Whereas GCN GPUs show a massive boost.

 

[DDH]

Why is NV getting slower results in DX12 mode at 4K, what's going on, drivers not ready?

It is a strange result. Probably why they came out saying it isnt of any value. Interestingly though they released a new driver for alpha AotS.

Also, oxide posted on their blog that the information surrounding MSAA being bugged is incorrect.

 

[bystander36]

I wonder if Nvidia has worked with them to insure that DX11 performs well, as they may not expect most gamers to be on DX12 yet, while AMD has focused on DX12, banking on everyone focusing on DX12.

It is clear this is more than a driver overhead issue for AMD, or we would have seen large disparities like this in current titles. DX11 support looks pretty much abandoned at this stage of development.

I'll be interested in seeing what things will be like at release.

 

[Silverforce11]

Well its clear from the many videos and dev interviews, AMD worked with the devs to focus on Mantle & DX12. If* (they confirmed it did have Mantle running in a few videos in the past) the game at one point had Mantle support, it would have been a waste of $ for AMD to also focus on the DX11 path.

There are definitely driver issues for NV to sort out, though, as they say DX12 is "less driver reliant" since it exposes the hardware directly... i don't know if its just a case of Oxide not optimizing DX12 for NV since they focus on DX11. etc

Certainly, its more in the hands of the devs.

But @zlatan did say, context switching in NV's in-order pipeline is costly on performance so they may not get additional performance advantages using asynchronous compute in DX12. If that's the case, it explains the regression in performance for the entire lineup.

 

[sandorski]

What is with the 390 DX11 results? There are many DX11 Games/Benchmarks that show much closer results between the 390/980 than these. Was this game made for DX12, taking advantage of DX12s' capabilities, then DX11 tacked on for backwards compatibility?

 

[Silverforce11]

What is with the 390 DX11 results? There are many DX11 Games/Benchmarks that show much closer results between the 390/980 than these. Was this game made for DX12, taking advantage of DX12s' capabilities, then DX11 tacked on for backwards compatibility?

Remember BF Hardline beta? AMD's DX11 = broken, crap perf, high frame times. Mantle worked great, smooth as silk. They just focus on that since most of their GPUs support Mantle..

What was stupid, was a few patches later, Mantle was dead and DX11 was optimized instead. lol

 

[Azix]

Does ashes use async compute?

I think the issue is that they have optimizations in dx11 not yet in or not possible in dx12. eg, if they are hacking msaa in dx11 beyond the standard implementation to make it faster, that might explain this.

There really shouldnt be this disparity as it seems oxide has their dx12 implementation solid. Nvidia should have been on top of it. Anyway, I do not expect their dx12 perf to go much higher than their dx11 if they fix something.

What is with the 390 DX11 results? There are many DX11 Games/Benchmarks that show much closer results between the 390/980 than these. Was this game made for DX12, taking advantage of DX12s' capabilities, then DX11 tacked on for backwards compatibility?

It's the other way around. apparently a very optimized dx11 engine with mantle and dx12 added. The performance difference is probably down to CPU bottlenecks

 

[Silverforce11]

It should since their devs have said in public, the usage of async compute helps their dynamic lighting so they can have thousands of individual light sources as well as particles.

 

[Silverforce11]

Combining this statement from Oxide:

There may also be some cases where D3D11 is faster than D3D12 (it should be a relatively small amount). This may happen under lower CPU load conditions and does not surprise us.

To this result (7870K APU, shows NV DX12 improving in performance):
http://www.computerbase.de/2015-08/...of-the-singularity-unterschiede-amd-nvidia/3/

Would mean the regression in performance for NV with the i7 is because the CPU is not the bottleneck and DX12 implementation may be a problem with NV's immature drivers.

We're not gonna get away from IHV optimizing for games & engines in their drivers. DX12 isn't gonna change that. Sadly.

 

[kagui]

It should since their devs have said in public, the usage of async compute helps their dynamic lighting so they can have thousands of individual light sources as well as particles.

just a side note about async compute on other game "The Tomorrow Children" http://fumufumu.q-games.com/archives/TheTechnologyOfTomorrowsChildrenFinal.pdf

they go from 33 ms to 27 ms by using async around 20%

and that is a game that only targets GCN arch (PS4)

 

[cmdrdredd]

I wonder if Nvidia has worked with them to insure that DX11 performs well, as they may not expect most gamers to be on DX12 yet, while AMD has focused on DX12, banking on everyone focusing on DX12.

It is clear this is more than a driver overhead issue for AMD, or we would have seen large disparities like this in current titles. DX11 support looks pretty much abandoned at this stage of development.

I'll be interested in seeing what things will be like at release.

From what I read and I cannot verify any of this, is that Nvidia has some driver voodoo going on for DX11 in general to help with CPU bottlenecks and relieve some of that overhead. AMD doesn't have the same thing going on in their drivers which is why DX11 is such a wide margin in favor of Nvidia. When we go to DX12 there's very little driver optimization right now, if any at all. So it relies on the raw performance of the GPU at the moment. We are simply seeing that AMD's GPUs are hindered by a certain amount of driver overhead in DX11 without game specific optimizations to help out any.

 

[Silverforce11]

just a side note about async compute on other game "The Tomorrow Children" http://fumufumu.q-games.com/archives/TheTechnologyOfTomorrowsChildrenFinal.pdf

they go from 33 ms to 27 ms by using async around 20%

and that is a game that only targets GCN arch (PS4)

Yeah I saw that, the performance gain will change based on the load of compute, I don't know which games use more or less. I'm just assuming there will be a gain, how big, is entirely dependent on the game.

The odd result is no gain or worse performance. Even more odd is that Oxide have said they have given source code to all the IHV for over a year and everyone has been testing and optimizing it in their labs.

I did notice NV's own official guide, showed DX11 vs DX12 using a 2Ghz CPU, with some decent gains. Their results mirror the computerbase.de's 7870K APU results. But on a faster CPU, they have negative gains.

 

[cmdrdredd]

Yeah I saw that, the performance gain will change based on the load of compute, I don't know which games use more or less. I'm just assuming there will be a gain, how big, is entirely dependent on the game.

The odd result is no gain or worse performance. Even more odd is that Oxide have said they have given source code to all the IHV for over a year and everyone has been testing and optimizing it in their labs.

I did notice NV's own official guide, showed DX11 vs DX12 using a 2Ghz CPU, with some decent gains. Their results mirror the computerbase.de's 7870K APU results. But on a faster CPU, they have negative gains.

I think Nvidia has more work to do on DX12 drivers. All currently released games are still DX11 so my assumption is they focused on getting DX11 optimized as much as possible and probably ignored DX12 quite a bit. There's no reason for performance to go down, unless the game engine is doing something specific that doesn't play nice with Nvidia hardware. Maybe something Maxwell just doesn't handle well.