What's a new benchmark without some controversy?
Just a couple of days before publication of this article, NVIDIA sent out an information email to the media detailing its “perspective” on the Ashes of the Singularity benchmark. First, NVIDIA claims that the MSAA implementation in the game engine currently has an application-side bug that the developer is working to address and thus any testing done with AA enabled was invalid. (I happened to get wind of this complaint early and did all testing without to AA avoid the complaints.) Oxide and Stardock dispute this claim as a “game bug” and instead chalk up to early drivers and a new API.
Secondly, and much more importantly, NVIDIA makes the claim that Ashes of the Singularity, in its current form, “is [not] a good indicator of overall DirectX 12 gaming performance.”
What’s odd about this claim is that NVIDIA is usually the one in the public forum talking about the benefits of real-world gaming testing and using actual applications and gaming scenarios for benchmarking and comparisons. Due to the results you’ll see in our story though, NVIDIA appears to be on the offensive, trying to dissuade media and gamers from viewing the Ashes test as indicative of future performance.
NVIDIA is correct in that the Ashes of the Singularity benchmark is “primarily useful to understand how your system runs a series of scenes from the alpha version of Ashes of Singularity” – but that is literally every game benchmark. The Metro: Last Light benchmark is only useful to tell you how well hardware performs on that game. The same is true of Grand Theft Auto V, Crysis 3, etc. Our job in the media is to take that information in aggregate and combine with more data points to paint an overall picture of any new or existing product. It just happens this is the first DX12 game benchmark available and thus we have a data point of exactly one: and it’s potentially frightening for the company on the wrong side.
Do I believe that Ashes’ performance will tell you how the next DX12 game and the one after that will perform when comparing NVIDIA and AMD graphics hardware? I do not. But until we get Fable in our hands, and whatever comes after that, we are left with this single target for our testing.