Question DLSS interaction with Global Illumination

[coercitiv]

There's a new RTX showcase demo from Nvidia which allows turning RTX and DLSS features on and off in real-time. A video showcase can be seen here.

Since DLSS can be switched on in real-time, one can easily see the effect it has on the scene illumination. I'll include a sample bellow:


As you can see, enabling DLSS has a strong effect on perceived illumination for both highlights and shadows. I'd like to stress that it doesn't matter what screenshot one likes best out of the two, if this was a real game scene then the game designer would probably tweak the lighting to get closer to one presentation or the other depending on artistic intent.

The big question is why would DLSS be tuned to affect perceived illumination in a way that at best makes it look like native rendering with RTX Global Illumination is not accurate enough?

PS: It would be great if the discussion does not immediately degenerate into DLSS BAD vs DLSS GOOD from the first page.

 

[Mopetar]

DLSS takes custom hardware and the AMD cards don't have it, after that it takes programming a supercomputer to generate the algorithm to run on that hardware which AMD probably don't have either.

Really you just need something that can effectively run a trained neural network. Nvidia is using the tensor cores for this since they can do matrix operations and other sorts of things that ML algorithms frequently use. This could be done with other hardware, perhaps not as efficiently. Even tensor cores themselves are fairly general purpose considering they can be used for a lot of things that aren't DLSS. Actual dedicated hardware certainly isn't necessary, but may actually provide a big boost to efficiency or performance.

The training could be done on any hardware, but obviously one that can deal with a massive data set used for training and refining the network more quickly would be most beneficial, particularly if developers don't give you data ahead of time to be able to train the neural network and you need to do it after the game launches.

 

[Dribble]

This is arbitrarily false. nVidia developed Tensor Cores for datacenter use. When building cards for consumer use, they decided to add a small number of these core types. They have very little usage in a consumer workload. They decided to have some parts of DLSS use them, as otherwise they sit there doing nothing at all, taking up die space, and consuming power.

nVidia's thought process behind dedicated hardware for both RT and DLSS is vendor lock in, which has been there goal for pretty much every single piece of proprietary tech they have developed in the last 20 years.

What's false?
That they have custom hardware - you agree they do. What it was originally developed for doesn't matter. If you want DLSS you really need the hardware.
What has the vendor lock in got to do with AMD developing a DLSS equivalent?

Really you just need something that can effectively run a trained neural network. Nvidia is using the tensor cores for this since they can do matrix operations and other sorts of things that ML algorithms frequently use. This could be done with other hardware, perhaps not as efficiently. Even tensor cores themselves are fairly general purpose considering they can be used for a lot of things that aren't DLSS. Actual dedicated hardware certainly isn't necessary, but may actually provide a big boost to efficiency or performance.

The training could be done on any hardware, but obviously one that can deal with a massive data set used for training and refining the network more quickly would be most beneficial, particularly if developers don't give you data ahead of time to be able to train the neural network and you need to do it after the game launches.

That's just like saying I can use my cpu to do my game graphics. Sure you can but it won't be very good at it, and anyway it's busy doing other stuff so doesn't have time to do it. Hence we need some specialised hardware, which will be orders of magnitude faster at that specialist task then more general purpose hardware.
The super computer training is the same as above - sure you could do it on all sorts of things eventually but to do it efficiently you need the right super computer, and you need experience in training neural networks of that type on it. The fact that it took Nvidia so long to do suggests it's not a simple task.

 

[dr1337]

What's false?
That they have custom hardware - you agree they do. What it was originally developed for doesn't matter. If you want DLSS you really need the hardware.
What has the vendor lock in got to do with AMD developing a DLSS equivalent?


That's just like saying I can use my cpu to do my game graphics. Sure you can but it won't be very good at it, and anyway it's busy doing other stuff so doesn't have time to do it. Hence we need some specialised hardware, which will be orders of magnitude faster at that specialist task then more general purpose hardware.
The super computer training is the same as above - sure you could do it on all sorts of things eventually but to do it efficiently you need the right super computer, and you need experience in training neural networks of that type on it. The fact that it took Nvidia so long to do suggests it's not a simple task.

There was a time when game physics were too difficult for CPUs to manage and physx accelerator cards were a thing. Now its all done on the CPU or just run on the shaders naturally. Take a look at the computing landscape as it is today. CPUs are getting better and better vector and AI performance, and even AMD can do AI reasonably well without dedicated tensor cores. Shoot I mean theyre squeezing 53 TOPS out of vega as is in mi50, there should be plenty of AI compute available for AMD to introduce a DLSS competitor without having to add dedicated cores. Unless theres a magic DLSS v3 that turns 720p into 8k with perfect quality but *needs* tensor cores, maybe there will be a place for them in gaming GPUs. But as things stand it really looks like AI upscaling will be easy enough to do without a truckload of AI compute hardware on your graphics card

 

[Mopetar]

That's just like saying I can use my cpu to do my game graphics. Sure you can but it won't be very good at it, and anyway it's busy doing other stuff so doesn't have time to do it.

It's all relative. I'd wager the general GPU hardware handles a neural net better relatively speaking than a CPU handled graphical rendering.

There are probably other ways to produce a similar effect to DLSS with a far different implementation which might benefit more from a different kind of specialized hardware.

Frankly I still don't care about the feature and think it creates a lot of bad incentives where there's more emphasis placed on being able to fake a good enough high resolution image than there is in being able to actually render one.

The only use cases for this technology I've actually thought were reasonable is extending the life of an old card that would normally require dropping resolution or something like the Switch where a portable console can have an alternative to just pixel doubling when connected to a high resolution TV.

 

[Stuka87]

What's false?
That they have custom hardware - you agree they do. What it was originally developed for doesn't matter. If you want DLSS you really need the hardware.
What has the vendor lock in got to do with AMD developing a DLSS equivalent?

You said that DLSS requires hardware, but this isn't true. nVidia chose to partially use dedicated hardware to lock the solution to their cards. This is the same thing they did with many other techs they have introduced. ALL of which (except DLSS as its new) have since been un-tied to the dedicated hardware and can now run on anything. This includes things like PhysX and G-Sync. PhysX now runs primarily on CPUs, with GPUs used for the rendering side, and G-Sync was replaced with an AMD developed open standard that accomplishes the exact same thing without the vendor tie in.

 

[Thala]

You said that DLSS requires hardware, but this isn't true. nVidia chose to partially use dedicated hardware to lock the solution to their cards.

This makes no sense. If you implement the algorithm in the driver while using any other HW resource than tensor cores, it would be locked to nVidia cards as well - because you frankly cannot run an AMD GPU with an NVidia driver. On the other hand, the idea of using tensor cores for neural network inference tasks is not a totally absurd, to say the least...
With other words, using tensor cores is the natural technical solution for the problem at hand and has nothing to do with locking the solution - because it can be locked without using special hardware. Likewise, if AMD had the know-how, they could implement something similar to DLSS by using any HW resource their GPUs have available.

 

[GodisanAtheist]

I think it's also worth noting that part of the reason DLSS provides such an uplift in performance is precisely because it is using those "worthless" tensor cores and not having to have to compete for performance with their shaders while rendering a game.

Now, if NV used the die space associated with tensor cores and just crammed more shader resources in there, what would the performance look like? Dunno.

But at the end of the day NV traded larger dies for a huge boost in performance (and quality, if we're talking 4k and up).

 

[Stuka87]

This makes no sense. If you implement the algorithm in the driver while using any other HW resource than tensor cores, it would be locked to nVidia cards as well - because you frankly cannot run an AMD GPU with an NVidia driver. On the other hand, the idea of using tensor cores for neural network inference tasks is not a totally absurd, to say the least...
With other words, using tensor cores is the natural technical solution for the problem at hand and has nothing to do with locking the solution - because it can be locked without using special hardware. Likewise, if AMD had the know-how, they could implement something similar to DLSS by using any HW resource their GPUs have available.

Obviously you can't use an nVidia driver on an AMD card. But if it was purely a software solution that game developers are adding to their games, then the same code in the game could be made to work on other GPUs that implement the feature. Because DLSS is essentially using DirectML (A DirectX feature) for the upscaling. But it deviates from the DirectX standard.

I think it's also worth noting that part of the reason DLSS provides such an uplift in performance is precisely because it is using those "worthless" tensor cores and not having to have to compete for performance with their shaders while rendering a game.

Now, if NV used the die space associated with tensor cores and just crammed more shader resources in there, what would the performance look like? Dunno.

But at the end of the day NV traded larger dies for a huge boost in performance (and quality, if we're talking 4k and up).

But it should be noted DLSS is markedly slower than straight upscaling with a sharpening filter. Yes, DLSS 2.0 looks better in many (but not all) cases than upscaling with a sharpening filter, but its also notably slower. The question is, how would it differ on just straight general compute cores. And, we most likely will know one day, but my thinking is it will be a while. Just like it was a while before PhysX was able to run on general compute and CPUs.

Many think (rightfully so) that Ampere was a failure because it offered very little rastering performance increases, even though the dies were much larger, and therefore the cost was a lot higher.

 

[Dribble]

The question is, how would it differ on just straight general compute cores. And, we most likely will know one day, but my thinking is it will be a while.

We have had upscaling using gpu cores for years now. Devs have spend a lot of effort on it in fact as it's key for the consoles which upscale most of the time. We know what they can do. Adding AI cores to the gpu is what enabled Nvidia to do something different with DLSS. AMD don't have the AI cores, so basically have the same general shading hardware that's been available for many years. I don't understand how AMD probably with a couple of devs working on this are going to magically do something much better then 10 years of console development managed?

The almost certain truth is AMD won't achieve much until they release cards with the missing AI cores added.

 

[Stuka87]

The almost certain truth is AMD won't achieve much until they release cards with the missing AI cores added.

Do you have some insider information that has alluded the entire industry to make this claim? AI Cores can make certain types of calculations faster, but they don't have any sort of majic abilities that general compute cores are incapable of doing.

 

[JoeRambo]

AI Cores can make certain types of calculations faster, but they don't have any sort of majic abilities that general compute cores are incapable of doing.

They don't have any magic for sure, but they provide extra processing power that can be used at same time as shader resources. On AMD GPUs these calculations will compete with other rendering tasks.
So no magic involved, just common sense.

On paper Nvidia can throw way more computational power at DLSS 2.0. I have no clue what internal data formats they are using, but even if the format is FP16 the disparity between resources is huge:

~45TF FP16 for Radeon for ALL tasks versus ~60TF FP16 in Tensor cores + ~35TF in chip

A good baseline of what is available. Moving to different formats like INT8 keeps proportions, while NV could potentially make use of sparsity up to doubling available computational resources on Tensor side of equation.

Still, i feel with solutions like DLSS, the proverb perfect is the enemy of good is as true as ever. You basically take a hit in image quality to increase performance big time. So AMD can get away with solution that looks worse than DLSS, but they need to get it out soon, as they are getting slaughtered in DLSS 2.0 supporting games.

 

[coercitiv]

Going back towards the original topic, it appears DLSS has some issues with HDR mode in Metro Exodus Enhanced edition. Ghosting and gamma issues are mentioned, with an example for "ghosting" from Andreas Schilling @ HardwareLUXX bellow.

https://twitter.com/x/status/1390552135994068992

They will probably get fixed by the dev, it's still odd how DLSS interacts with gamma, an issue which seems strongly related to the OP.

 

[coercitiv]

Here's an interesting example on how an image reconstruction tehnique is used to manipulate scene colorization and lighting among (many) other aspects.
https://intel-isl.github.io/PhotorealismEnhancement/

There's a video there everybody should watch, even if just for the first few minutes. What I find fascinating is how much additional information is extracted from the game engine to obtain such results.

 

[Dribble]

Going back towards the original topic, it appears DLSS has some issues with HDR mode in Metro Exodus Enhanced edition. Ghosting and gamma issues are mentioned, with an example for "ghosting" from Andreas Schilling @ HardwareLUXX bellow.

https://twitter.com/x/status/1390552135994068992

They will probably get fixed by the dev, it's still odd how DLSS interacts with gamma, an issue which seems strongly related to the OP.

Apparently fixed: https://wccftech.com/metro-exodus-pc-enhanced-edition-hotfix-dlss/

 

[JoeRambo]

Apparently fixed: https://wccftech.com/metro-exodus-pc-enhanced-edition-hotfix-dlss/

Yay. Incredible game, anyone can get it from Humble Bundle's "choice" for $12 i think.