Discussion AMD Gaming Super Resolution GSR

[DisEnchantment]

New Patent came up today for AMD's FSR

Patent Public Search | USPTO

appft.uspto.gov

https://www.freepatentsonline.com/y2021/0150669.html

20210150669 GAMING SUPER RESOLUTION

Abstract
A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display

[0008] Conventional super-resolution techniques include a variety of conventional neural network architectures which perform super-resolution by upscaling images using linear functions. These linear functions do not, however, utilize the advantages of other types of information (e.g., non-linear information), which typically results in blurry and/or corrupted images. In addition, conventional neural network architectures are generalizable and trained to operate without significant knowledge of an immediate problem. Other conventional super-resolution techniques use deep learning approaches. The deep learning techniques do not, however, incorporate important aspects of the original image, resulting in lost color and lost detail information.

[0009] The present application provides devices and methods for efficiently super-resolving an image, which preserves the original information of the image while upscaling the image and improving fidelity. The devices and methods utilize linear and non-linear up-sampling in a wholly learned environment.

[0010] The devices and methods include a gaming super resolution (GSR) network architecture which efficiently super resolves images in a convolutional and generalizable manner. The GSR architecture employs image condensation and a combination of linear and nonlinear operations to accelerate the process to gaming viable levels. GSR renders images at a low quality scale to create high quality image approximations and achieve high framerates. High quality reference images are approximated by applying a specific configuration of convolutional layers and activation functions to a low quality reference image. The GSR network approximates more generalized problems more accurately and efficiently than conventional super resolution techniques by training the weights of the convolutional layers with a corpus of images.

[0011] A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display.

[0012] A processing device is provided which includes memory and a processor configured to receive an input image having a first resolution. The processor is also configured to generate a plurality of non-linear down-sampled versions of the input image via a non-linear upscaling network and generate one or more linear down-sampled versions of the input image via a linear upscaling network. The processor is also configured to combine the non-linear down-sampled versions and the one or more linear down-sampled versions to provide a plurality of combined down-sampled versions. The processor is also configured to convert the combined down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution by assigning, to each of a plurality of pixel blocks of the output image, a co-located pixel in each of the combined down-sampled versions and provide the output image for display.

[0013] A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display.

It uses Inferencing for upscaling. As will all ML models, how you assemble the layers, what kind of parameters you choose, which activation functions you choose etc, matters a lot, and the difference could be night and day in accuracy, performance and memory

 

[Shamrock]

If you've got Navi 31 caliber card, why are you even running FSR/DLSS at all. Those should have plenty of horse power even with the newest titles at the most demanding resolutions.

For those cards it's only something to consider for when the card is several years old and can't keep up like it used to. Of course by then you'd hope it supports whatever newest version of FSR/DLSS is available.

Because some people still run games with Cryengine. For instance, I'm big into Mechwarrior Online, and its Cryengine. My 6700xt only manages 55fps @1440p.

 

[Saylick]

Some new developments about FSR 2.0 and RSR: https://videocardz.com/newz/amd-fsr...ally-launches-q2-2022-rsr-launches-march-17th

 

[moinmoin]

Btw. I'm pretty sure if FSR 2.0 is not actually building on it might at least be heavily inspired by Unreal Engine 5's TSR.

 

[Saylick]

Btw. I'm pretty sure if FSR 2.0 is not actually building on it might at least be heavily inspired by Unreal Engine 5's TSR.

I've read this as well, that FSR 2.0 is based on Gen 5 TAAU, which has been backported to some Unreal Engine 4 games. From what I've read, Gen 5 TAAU can be as good as DLSS 2.0, but I'm not sure how much tuning is required on a game-by-game basis to achieve that. For AMD, FSR 2.0 will come handy at lower internal resolutions where FSR 1.0 has traditionally struggled, e.g. when you're starting with 540p and upscaling to 1080p, because it's a temporal upscaler. The challenge, however, is to tune the algorithm, assuming it doesn't use a neutral network, to mitigate ghosting or shimmering. That's really the secret sauce of DLSS 2.x, that Nvidia have worked out the usual kinks of TAA. The neural network can run on shaders, but they'll never admit that because it harms the sales of their latest GPUs. If FSR 2.0 runs on shaders and is sufficiently good enough with wide adoption, it might force Nvidia to open their hand and make DLSS more open source and be able to run on shaders, a la Freesync vs Gsync. Someone in the Videocardz comments mentioned Nvidia vs AMD as Betamax vs VHS, and it's not hard to see why that analogy was made.

 

[Panino Manino]

FSR1.0 was child's toy, just to say that they had something.
This FSR2.0 is looking just a good temporal thing that you find on some engines already. Yes, it's good if you can force this on any game, but still, I'm still feel that AMD is just playing games pretending that they have a competitor for his rival's product.
While it's nice that you don't need extra hardware to use this that extra hardware is not being used for nothing.

I really hope that AMD is working hard behind the scenes on the software for when the hardware (Xilinx) becomes available on their GPUs.

 

[GodisanAtheist]

I've read this as well, that FSR 2.0 is based on Gen 5 TAAU, which has been backported to some Unreal Engine 4 games. From what I've read, Gen 5 TAAU can be as good as DLSS 2.0, but I'm not sure how much tuning is required on a game-by-game basis to achieve that. For AMD, FSR 2.0 will come handy at lower internal resolutions where FSR 1.0 has traditionally struggled, e.g. when you're starting with 540p and upscaling to 1080p, because it's a temporal upscaler. The challenge, however, is to tune the algorithm, assuming it doesn't use a neutral network, to mitigate ghosting or shimmering. That's really the secret sauce of DLSS 2.x, that Nvidia have worked out the usual kinks of TAA. The neural network can run on shaders, but they'll never admit that because it harms the sales of their latest GPUs. If FSR 2.0 runs on shaders and is sufficiently good enough with wide adoption, it might force Nvidia to open their hand and make DLSS more open source and be able to run on shaders, a la Freesync vs Gsync. Someone in the Videocardz comments mentioned Nvidia vs AMD as Betamax vs VHS, and it's not hard to see why that analogy was made.

- NV will never open up DLSS for use by competitors. They've demonstrated time and again that they'd rather make their exclusive option the "NV Premium" and the industry standard is for the unwashed plebs.

 

[Saylick]

- NV will never open up DLSS for use by competitors. They've demonstrated time and again that they'd rather make their exclusive option the "NV Premium" and the industry standard is for the unwashed plebs.

Yeah, you're probably right lol. But if an open source, or at least widely available, option is good enough, that's a win for the average consumer.

 

[Leeea]

Because some people still run games with Cryengine. For instance, I'm big into Mechwarrior Online, and its Cryengine. My 6700xt only manages 55fps @1440p.

I love MWO, but:

MWO is a single threaded game. Your GPU does not matter. Get the fastest single threaded CPU you can find.

It is all about the CPU, something like a 5600x or 5800x3D is likely your best bet. My CPU upgrade in MWO went from 40 - 50 fps at lowest settings to over a 100 fps at highest* settings with the same video card.

*except for particles, leave that at low

if you have not done it, turn your particle effects to the lowest setting. It will make no visual difference, but reduces the CPU impact of a horrifically coded particle system in MWO. MWO is very much a game held together with band aids and hope.

 

[Dribble]

Yeah, you're probably right lol. But if an open source, or at least widely available, option is good enough, that's a win for the average consumer.

This is just another temporal upscaler using shaders of which there are several, to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware, and secondly write the maching learning software. That's a bit harder as the only people who have written the software for gaming so far are Nvidia and they aren't going to share. The best bet for a real open source competitor might be Intel as they are hinting at having specialist machine learning hardware for their cards so will be developing a DLSS competitor. It might be open source depending on how generious Intel are feeling.

 

[Saylick]

This is just another temporal upscaler using shaders of which there are several, to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware, and secondly write the maching learning software. That's a bit harder as the only people who have written the software for gaming so far are Nvidia and they aren't going to share. The best bet for a real open source competitor might be Intel as they are hinting at having specialist machine learning hardware for their cards so will be developing a DLSS competitor. It might be open source depending on how generious Intel are feeling.

I think it's been confirmed multiple times already that the secret sauce in DLSS isn't that it requires tensor or matrix units to run; it's that it uses a neural network (i.e. the deep learning aspect) to do the interpolating of samples between frames. You can run the inferencing on the neural network on vector units, but obviously it would run faster on an execution unit that can do matrix math. Intel is already developing their version of DLSS called XeSS which can run on either shaders using the DP4a instruction or on dedicated matrix units (Intel calls them XMX in their architecture). According to Intel, the frame time between running the neural network on dedicated matrix units vs. shaders isn't all that different. If AMD can implement the neural network component but open it up to all GPUs that support DP4a, which apparently is most, if not all, GPUs within the last couple of generations, that's a big win. It's precisely what Intel intends to accomplish anyways with XeSS, as they plan on making it open source as well.

By offering a DP4a version of XeSS, game developers will be able to use XeSS on virtually all modern hardware, including competing hardware. In that respect Intel is taking a page from AMD’s playbook, targeting their own hardware while also letting customers of competitors benefit from this technology – even if by not quite as much. Ideally, that will be a powerful carrot to entice game developers to implement XeSS in addition to (or even in place of) other upscaling techniques. And while we won’t put the cart before the horse, should XeSS live up to all of Intel’s performance and image quality claims, then Intel would be in the unique position of being able to offer the best of both worlds: an upscaling technology with wide compatibility like AMD’s FSR and the image quality of NVIDIA’s DLSS. As an added kicker, Intel is also planning on eventually open sourcing the XeSS SDK and tools. At this juncture there are no further details on their commitment – presumably, they want to finish and refine XeSS before releasing their tech to the world – but this would be a further feather in Intel’s cap if they can deliver on that promise as well.

 

[maddie]

This is just another temporal upscaler using shaders of which there are several, to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware, and secondly write the maching learning software. That's a bit harder as the only people who have written the software for gaming so far are Nvidia and they aren't going to share. The best bet for a real open source competitor might be Intel as they are hinting at having specialist machine learning hardware for their cards so will be developing a DLSS competitor. It might be open source depending on how generious Intel are feeling.

Machine learning is simply math. Why is there such a mystique about it? Any computing device can do it as you don't NEED tensor cores, They're configured to do it faster than general purpose hardware. Who amongst us can speak with authority on whether an algorithm designed for shader cores cannot work?

The huge discrepancy in shader vs tensor core numbers tell me that ML can be done on GPUs without dedicated matrix cores.

Nvidia's marketing is awesome.

 

[Saylick]

I'm a fool... why didn't I dig into VCZ article a little more... Anyways, some answers to our questions:

 

[Shamrock]

I love MWO, but:

MWO is a single threaded game. Your GPU does not matter. Get the fastest single threaded CPU you can find.

It is all about the CPU, something like a 5600x or 5800x3D is likely your best bet. My CPU upgrade in MWO went from 40 - 50 fps at lowest settings to over a 100 fps at highest* settings with the same video card.

*except for particles, leave that at low

if you have not done it, turn your particle effects to the lowest setting. It will make no visual difference, but reduces the CPU impact of a horrifically coded particle system in MWO. MWO is very much a game held together with band aids and hope.

I have a 5600x with my 6700xt, only gets 55fps with everything on high. I will try lowering particles.

 

[Dribble]

Machine learning is simply math. Why is there such a mystique about it? Any computing device can do it as you don't NEED tensor cores, They're configured to do it faster than general purpose hardware. Who amongst us can speak with authority on whether an algorithm designed for shader cores cannot work?

The huge discrepancy in shader vs tensor core numbers tell me that ML can be done on GPUs without dedicated matrix cores.

Nvidia's marketing is awesome.

Drawing triangles on the screen is math, you can do it with a cpu so you don't NEED a gpu but it doesn't mean it will do it particularly well, even the lowest of the lowest end gpu hardware can run rings around what you can do with just general purpose cpu cores.

As to why using a neural net on dedicated hardware is better well the proof is there to see in DLSS working better then any shader solution. This is despite only one company (Nvidia) spending a relatively short time working on it (say 5 years) vs probably hundreds of companies spending much longer working on upscaling using shaders (since the first consoles with hardware shader support basically).

 

[maddie]

Drawing triangles on the screen is math, you can do it with a cpu so you don't NEED a gpu but it doesn't mean it will do it particularly well, even the lowest of the lowest end gpu hardware can run rings around what you can do with just general purpose cpu cores.

As to why using a neural net on dedicated hardware is better well the proof is there to see in DLSS working better then any shader solution. This is despite only one company (Nvidia) spending a relatively short time working on it (say 5 years) vs probably hundreds of companies spending much longer working on upscaling using shaders (since the first consoles with hardware shader support basically).

Your analogy likens using a few CPU cores to replace many more GPU cores and compares it to using a GPU high count shader cores to much fewer Tensor cores.

That's not quite the comparison you think it is. It's opposing itself, as each half is moving in the opposite direction.

Maybe Nvidia has their upscaling solutions working better simply because they started earlier in development. One kudos I will give them is their constant search for techniques that are unique. Helps with selling, but you have to constantly keep it churning. In any case, FSR 2.0 seems to be a major leap based on rumors and we'll see soon how much Tensor cores are essential.

Saylick has 2 posts just above this with some very interesting info. Read it.

 

[Stuka87]

This is just another temporal upscaler using shaders of which there are several, to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware, and secondly write the maching learning software. That's a bit harder as the only people who have written the software for gaming so far are Nvidia and they aren't going to share. The best bet for a real open source competitor might be Intel as they are hinting at having specialist machine learning hardware for their cards so will be developing a DLSS competitor. It might be open source depending on how generious Intel are feeling.

People need to stop spreading this fallacy. nVidia is using tensor cores because they want to keep it proprietary. Just like they do with everything. Just look at G-Sync. nVidia claimed it had to be done in hardware, and that you had to have one of their cards to use it. Only for that to be proven entirely false and now everybody has access to the same tech with any brand of video card and almost any monitor.

The same thing will happen with DLSS. nVidia will have a few years of "our way or the highway", and then an open standard will come along and completely replace their proprietary version.

 

[GodisanAtheist]

Maybe we need a Dribble's Law of the internet: All discussions about ancillary graphics technology will devolve into a discussion about why Nvidia's proprietary approach is better or not...

 

[RnR_au]

FSR2 will be announced very shortly.

AMD Set to Announce FSR 2.0 Featuring Temporal Upscaling on March 17th

AMD is preparing to announce their FidelityFX Super Resolution (FSR) successor tomorrow, on March 17th before a showcase of the technology at GDC 2022 as we previously reported on according to leaked slides obtained by VideoCardz. AMD FSR 2.0 will use temporal data and optimized anti-aliasing to...

www.techpowerup.com

 

[Dribble]

Maybe we need a Dribble's Law of the internet: All discussions about ancillary graphics technology will devolve into a discussion about why Nvidia's proprietary approach is better or not...

Seems more to be people desperate to claim that things like dedicated AI hardware and associated software are all snake oil and don't actually do anything because they are proprietary and made by Nvidia, even when there is clear proof otherwise.

 

[Saylick]

And we're official.

AMD Teases FSR 2.0: Temporal Upscaling Tech for Games Coming in Q2

www.anandtech.com

 

[Stuka87]

More details here:

 

[Stuka87]

Seems more to be people desperate to claim that things like dedicated AI hardware and associated software are all snake oil and don't actually do anything because they are proprietary and made by Nvidia, even when there is clear proof otherwise.

Its more about nVidia and their "brand ambassadors" (secret or otherwise) always saying their way is the only way. And it has been proven wrong every single time. Going all the way back to PhysX, nVidia's proprietary systems have always been replaced by open standards that have in the end been better for everybody involved.

 

[maddie]

Seems more to be people desperate to claim that things like dedicated AI hardware and associated software are all snake oil and don't actually do anything because they are proprietary and made by Nvidia, even when there is clear proof otherwise.

Is clear logical thinking so hard to expect? No one has claimed what you write. You are the one claiming the essential, impossible without, argument. ML AI image improvement can't be done effectively without Tensor/matrix cores. That is what YOU said. The opposing argument says no, general purpose shader cores can be substituted. You are the one making ridiculous claims and then accusing others of exactly what you are doing. What a shameful act, but so common today.

 

[Dribble]

Is clear logical thinking so hard to expect? No one has claimed what you write. You are the one claiming the essential, impossible without, argument. ML AI image improvement can't be done effectively without Tensor/matrix cores. That is what YOU said. The opposing argument says no, general purpose shader cores can be substituted. You are the one making ridiculous claims and then accusing others of exactly what you are doing. What a shameful act, but so common today.

Quote where I said that?

 

[Justinus]

Quote where I said that?

to equal DLSS they need the machine learning version which firstly requires AMD to come out with some cards with the dedicated machine learning hardware