Discussion AMD Gaming Super Resolution GSR

[DisEnchantment]

New Patent came up today for AMD's FSR

Patent Public Search | USPTO

appft.uspto.gov

https://www.freepatentsonline.com/y2021/0150669.html

20210150669 GAMING SUPER RESOLUTION

Abstract
A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display

[0008] Conventional super-resolution techniques include a variety of conventional neural network architectures which perform super-resolution by upscaling images using linear functions. These linear functions do not, however, utilize the advantages of other types of information (e.g., non-linear information), which typically results in blurry and/or corrupted images. In addition, conventional neural network architectures are generalizable and trained to operate without significant knowledge of an immediate problem. Other conventional super-resolution techniques use deep learning approaches. The deep learning techniques do not, however, incorporate important aspects of the original image, resulting in lost color and lost detail information.

[0009] The present application provides devices and methods for efficiently super-resolving an image, which preserves the original information of the image while upscaling the image and improving fidelity. The devices and methods utilize linear and non-linear up-sampling in a wholly learned environment.

[0010] The devices and methods include a gaming super resolution (GSR) network architecture which efficiently super resolves images in a convolutional and generalizable manner. The GSR architecture employs image condensation and a combination of linear and nonlinear operations to accelerate the process to gaming viable levels. GSR renders images at a low quality scale to create high quality image approximations and achieve high framerates. High quality reference images are approximated by applying a specific configuration of convolutional layers and activation functions to a low quality reference image. The GSR network approximates more generalized problems more accurately and efficiently than conventional super resolution techniques by training the weights of the convolutional layers with a corpus of images.

[0011] A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display.

[0012] A processing device is provided which includes memory and a processor configured to receive an input image having a first resolution. The processor is also configured to generate a plurality of non-linear down-sampled versions of the input image via a non-linear upscaling network and generate one or more linear down-sampled versions of the input image via a linear upscaling network. The processor is also configured to combine the non-linear down-sampled versions and the one or more linear down-sampled versions to provide a plurality of combined down-sampled versions. The processor is also configured to convert the combined down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution by assigning, to each of a plurality of pixel blocks of the output image, a co-located pixel in each of the combined down-sampled versions and provide the output image for display.

[0013] A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display.

It uses Inferencing for upscaling. As will all ML models, how you assemble the layers, what kind of parameters you choose, which activation functions you choose etc, matters a lot, and the difference could be night and day in accuracy, performance and memory

 

[Frenetic Pony]

If this works like the review purports, it's a win for everyone. Developers can target one implementation, tons of nvidia cards are supported and obviously AMD gets a big feather in their cap for showing that they could pull it off and for having "feature parity" with DLSS, which so many outlets absolutely obsess over. Gamers can tick a box and most won't be able to tell the difference in IQ most of the time but they'll notice framerates up over 60FPS.

Also, guaranteed support on a huge breadth of titles because of the way it can help extend the console lifecycle and let more developers release "60 FPS/4K/RT" games.

Sadly I doubt the old consoles are getting any more titles, it's not just technical hurdles and the fact that FSR2 wouldn't run that well on Xbox One/PS4 anyway. But that by the end of the year new gen will have an install base of like 50 million and developers just don't want to deal with supporting two and a half console generations alongside PC. It's why Gotham Knights just dropped previous gen consoles certainly.

Still, it'll be cool for any remaining devs not using UE5

 

[Panino Manino]

Sadly I doubt the old consoles are getting any more titles, it's not just technical hurdles and the fact that FSR2 wouldn't run that well on Xbox One/PS4 anyway. But that by the end of the year new gen will have an install base of like 50 million and developers just don't want to deal with supporting two and a half console generations alongside PC. It's why Gotham Knights just dropped previous gen consoles certainly.

Still, it'll be cool for any remaining devs not using UE5

Would be possible that the PS4Pro having a GPU based on Polaris and packed 16bit would support it, but the One X having a GPU based on Hawaii (I think) would not?

 

[blckgrffn]

Sadly I doubt the old consoles are getting any more titles, it's not just technical hurdles and the fact that FSR2 wouldn't run that well on Xbox One/PS4 anyway. But that by the end of the year new gen will have an install base of like 50 million and developers just don't want to deal with supporting two and a half console generations alongside PC. It's why Gotham Knights just dropped previous gen consoles certainly.

Still, it'll be cool for any remaining devs not using UE5

Oh no, I meant the RDNA2 consoles. They'll enjoy longer lives if they can maintain some semblance of parity with PCs longer.

The PS4 and last gen xbox are dead to me

 

[Panino Manino]

Oh no, I meant the RDNA2 consoles. They'll enjoy longer lives if they can maintain some semblance of parity with PCs longer.

The PS4 and last gen xbox are dead to me

Talking about consoles, this would be excellent for Nintendo right?
Their next Switch could have an DLSS equivalent without paying Nvidia for the extra hardware.

 

[moinmoin]

Their next Switch could have an DLSS equivalent without paying Nvidia for the extra hardware.

Why the next Switch? Not perfectly sure but FSR2 should run on the current Switch as well, so any new Switch game could make use of it.

 

[blckgrffn]

FSR2 has some real hardware requirements, and the Switch is seriously approaching potato status vs current gen consoles and steam deck. I am not holding my breath on that at all.

Now, the replacement that comes out in a couple years? That's probably a great priority for better scaling from mobile screen resolution to 4K TV resolution. It's unlikely a handheld in this decade will really confidently push 4K the way the big consoles do, but really well upscaled 1080p/1440p? That seems way more realistic.

 

[Panino Manino]

Why the next Switch? Not perfectly sure but FSR2 should run on the current Switch as well, so any new Switch game could make use of it.

But a new Switch will eventually happen.
Not that Nintendo have this Open Source alternative for free they will not have to pay extra for Nvidia to make a GPU with Tensor cores. Tensor cores can help do amazing things, but just to upscale games? it's unnecessary cost. Nintendo can now ask for a smaller and cheaper GPU or one with a bit more horsepower.

 

[moinmoin]

the Switch is seriously approaching potato status vs current gen consoles and steam deck.

That always had been true for the CPU part. The GPU is pretty competitive still considering the form factor.

 

[blckgrffn]

But a new Switch will eventually happen.
Not that Nintendo have this Open Source alternative for free they will not have to pay extra for Nvidia to make a GPU with Tensor cores. Tensor cores can help do amazing things, but just to upscale games? it's unnecessary cost. Nintendo can now ask for a smaller and cheaper GPU or one with a bit more horsepower.

My money is on Samsung ARM CPU with RDNA2 graphics for next gen Switch. The licensing and IP is already worked out and going with RDNA2 seriously streamlines their cross platform ability. Nintendo backwards compatibility tends to run on the "you can just buy it again" principle" - and it works for them.

That always had been true for the CPU part. The GPU is pretty competitive still considering the form factor.

I mean, it helps to have almost no direct competition. I think a case could be made that its weaker than every ipad and iphone currently being sold, it's just got buttons and actual games.

 

[Stuka87]

But a new Switch will eventually happen.
Not that Nintendo have this Open Source alternative for free they will not have to pay extra for Nvidia to make a GPU with Tensor cores. Tensor cores can help do amazing things, but just to upscale games? it's unnecessary cost. Nintendo can now ask for a smaller and cheaper GPU or one with a bit more horsepower.

Its already rumored that BotW 2 will be the launch title of the new Switch (and I swear, they better not call it 'New Switch' like they did with the DS)

 

[blckgrffn]

Its already rumored that BotW 2 will be the launch title of the new Switch (and I swear, they better not call it 'New Switch' like they did with the DS)

Switch U to confuse everyone and drop the ball hard. It'll have some weird 2nd screen accessory or required motion control for all launch titles.

They can only succeed every other generation.

I want Astral Chain and Bayo 3 to be on next gen hardware. FSR could help a lot.

 

[Saylick]

TPU/Hardware Unboxed just released a YouTube video review of FSR2.0 but I found this slide particularly interesting.

More testing is needed but it looks like FSR2.0 minimums are better than DLSS minimums. I'd like to see this testing across more games to see if this trend is a fluke or not, but if FSR2.0 actually provides better minimums, that's a bonus.

Edit: Ah, might have gotten ahead of myself. The trend reverses at 1440p. Perhaps it has more to do with the resolution as well.

 

[uzzi38]

FSR2 has some real hardware requirements, and the Switch is seriously approaching potato status vs current gen consoles and steam deck. I am not holding my breath on that at all.

Now, the replacement that comes out in a couple years? That's probably a great priority for better scaling from mobile screen resolution to 4K TV resolution. It's unlikely a handheld in this decade will really confidently push 4K the way the big consoles do, but really well upscaled 1080p/1440p? That seems way more realistic.

Ignore the h/w requirements on AMD's site, FSR2.0 works on all hardware that can run the necessary APIs to run whatever game.

There would need to be a port to the Switch's rendering API(s), but that's it.

 

[uzzi38]

Over down on Beyond3D Forums, Nick Thibieroz posted why exactly FSR 2.0 is launching with a single game title:

AMD FSR antialiasing discussion

TSR has quite terrible ghosting and the end presentation always seemed way too blurry in motion. I never understood why people seemed to like it so much. Ratchet and Clank's TAAU is perhaps the only one I would say is great? It really wasn't a given that FSR 2 would end up as well as it did. It...

forum.beyond3d.com

If you don't know who that is, then: https://twitter.com/NickThibieroz?t=yDmgN_llhnVgLubTEKyd9g&s=09

 

[Frenetic Pony]

Ignore the h/w requirements on AMD's site, FSR2.0 works on all hardware that can run the necessary APIs to run whatever game.

There would need to be a port to the Switch's rendering API(s), but that's it.

"Switch"-ing (not sorry) from Nvidia to AMD for GPU compatibility wouldn't be the easiest thing. But as the Skyward Sword port shows doing things like emulating previous GPU architecture in realtime is possible. Nintendo could certainly make an official emulator for backwards compat for GPU instructions.

Though I'd suspect upscaling like FSR2 might be more useful in "docked" mode, as handheld would probably have a relatively low res screen (maybe 900p?) and the overhead for using fancy upscaling just wouldn't be worth it.

Really though if I were designing it, more important to have 1080p so then you can sell a VR kit it slots into, imagine Nintendo having the first real VR mass market success (Skyward Sword VR anyone? How about Metroid VR ). Just add stuff like actual storage so you can download games and maybe the PS5's cool haptic feedback stuff, to a redesigned controller set obviously, and you've got yourself a highly successful Switch 2. (New faster hardware is a given)

 

[Aapje]

Over down on Beyond3D Forums, Nick Thibieroz posted why exactly FSR 2.0 is launching with a single game title:

Deathloop also has one of the best DLSS implementations, so the studio just seems to be very enthusiastic about upscaling, being willing to put in more effort than others.

The attention they are now getting seems like a just reward.

For other game studios it also has advantages to lag behind a bit, since then they don't have to deal with bugs as much, although being early can be great marketing.

 

[uzzi38]

"Switch"-ing (not sorry) from Nvidia to AMD for GPU compatibility wouldn't be the easiest thing. But as the Skyward Sword port shows doing things like emulating previous GPU architecture in realtime is possible. Nintendo could certainly make an official emulator for backwards compat for GPU instructions.

Though I'd suspect upscaling like FSR2 might be more useful in "docked" mode, as handheld would probably have a relatively low res screen (maybe 900p?) and the overhead for using fancy upscaling just wouldn't be worth it.

Really though if I were designing it, more important to have 1080p so then you can sell a VR kit it slots into, imagine Nintendo having the first real VR mass market success (Skyward Sword VR anyone? How about Metroid VR ). Just add stuff like actual storage so you can download games and maybe the PS5's cool haptic feedback stuff, to a redesigned controller set obviously, and you've got yourself a highly successful Switch 2. (New faster hardware is a given)

I... never said anything about switching to AMD hardware though. I don't believe that will happen either, but that doesn't prevent FSR from being used over there. Or rather, there's already a game that uses FSR 1.0 on the Switch, although I forget which.

You don't need to own AMD hardware to use FSR, so I have no clue what you're talking about when you say that.

 

[moinmoin]

Or rather, there's already a game that uses FSR 1.0 on the Switch, although I forget which.

Nintendo Switch Sports listed FidelityFX and FSR 1.0 licenses among its licensing info of used external software:

Nintendo Switch Sports to support AMD FidelityFX Super Resolution on the Switch's Nvidia Tegra X1 SoC

The soon-to-launch Nintendo Switch Sports game will likely feature AMD FidelityFX Super Resolution (FSR) image upscaling when the Switch is connected to a 1080p display. Evidence for this was found in the EULA of the Switch Sports game that is slated to launch in the coming weeks.

www.notebookcheck.net

Considering that's Nintendo itself we can expect other developers to have done so already as well.

 

[Panino Manino]

Makes no sense for Nintendo to abandon Nvidia so fast. I'm sure that the "Switch Family" will all use Nvidia and it's platform and APIs.

The only problem for Nintendo to use FRS2.0 and is how good the Nvidia GPU can do low precision packed math, because the performance gains seems heavy dependent on this:

 

[blckgrffn]

What the Nintendo does in the future is probably not a this thread item, I just don't think Nintendo cares about the technology as much as they care about their form factor and making a profitable console. I don't think Nintendo would blink at "abandoning" nvidia if they thought a solution met their first two needs.

They've already start talking about how they can possibly get people to transition to their next console as a high risk item, and given nvidia hasn't done anything to produce a successor to the SoC in the Switch, it seems like they are going to have to really up the ante and aren't going to be shackled by backwards compat, as that just gives them another reason to sell more games.

Given if they back FSR into their titles they can use it in the future regardless, that's just another reason that a hardware agnostic approach his better for everyone.

 

[Panino Manino]

What the Nintendo does in the future is probably not a this thread item, I just don't think Nintendo cares about the technology as much as they care about their form factor and making a profitable console. I don't think Nintendo would blink at "abandoning" nvidia if they thought a solution met their first two needs.

They've already start talking about how they can possibly get people to transition to their next console as a high risk item, and given nvidia hasn't done anything to produce a successor to the SoC in the Switch, it seems like they are going to have to really up the ante and aren't going to be shackled by backwards compat, as that just gives them another reason to sell more games.

Given if they back FSR into their titles they can use it in the future regardless, that's just another reason that a hardware agnostic approach his better for everyone.

Using image reconstruction by default would allow Nintendo to continue using outdated hardware tech.
And seeing how much the Switch is selling even Nvidia will offer special conditions and treatment to keep their deal.

 

[Mopetar]

Makes no sense for Nintendo to abandon Nvidia so fast. I'm sure that the "Switch Family" will all use Nvidia and it's platform and APIs.

The only problem for Nintendo to use FRS2.0 and is how good the Nvidia GPU can do low precision packed math, because the performance gains seems heavy dependent on this:

Nintendo sticking with NVidia would be even more of a surprise. They don't have a great track record of working well with other companies and both Sony and Microsoft dropped them after a single product generation.

As other posters have pointed out, they could probably get some kind of custom design from AMD/Samsung whereas NVidia is more likely to just sell them something they already have made. Nintendo probably cares less than other companies about having the absolute latest technology, but there aren't really many strong arguments for them to need to stick with NVidia and there's a not a lot of reason for NVidia to devote resources to giving Nintendo anything special.

 

[Frenetic Pony]

Using image reconstruction by default would allow Nintendo to continue using outdated hardware tech.
And seeing how much the Switch is selling even Nvidia will offer special conditions and treatment to keep their deal.

Eh, that won't help Nintendo necessarily. With XeSS and FSR2 out every developer is going to be using reconstruction, bigger devs have been trying to get it right this entire past generation. So a Switch 2 with upscaling isn't going to be any nearer to a Series S with upscaling than both would be without.

Otherwise, yeah there does seem to be the hint that Nintendo is sticking with Nvidia. No fiddly backwards compatibility to deal with, and maybe Nvidia has learned a lesson about being difficult after MS and Sony went with AMD for two generations of consoles running (two and a half for MS).

 

[Panino Manino]

Eh, that won't help Nintendo necessarily. With XeSS and FSR2 out every developer is going to be using reconstruction, bigger devs have been trying to get it right this entire past generation. So a Switch 2 with upscaling isn't going to be any nearer to a Series S with upscaling than both would be without.

Otherwise, yeah there does seem to be the hint that Nintendo is sticking with Nvidia. No fiddly backwards compatibility to deal with, and maybe Nvidia has learned a lesson about being difficult after MS and Sony went with AMD for two generations of consoles running (two and a half for MS).

I never meant in the sense of Nintendo achieving parity or closing the gap with the competitor's hardware. Just that a Switch successor with that feature would be a better Switch.

 

[Dribble]

Makes no sense for Nintendo to abandon Nvidia so fast. I'm sure that the "Switch Family" will all use Nvidia and it's platform and APIs.

The only problem for Nintendo to use FRS2.0 and is how good the Nvidia GPU can do low precision packed math, because the performance gains seems heavy dependent on this:

If you are always going to do the upscaling in every game (locked console ecosystem) then DLSS is better - it's give both better quality and higher performance.