Discussion AMD Gaming Super Resolution GSR

DisEnchantment

Golden Member
Mar 3, 2017
1,590
5,722
136
New Patent came up today for AMD's FSR




20210150669
GAMING SUPER RESOLUTION

Abstract
A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display


[0008] Conventional super-resolution techniques include a variety of conventional neural network architectures which perform super-resolution by upscaling images using linear functions. These linear functions do not, however, utilize the advantages of other types of information (e.g., non-linear information), which typically results in blurry and/or corrupted images. In addition, conventional neural network architectures are generalizable and trained to operate without significant knowledge of an immediate problem. Other conventional super-resolution techniques use deep learning approaches. The deep learning techniques do not, however, incorporate important aspects of the original image, resulting in lost color and lost detail information.

[0009] The present application provides devices and methods for efficiently super-resolving an image, which preserves the original information of the image while upscaling the image and improving fidelity. The devices and methods utilize linear and non-linear up-sampling in a wholly learned environment.

[0010] The devices and methods include a gaming super resolution (GSR) network architecture which efficiently super resolves images in a convolutional and generalizable manner. The GSR architecture employs image condensation and a combination of linear and nonlinear operations to accelerate the process to gaming viable levels. GSR renders images at a low quality scale to create high quality image approximations and achieve high framerates. High quality reference images are approximated by applying a specific configuration of convolutional layers and activation functions to a low quality reference image. The GSR network approximates more generalized problems more accurately and efficiently than conventional super resolution techniques by training the weights of the convolutional layers with a corpus of images.

[0011] A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display.

[0012] A processing device is provided which includes memory and a processor configured to receive an input image having a first resolution. The processor is also configured to generate a plurality of non-linear down-sampled versions of the input image via a non-linear upscaling network and generate one or more linear down-sampled versions of the input image via a linear upscaling network. The processor is also configured to combine the non-linear down-sampled versions and the one or more linear down-sampled versions to provide a plurality of combined down-sampled versions. The processor is also configured to convert the combined down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution by assigning, to each of a plurality of pixel blocks of the output image, a co-located pixel in each of the combined down-sampled versions and provide the output image for display.

[0013] A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display.

It uses Inferencing for upscaling. As will all ML models, how you assemble the layers, what kind of parameters you choose, which activation functions you choose etc, matters a lot, and the difference could be night and day in accuracy, performance and memory

1621500232759.png

1621500205339.png
 
Last edited:

GodisanAtheist

Diamond Member
Nov 16, 2006
6,719
7,016
136
Interesting!

So If I am understanding this correctly, AMD's implementation is still centered around a trained neural network, it will simply use additional "non-linear" data in its upscaling techniques and the algo will be run entirely on shaders since AMD's cards don't have tensor cores sitting around.

I wonder if this technique will actually end up benefitting NV even more, thanks to the vast but inefficiently used shader array on their cards.

Hell, now I'm starting to wonder if the shader doubling was so they could ditch tensor cores on their next arch design, bring DLSS 3.0 over to shader processing, and gain that space back...
 

coercitiv

Diamond Member
Jan 24, 2014
6,151
11,686
136
Hell, now I'm starting to wonder if the shader doubling was so they could ditch tensor cores on their next arch design, bring DLSS 3.0 over to shader processing, and gain that space back...
How are you able to asses the compute requirements of DLSS based on the compute requirements of another image reconstruction technique?
 

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
As people have mentioned elsewhere, AMD filed for this patent in the end of 2019, roughly 2-3 months before DLSS 2.0 was even announced. I'm quite convinced the actual implementation of GSR has evolved a lot since then. Possibly to the point of having nothing anything in common with this patent.

Somebody on reddit also mentioned that this seems to require per-game training and all-in-all looks more similar to DLSS 1.
 

Shamrock

Golden Member
Oct 11, 1999
1,438
558
136
RedGamingTech said today, that FSR will almost certainly be able to be used at least back to Vega, and maybe even Polaris!

 

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
RedGamingTech said today, that FSR will almost certainly be able to be used at least back to Vega, and maybe even Polaris!
[RANT warning]
I hate it when people attribute to "sources" and "leaks" stuff that AMD reps themselves has said in interviews.

There was a long interview last year (I think it was this one) where both Scott Herkelman and Frank Azor essentially said that while they initially they something that would only work on their cards, game devs really wanted something platform-agnostic (a la Freesync) they could only implement once so they decided to go that way. (and if I remember correctly Frank Azor even confirmed it in his interview with the very same RedGamingTech not 2 weeks ago)

The gamedevs can only implement it "once" if it works on Nvidia. AMD also wants it to work on consoles as well, so it must be API agnostic as well. All of this means that it 99% works on Vega which has rapid-packed math. The only reason it might not work on polaris is if the performance hit (running some 4bit calculations) is too much.
[/RANT]

So essentially getting info from "top secret sources" that the top brass of the company confirmed months ago.

At least RedGamingTech actually seems to be quite decent compared to the disasters of MLID or Adored.
 
Last edited:

DisEnchantment

Golden Member
Mar 3, 2017
1,590
5,722
136
Somebody on reddit also mentioned that this seems to require per-game training and all-in-all looks more similar to DLSS 1.
It may be.
But, from a pure ML modelling perspective, it is hard to tell how the training is done.
The description in the patent is so generic that you can just say we assembled the network in some special way and select some specific activation function and achieve better results. Thats what the patent is about which basically means anything from ML perspective.
When doing ML modelling for a specific problem, it is not uncommon that you eventually figure out a more deterministic way to represent the problem as an actual mathematical model and this happens regularly in highly automated driving.

So essentially getting info from "top secret sources" that the top brass of the company confirmed months ago.
Video Thumb is actually Exclusive info from USPTO :)
 
  • Like
Reactions: Tlh97 and Leeea

Dribble

Platinum Member
Aug 9, 2005
2,076
611
136
The patent looks very like DLSS, I suspect because AMD want some protection if they make a DLSS variant in the future (i.e. something that needs the AMD equivalent of tensor cores). Doesn't mean that's what FSR on the current cards will be.
 
  • Like
Reactions: Leeea and Tlh97

VirtualLarry

No Lifer
Aug 25, 2001
56,227
9,990
126
So If I am understanding this correctly, AMD's implementation is still centered around a trained neural network, it will simply use additional "non-linear" data in its upscaling techniques and the algo will be run entirely on shaders since AMD's cards don't have tensor cores sitting around.
This... talk of "non-linear" scaling... intrigues me.

Remember back when, CD-ROM was a new thing? Well, there was an early title, called "Compton's Multi-Media Encyclopedia" on CD-ROM. In order to fit the "Multi-Media" onto the disc, they used compression. A fairly unique type of it, it's called "Fractal Compression" (look it up, it's patented, or was). The really unique thing about "Fractal Compression", was that, once the original image was compressed that way, the resultant output when de-compressed, could easily and probably nearly infinitely scaled-up in resolution. It was really revolutionary technology, if a bit obscure.

I really wonder if AMD is now essentially doing those steps IN REAL-TIME on the shader processors. That would be, to me, quite frankly rather amazing, if true. It was a fairly compute-intensive process, as I understand it, to fractally-compress an image file.
 
  • Like
Reactions: Tlh97 and Leeea

Stuka87

Diamond Member
Dec 10, 2010
6,240
2,559
136
This... talk of "non-linear" scaling... intrigues me.

Remember back when, CD-ROM was a new thing? Well, there was an early title, called "Compton's Multi-Media Encyclopedia" on CD-ROM. In order to fit the "Multi-Media" onto the disc, they used compression. A fairly unique type of it, it's called "Fractal Compression" (look it up, it's patented, or was). The really unique thing about "Fractal Compression", was that, once the original image was compressed that way, the resultant output when de-compressed, could easily and probably nearly infinitely scaled-up in resolution. It was really revolutionary technology, if a bit obscure.

I really wonder if AMD is now essentially doing those steps IN REAL-TIME on the shader processors. That would be, to me, quite frankly rather amazing, if true. It was a fairly compute-intensive process, as I understand it, to fractally-compress an image file.

I might still have that sitting around here someplace. It helped in many high school reports!


The patent looks very like DLSS, I suspect because AMD want some protection if they make a DLSS variant in the future (i.e. something that needs the AMD equivalent of tensor cores). Doesn't mean that's what FSR on the current cards will be.

The chances of AMD ever going with proprietary hardware are less than one of us being struck by lightning. There is no need for custom hardware for upscaling (we have been through this already), and there is zero reason they would come up with something that would not work on the consoles.
 
  • Like
Reactions: Mopetar and Tlh97

Dribble

Platinum Member
Aug 9, 2005
2,076
611
136
The chances of AMD ever going with proprietary hardware are less than one of us being struck by lightning. There is no need for custom hardware for upscaling (we have been through this already), and there is zero reason they would come up with something that would not work on the consoles.
definition of proprietary:
relating to an owner or ownership.
"the company has a proprietary right to the property"

That would be everything in an AMD gpu being as it's all patented.

The whole gpu is custom (which for some reason you called proprietary) hardware - it's all specialised to do certain tasks (draw, texture, etc). This is no different - having hardware specialised for machine learning tasks such as image manipulation is to be expected because it's the most efficient - even phones have it.
 
  • Like
Reactions: Tlh97

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
While not AMD's solution Unreal Engine's built-in Temporal Super Resolution looks quite cool:


And it looks to be optimized for RDNA2 as well

before. To meet these demands, we've written a Temporal Super Resolution algorithm from scratch that replaces UE4's TemporalAA for higher-end platforms.

Temporal Super Resolution has the following properties:

  • Output approaching the quality of native 4k renders at input resolutions as low as 1080p, allowing for both higher framerates and better rendering fidelity.
  • Less ghosting against high-frequency backgrounds.
  • Reduced flickering on geometry with high complexity.
  • Runs on any Shader Model 5 capable hardware: D3D11, D3D12, Vulkan, PS5, XSX. Metal coming soon.
  • Shaders specifically optimized for PS5's and XSX's GPU architecture.

While It's obviously a WIP It still looks significantly better than UE's out of the box TAA.

Unfortunately the comparison images are rather useless as Nanite is resolving geometry to fit a 1080p image. Still I'll post them here just for reference:

Ny2Vr8S.jpg

DIOOMj1.jpg

All in all it still looks WAY better than UE4's TAA IMO. Hopefully I get the time to make some more detailed images in the engine at some point ...
 

Kepler_L2

Senior member
Sep 6, 2020
308
977
106
TAA and DLSS are good for static images but have issues with motion. We won't know how TSR compares to either until games built with UE5 come out.

Also FSR is likely to be better than all 3 solutions at motion clarity but worse at static images. Will be interesting to see what people prefer.
 

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
TAA and DLSS are good for static images but have issues with motion. We won't know how TSR compares to either until games built with UE5 come out.

Also FSR is likely to be better than all 3 solutions at motion clarity but worse at static images. Will be interesting to see what people prefer.

That is true for FSR. UE5 TSR can be checked out right now (and it's quite different from UE4 TAA), though it's WIP. Just download the Early Access Unreal engine and load the Valley of the Ancient demo. TSR is enabled by default, but you can easily disable it. From there on out just capture your screen and do comparisons.


I'll try to get to it myself some day when I'm not stomped by work.

EDIT:

And while the terrible JPG resolution and Nanite trying to optimze a polygon per pixel (meaning the upscaled image has near 4x less geometry detail) skews things, It's still pretty evident the new algorithm (TSR) is way better than the older one (TAA).

Here is a zoomed in crop of the screenshots shared above. I tried to focus on an area that should use similar mesh-quality (stuff further away). Unfortunately JPG artifacts totally murder the quality, but it isn't completely useless

TSR (1080p -> 4K)

Native 4K (20 FPS):
6tLpCl7.png


TSR upscaled from 1080p (44 FPS):
08phG5B.png


This looks pretty decent for a 2x upscale.

TAA (70% -> 100% equivalent to 1512p -> 4K upscaling)

70%
BK7hotW.png


100%
7RR5zzQ.png


Look at the poor detail resolving of the fence. Actually the entire image looks blurred in the fullscreen comparison despite the source resolution being WAY higher.

TSR should also be better in motion as they promise:
  • Less ghosting against high-frequency backgrounds.
  • Reduced flickering on geometry with high complexity.

I really hope to see a good Native vs TAA, TSR and DLSS (as Nvidia's plugin seems to be updated to work with UE5) )comparison in motion, preferrably side-by-side.
Nanite is configurable to produce the same level of detail and maps can be used where it's off.

I wish I had the time to do it, maybe someday
 
Last edited:

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
So I got around to play a little with the Unreal Engine 5 Early Access and the Ancient Valley demo

I tired out the new Unreal Engine Temporal Super Resolution (TSR) upsampling method (not to be confused with GSR) and compared it to it's original TAA upsampling and the Bilinear (naïve) method.

If It's not FidelityFX GSR why bother?
  • Because Unreal Engine is very widely used and it's TAA is the defacto standard for last-gen UE4 games using upsampling. The same will probably be true for TSR for next-gen UE5 games.
  • TSR is a lightweight algorithm (no Tensor units required) with shaders specifically optimized for PS5’s and XSX’s GPU architecture(source). It's a very good baseline for what GSR can and needs to accomplish.
  • It has some properties required for good upscaling that TAA doesn't have and GSR needs to have (temporal stability, less ghosting via motion-vector use) Here's what Epic has to say about it:
  • Output approaching the quality of native 4k renders at input resolutions as low as 1080p, allowing for both higher framerates and better rendering fidelity.
  • Less ghosting against high-frequency backgrounds.
  • Reduced flickering on geometry with high complexity.
  • Runs on any Shader Model 5 capable hardware: D3D11, D3D12, Vulkan, PS5, XSX. Metal coming soon.
  • Shaders specifically optimized for PS5's and XSX's GPU architecture.

Test Setup
All of the tests were run at 1440p on my home rig: Radeon 6800XT, Ryzen 3700X, 32GB of DRR4 @ 3600CL14

Caveats:
UE5 is still in early beta, and so are most of the algorithms used. Expect Nanite, Lumen and TSR to improve. There are still bugs particularily around character (just look this native 4K video around the character's hair). That's Lumen only. Once you add TSR at times you get additional ghosting on the character(well visible on my video with 480p -> 1440p).

Still Image Comparison:

ffW6YHU.jpg


35Cho5g.jpg


cg05Hdl.jpg
lnyy0zA.jpg


lOUn5VY.jpg


bcvTGNQ.jpg
1jAYtNo.jpg


66mt1HW.jpg


dIok6ED.jpg

xHj6qpD.jpg
FB8Qawv.jpg

Here is a collage (I added a downsampled 2880p version for good measure, to see if it makes any major difference to geometry due to how Nanite operates)
ag10XgB.png


Here is the entire image gallery in PNG format including some easter eggs (e.g. 288p retro campfire). The ones in the collage above are behind the "load more" button:


Video Comparison:

Youtube (a bit blurrier but with chapters):

Vimeo:


Here is a list of resolutions and methods I ran through* the console commands and approximate performance I got:

Resolution / Upscaling methodConsole commandAverage FPS (approx near the starting rock)
1440p Nativer.TemporalAA.Upsampling 0
r.ScreenPercentage 100
44 FPS
720p Bilinear (naïve method)r.TemporalAA.Upsampling 0
r.ScreenPercentage 50
85 FPS
720p TAAr.TemporalAA.Upsampling 1
r.TemporalAA.Algorithm 0
r.ScreenPercentage 50
81 FPS
720p TSRr.TemporalAA.Upsampling 1
r.TemporalAA.Algorithm 1
r.ScreenPercentage 50
79 FPS
1080p Bilinearr.TemporalAA.Upsampling 0
r.ScreenPercentage 75
61 FPS
1080p TAAr.TemporalAA.Upsampling 1
r.TemporalAA.Algorithm 0
r.ScreenPercentage 75
61 FPS
1080p TSRr.TemporalAA.Upsampling 1
r.TemporalAA.Algorithm 1
r.ScreenPercentage 75
58 FPS

* In the video I accidentally skipped vanilla 1080p (Bilinear) and played around with 480p + TSR in the end (r.ScreenPecentage 30)


Conclusion:
Overall TSR looks really really good IMO considering the circumstances.

In actual gameplay in motion it fixes most of the problems I have with TAA upsampling (e.g. why I can't stand it in Cyberpunk below 90%)

Positives:
+ Very small performance hit
+ No exotic hardware requirements (works even with Vega)
+ Excellent temporal stability and no flickering on faraway objects with complex geometry
+ Looks considerably better than TAA, particularily on the edges of faraway objects. 720p TSR sometimes even beats 1080p TAA

Negatives:
+ Still bugs and artifacts on moving objects/charactes
+ Nanite optimizes geometry to show 1 polygon per pixel. This means with upsampling scene geometry is considerably simplified (essentially 4x for 50% upscale)

Unfortunately I don't have a 4K screen, but from 1080p up the upscaling seems excellent (so 50% Screen Percentage for 4K should be very decent).

Even if AMD can't beat TSR with it's GSR it would still be a very good upscaling method (well worth it for consoles at least). If they manage to do even slightly better it's a true and honest DLSS competitor.

Overall, I'm quite impressed
 
Last edited:

Gideon

Golden Member
Nov 27, 2007
1,608
3,573
136
Well, it's coming already this month!


The not-so-good news is that it's indeed not using motion vectors at all (according to Ian Cutress it's per-frame only). The quality of the GTX 1060 screenshot also leaves a lot to be desired.

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".

I wonder how well it does with moving objects like thin wires. I'm afraid not so well. Imo they should just copy what UE5 did. It looks awesome in motion and has next to no performance hit.
 

exquisitechar

Senior member
Apr 18, 2017
655
862
136
The not-so-good news is that it's indeed not using motion vectors at all (according to Ian Cutress it's per-frame only). The quality of the GTX 1060 screenshot also leaves a lot to be desired.

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".
That's really a letdown after all the waiting.
 

Tup3x

Senior member
Dec 31, 2016
944
925
136
Based on the screenshots the lower resolution is very obvious. DLSS1 level at best. It should be very straight forward to implement but it does not have any further in UE5 for example.
 
Last edited:

CakeMonster

Golden Member
Nov 22, 2012
1,384
482
136
That could be disappointing, but I mean why complain when it will almost certainly be better than the old resolution scaling in-game anyway. I'm sure they'll work on it for some time, and probably improve quality, and if not, at least performance.
 

GodisanAtheist

Diamond Member
Nov 16, 2006
6,719
7,016
136
Imo, so long as these IQ reducing features don't get baked in and hidden from the end user or in some way cripple or prevent PC gamers from native res gaming, bring them all on.

My line with DLSS has always been if I need it I bought too little GPU, and when I should want it is to squeeze a bit more out of an aging card to get it up to a FPS number that's functional for me. I feel the same way about FSR or whatever other tech.

If this keeps AMD cards lean and mean in the future (free of hardware bloat) and fighting on the native res but can be touted as feature parity and makes them more competitive in all the benchmarks then I'm all for it.
 

ThatBuzzkiller

Golden Member
Nov 14, 2014
1,120
260
136
There's still potential to improve image quality for FSR if we use higher internal resolutions like 75% of native with a minimum output resolution of 1440p. FSR isn't going to work very well with low internal resolutions but the same applies to DLSS as well ...
 

Dribble

Platinum Member
Aug 9, 2005
2,076
611
136
That could be disappointing, but I mean why complain when it will almost certainly be better than the old resolution scaling in-game anyway. I'm sure they'll work on it for some time, and probably improve quality, and if not, at least performance.
I don't know - I would have thought the game engine teams will have spent more time working on good scaling then AMD - particularly for any engine that works on a console as they scale all the time. Would be interesting to compare UE5 built in scaling to this for example.
 

Heartbreaker

Diamond Member
Apr 3, 2006
4,222
5,224
136

From the Link:
" FSR is going to be a purely spatial upscaling technology; it will operate on a frame-by-frame basis, without taking into account motion data (motion vectors) from the game itself."

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".

Once you start requiring motion vectors, it then requires more developer work and more tie ins to the game engine, so less support. That and you actually need to do the work of developing a good algorithm, which can be a lot of work.

This is just a dumb simple spatial resize algorithm (AKA what we already had), likely with a bit extra work applied to the sharpening. Overall it's not much more than formalizing what AMD already does as DLSS competitor.

They are just hoping people will use it in reviews for comparing to DLSS for frame rates.
 
Last edited: