Better IQ: Downscaling (AMD+NV)

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
1. Introduction

Many games of the DX9 era and surprisingly some current DX11 titles as well don't support MSAA. To sum up the current situation:


  • Driver based MSAA-hacks only possible in DX9, but not for all games (e.g. GTA 4, Dead Island)
  • No possibility to force MSAA in DX10+ (e.g. Crysis 2 DX11)
  • Sometimes bad MSAA quality in DX10+ (e.g. Max Payne 3 DX11), doesn't combat shader aliasing
  • Postprocessing-AA (FXAA, SMAA, MLAA) can be blurry and often fail in motion.


There is another solution that works in almost every game, be it DX9/10/11/OpenGL, that doesn't blur and that combats edge aliasing, shader aliasing and transparency aliasing alike. Colloquially we call it "downsampling" (or more correctly downscaling).


2. What is downscaling and what can it look like?

When using downscaling, the image is rendered at a higher resolution than the one it will be displayed at. This leads to a significantly larger amount of data that a downscaling filter can use to smooth the image and reduce aliasing artifacts.
For example the image is rendered in 3840x2160, downscaled to 1920x1080 and then displayed. Below are some screenshot examples


MP3 1080p...............................................................MP3 3840x2160@1920x1080



GTA4 1680x1050...............................................................GTA4 3138x2100@1569x1050

3. Questions and answers



  • Q: What are the scenarios that downscaling is recommended for?
  • A: Games where no AA is possible and/or games where only FXAA/SMAA/MLAA seem insufficient.


  • Q: What do I need?
  • A:

    • A powerful graphics card or SLI/CF.
    • Current drivers and Fermi and Kepler cards are recommended.
    • Windows Vista or Windows 7 is required, Windows XP is not supported
    • Dual-link DVI for 60Hz with certain cards (see below)
    • In certain cases: A display that can run below 60Hz


  • Q: What resolutions are recommended?
  • A: Ideally, you will be running a resolution that is 4 times higher than normal, but there are intermediate steps. There are (within reason) no limits as to what resolutions you can create. However, a factor of 4 (2x2) and 2.25 (1.5x1.5) is recommended for best sharpness and quality.

    For example on a 1080p screen you can run:
    • 1920x1080 (native)
    • 2880x1620@1920x1080 (factor 1.5 per axis)
    • 3840x2160@1920x1080 (factor 2.0 per axis)


    If you want 2x2 and lack the graphics power to run 3840x2160, you could also run these:

    • 2560x1440@1280x720
    • 3360x2100@1680x1050



  • Q: Can downscaling damage my display or my graphics card?
  • A: No. The worst thing that could happen is that during a failed creation of a custom resolution, your display may go blank and you need to reboot (this won't happen when using R300+ drivers)


  • Q: Are there limits to the resolution and/or the refresh rate?
  • A: Yes, but this depends on your graphics card and/or on the connection. With most Fermi cards (400 and 500 series), you are limited to a maximum pixel clock of 165MHz (it may work a little bit above 165MHz, but image quality could degrade severely, see next question). This limits downscaling to 3840x2160 at a refresh rate below 60Hz. It has been observed that 60Hz are possible when using a card that has Pure Video 5 functionality like a GT520 and all Kepler cards (600 series):
    http://en.wikipedia.org/wiki/Nvidia_PureVideo#The_Fifth_Generation_PureVideo_HD
    This limit might also be void if using a Dual-link DVI connection usually found in larger displays (2560x1440 and above) that raises the maximum pixel clock above 165MHz. Other connection types (Display Port, HDMI) might do the trick as well (unconfirmed).



  • Q: My custom resolution is blurry?
  • A: This may happen when using intermediate "uneven" downscaling factors instead of 2x2 or 1.5x1.5 or when going above the 165MHz pixel clock.


  • Q: Does the graphics driver have any influence?
  • A: Yes. It has been observed that the process of creating a custom resolution is influenced by the driver version. For example, on pre-R300 drivers the image would not come back after a failed test, requiring a reboot. The maximum achievable refresh rate has also been known to vary between driver versions. Currently, R300 drivers are recommended.


  • Q: Are there compatibility issues?
  • A: Generally, no. But in some games the HUD and text on screen may become very small. Some games require a resolution to run at 60Hz. For certain configurations, this limits the achievable resolution.


Important for resolutions below 60Hz:


  • Q: Why doesn't my custom resolution show up in Windows or in my games?
  • Every display has an associated EDID (Extended display identification data) where all relevant information stored. This includes a minimum refresh rate. If you create a custom resolution with a refresh rate that is higher, it will not be available to Windows or applications. For example, my display by default has a minimum refresh rate of 56Hz which is why 3840x2160@1920x1080@50Hz will not show up. In this case you will have to lower this limit by editing the EDID as described below.

    Warning:
    The following advanced procedure isn't very difficult, but great care should be taken nonetheless!

    In this link you will see a video description of all 3 steps involved:
    http://www.oc-burner.de/ftp/Videos/Downsampling/EDID/EDID.html

    You need three small and free programs for this:
    1. softMCCS
    2. Phoenix EDID Designer
    3. Monitor Asset Manager
    You can find them here:
    http://www.mediafire.com/?d60sfcpaqq407m2

    A short summary:
    1. Read the EDID from the monitor driver
    2. Edit the Hz-limit (and only that!) in the EDID
    3. Create a new monitor driver file (.inf) with the modified EDID
    4. Install the new driver file via the device manager

    It is recommended that before you proceed you save the original EDID and create an unchanged monitor driver - just in case. If you like, you can create a restore point as well.

4. Instructions (Nvidia)

Note: You should run a current driver (R300+).


  1. Go to the Nvidia control panel and select "Change resolution" under the "Display" section. Select "Customize"
  2. Select "Create Custom resolution"
  3. Select "Manual"
  4. Create the resolution

    For smaller resolutions like 3360x2100@1680x1050 you might be able to just change the rendering resolution and don't have to touch anything else. But when you get the message "test failed", you probably will need to increase the value for the total horizontal pixels. This will in turn increase the pixel clock which as mentioned above should not go beyond 165MHz on most configurations. You will have to play a bit with the refresh rate and the total pixel values to get your desired result.

    For example, on my display, the LG Flatron IPS235P (1920x1080) I had to increase the horizontal total pixels from originally 2200 to 2800 and lower the vertical total pixels from 1125 to 1090 for 3840x2160@54Hz to work and to stay below 165MHz.


 
Last edited:

njdevilsfan87

Platinum Member
Apr 19, 2007
2,330
251
126
Is there a difference between doing this and just using SSAA because I thought this is what SSAA did?
 

djsb

Member
Jun 14, 2011
81
0
61
Is there a difference between doing this and just using SSAA because I thought this is what SSAA did?

The difference is that SSAA is an AA driver setting to output a frame the size of the screen, while this outputs a frame without AA at all and just lets the monitor's scaler do its thing. edit: as I read it, that is.
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
Is there a difference between doing this and just using SSAA because I thought this is what SSAA did?

There are differences. The SSAA we all use is SGSSAA, meaning SSAA with a sparse grid. Downscaling is akin to OGSSAA, SSAA with an ordered grid. OGSSAA is practically dead because of compatibility issues and because it is a less efficient method than SGSSAA.

But the main advantage of downscaling is the near perfect compatibility. Contrary to SGSSAA it doesn't require MSAA to work, it cannot collide with certain effects which could otherwise cause blur. It works absolutely everywhere (except for very old games that don't support such high resolutions) without prejudice.
 

KingFatty

Diamond Member
Dec 29, 2010
3,034
1
81
This seems to put a lot of trust in the monitor's ability to scale the image - I would imagine that the quality you get will have a big variation based on what kind of monitor you have.

It's just hard for me to rely on the monitor's abilities, I much prefer to use the card's scaling?
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
Honestly I still have not completely understood where the downscaling takes place:
In the GPU or in the monitor.

I'm inclined to say that it is a GPU/driver thing because the GT520 can do things that my GTX580 cannot, namely 3840x2160@60Hz - on the same monitor with the same cable, that is. It may have to do with the way the display driver addresses the monitor.
 

Anarchist420

Diamond Member
Feb 13, 2010
8,645
0
76
www.facebook.com
Downscaling doesn't reduce most edge aliasing because it's ordered grid. 4x RGSS is the best and it's pretty much perfect if you can get the texture LoD to be perfect (i.e., not blurry, but not shimmering). 4x RGSS with adjustable texture LoD was a creation of 3dfx, ironically. It's amazing that the best AA quality was available 12 years ago.
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
Downscaling doesn't reduce most edge aliasing because it's ordered grid. 4x RGSS is the best and it's pretty much perfect if you can get the texture LoD to be perfect (i.e., not blurry, but not shimmering). 4x RGSS with adjustable texture LoD was a creation of 3dfx, ironically. It's amazing that the best AA quality was available 12 years ago.

I agree that quality isn't the best, but in my opinion it is only meant for the "severe cases" where nothing else works.
 

borisvodofsky

Diamond Member
Feb 12, 2010
3,606
0
0
Maybe it's not popular because people don't really care?

This not realllly AA, it's more like a blur filter, akin to those hq filters from those emulators.
 

Cerb

Elite Member
Aug 26, 2000
17,484
33
86
2. What is downscaling and what can it look like?
You used Flickr (1600x900 != 1920x1080!), and no AA in the 1st pic (nobody that cares about jaggies is going to use no AA for longer than it wakes to either fix that problem, or find their receipt), so there are two factors making it basically impossible to tell by your screenshots.

The second looks OK, but it's hard to compare, already downscaled, and the 1st pic is not using any AA, making it a bit of a false comparison.

Downscaling doesn't reduce most edge aliasing because it's ordered grid. 4x RGSS is the best and it's pretty much perfect if you can get the texture LoD to be perfect (i.e., not blurry, but not shimmering). 4x RGSS with adjustable texture LoD was a creation of 3dfx, ironically. It's amazing that the best AA quality was available 12 years ago.
But the performance hit can be bad, so SSAA was left as they already had it, OG, I guess as a compatibility measure, and instead focused on MSAA variations.
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
Maybe it's not popular because people don't really care?

This not realllly AA, it's more like a blur filter, akin to those hq filters from those emulators.

No, sir, this is OGSSAA and there is no blur involved ;)

You used Flickr (1600x900 != 1920x1080!), and no AA in the 1st pic (nobody that cares about jaggies is going to use no AA for longer than it wakes to either fix that problem, or find their receipt), so there are two factors making it basically impossible to tell by your screenshots.

The second looks OK, but it's hard to compare, already downscaled, and the 1st pic is not using any AA, making it a bit of a false comparison.

I don't understand. The pics on the left is without AA, the ones on the right is using downscaling, so one can compare and see the antialiasing effect. You have to click on the screenshots in flickr to show in the original resolution. The ones from MP3 are mine, the ones from GTA4 are older and from a buddy of mine when he didn't have a 1080p display, only 1680x1050, thus the lower res.

When making screenshots, fraps records the original resolution, so 3840x2160 for the MP3 screenshot on the right for example. Now in order to show what you actually see on the screen, you have to downscale the screenshots yourself, otherwise you would just see the huge resolution without any AA. I used a Lanczos3 filter for that, though I'm not sure what downscaling filter the GPU uses.
 
Last edited:

Cerb

Elite Member
Aug 26, 2000
17,484
33
86
I don't understand. The pics on the left is without AA, the ones on the right is using downscaling, so one can compare and see the antialiasing effect. You have to click on the screenshots in flickr to show in the original resolution. The ones from MP3 are mine, the ones from GTA4 are older and from a buddy of mine when he didn't have a 1080p display, only 1680x1050, thus the lower res.
AHw1O.png
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
Now I get it, thanks.
What other hosting service could I use? Imgur won't let me upload past 2MB, imageshack is not exactly loved around here, what else is there?
 

BFG10K

Lifer
Aug 14, 2000
22,709
2,971
126
This is OGSS, and pure OGSS has a very poor IQ/performance ratio. It’s also too slow in newer games. For older games the combined AA modes work fine and look better as well.

For newer games with no AA, I’d honestly just use FXAA. It works everywhere and has a very good IQ/performance ratio, often running faster than 2xMSAA. And unlike MLAA, it doesn’t suffer from pixel blinking or excessive blurring.

I consider it a very usable implementation of a “poor man’s SSAA”.

When making screenshots, fraps records the original resolution, so 3840x2160 for the MP3 screenshot on the right for example. Now in order to show what you actually see on the screen, you have to downscale the screenshots yourself, otherwise you would just see the huge resolution without any AA. I used a Lanczos3 filter for that, though I'm not sure what downscaling filter the GPU uses.
This pretty much invalidates any kind of image quality analysis using screenshots of this method given screenshots can’t actually capture what it’s doing.
 

BFG10K

Lifer
Aug 14, 2000
22,709
2,971
126
somewhat off topic question: Can most 2560x1600 monitors actually go to 110 Hz at 1920x1200 with the utilities in the OP and through custom resolutions?
No, why would they? LCDs are fixed to 60/120Hz maximum and changing the resolution makes no difference to this upper limit.
 

boxleitnerb

Platinum Member
Nov 1, 2011
2,601
2
81
This is OGSS, and pure OGSS has a very poor IQ/performance ratio. It’s also too slow in newer games. For older games the combined AA modes work fine and look better as well.

For newer games with no AA, I’d honestly just use FXAA. It works everywhere and has a very good IQ/performance ratio, often running faster than 2xMSAA. And unlike MLAA, it doesn’t suffer from pixel blinking or excessive blurring.

I consider it a very usable implementation of a “poor man’s SSAA”.


This pretty much invalidates any kind of image quality analysis using screenshots of this method given screenshots can’t actually capture what it’s doing.

FXAA has a very good IQ/performance ratio...in stills. In motion it can fail pretty bad.
The screenshot thing is unfortunate, but there is no way to grab the screenshot after downscaling. You just have to try it out yourself ;)
 

BFG10K

Lifer
Aug 14, 2000
22,709
2,971
126
FXAA has a very good IQ/performance ratio...in stills. In motion it can fail pretty bad.
I’ve been testing it extensively in a range of modern titles and I’ve haven’t seen any major issues with it, unlike MLAA which is almost universally unusable.

Sure, it’s not as good as 2xTrSS (especially for alpha textures) but it still provides a big improvement over no AA. It’s also extremely fast, even at very high resolutions.

IQ is very good considering it’s a shader post-filter, and it’s effective at getting AA into situations where none is possible due to compatibility and/or performance reasons.

The screenshot thing is unfortunate, but there is no way to grab the screenshot after downscaling. You just have to try it out yourself
I’ve been using OGSS modes since about 2004, so I’m very familiar with how they look. The fact is you need a minimum of 4x to provide usable increases to IQ, and you’ll drop to as low as 25% of your original performance. And even with such extreme punishment, your worst cases still only get 2xAA.

Like I said above, it’s too slow for new games and has inferior IQ compared to the combined modes, which work fine in older games. So it doesn’t really fit anywhere as having a practical use.