The scaling usually is done through the card, but as far as I know, the driver is limiting that somehow. For example with the same monitor, with a GT520 (!) (which has a VP5, maybe that is important) 3840x2160@60 Hz is perfectly possible. Put in a GTX580 and it suddenly is not. It's still some mystery I have to admit.
What happens when you select 2880x1800@60 Hz? You have to increase the total pixels (vertical) for higher resolutions to work. Standard on my screen is 2200 but as you see I'm using 2800 for 4K. Also you can lower total pixels (horizontal) to the bare minimum (in my case from 1125 to 1090) to get more headroom. Under no circumstances can you go past the 165 MHz DVI limit. If I reach any value above 165 MHz, I get a black screen or the test fails.
FRAPS sees the pre-downscaling image, yes. If you want to make screenshots that show what you're actually seeing on screen you have to downscale the screenshot manually, for example with FastStone Photo Resizer and a Lanczos filter.
SGSSAA is pretty straight forward. Basically there are two cases:
a) Game supports MSAA. You select SGSSAA via NV Inspector and adjust the LOD accordingly (-0.5 for 2xSGSSAA, -1.0 for 4xSGSSAA and -1.5 for 8xSGSSAA). LOD adjustment currently only works in DX9. DX10/11 support is coming with R310 drivers afaik.
Example (8xMSAA ingame selected):
b) Game doesn't support MSAA or you want to reduce blur (which can happen with SGSSAA when it collides with post processing effects). Then you would have to use AA compatibility bits.
Example 1 (forced 4xSGSSAA):
Note that I have forced MSAA which is required for SGSSAA to work. Forcing MSAA only vs. SGSSAA can require different compatibility bits.
Example 2:
Just selecting SGSSAA works, but it's blurry. Custom compatibility bits reduce/disable the blur. Sometimes this also can result in better smoothing of certain edges but with increased cost in VRAM and performance.
OGSSAA as a traditional AA method via the driver is basically dead nowadays. I would always use SGSSAA when possible, even with older games. If you want to give it a try anyway (only DX9 and older obviously as since DX10 it is not possible to force AA modes anymore) select one of these modes:
Comprehensive list of AA compatibility bits for MSAA and SGSSAA:
http://www.forum-3dcenter.org/vbulletin/showthread.php?t=490867
As for the front porch, sync and other values, I have no idea. The only thing I change is the target resolution at the top, the refresh rate and the total pixels (horizontal and vertical).
A note on the refresh frequency:
Some monitors (for example my LG IPS235) usually don't support much lower frequencies than 60 Hz. In my case, 56 Hz was the default limit and I could not use 3840x2160@54 Hz. I had to make a modified monitor driver that exposes lower frequencies. A video tutorial can be found here:
http://www.oc-burner.de/ftp/Videos/Downsampling/EDID/EDID.html
You will need three tools for that:
1. SoftMCCS
2. Phoenix EDID Designer
3. Monitor Asset Manager
All three are freeware and can be easily found on the web. Just follow the tutorial and you'll be fine. It's pretty straight forward.