The PS3 will not allow you to uncheck 480p because that is the minimum resolution it supports when an HDMI cable is connected. If it was the composite cable that came with it, it would allow 480i.
The aliasing issue is a PS3 feature. There is no hardware scaler on the PS3 that I know of, so the monitor will upscale the 720p to whatever resolution it is using because the PS3 won't, in this case 1920x1200 or 1920x1080. Monitors do not generally have the most advanced scalers, with a couple of exceptions, so they just use an algorithm to scale the smaller resolution up to full, without smoothing. The 360 does this generally a bit better.
The best PC analogy I can use, is if you set your video card to perform GPU scaling vs. centred timings and scale up 1280x800 or something like that to full resolution using either the video card or the monitor.
When the video card's hardware scaler does it, the output is smoothed out, which is great in games, but not great for text. When the monitor does it, text is more legible/sharper, but games look more aliased. That's because the video card has a built in hardware scaler chip to upscale and smooth simultaneously, while the monitor just upscales, with zero smoothing. The only monitors that I know of that smooth are the NEC LCD2x90WUXI-BK and Gateway XHD3000, both which are over $1000.00.
TVs do this to an extent, as well as some noise reduction, but the big issue is that you are sitting close to the monitor, which does reveal otherwise unseen details. I have both a 360 and PS3 and I can tell you that while graphically both are good, the 360 handles upscaling better and smooths things out a bit more. I can also say that with most PS3 games outside of Gran Turismo Prologue and Metal Gear Solid GP, all other games only do 720p max, and some will default down to 480p if you turn off the 720p option. Only these two are 1080p.
Keep in mind, that you can play with display settings till the cows come home, but nothing is really going to improve things drastically. Just turn off any 1080i resolution. Monitors are notoriously bad at de-interlacing.
I find there is one setting reachable in the A/V control panel of BluRay in the PS3 (accessed by hitting the triangle button) that affects how black levels are set with a monitor. If you find you are losing dark details you can set the AV to be either limited range or Full range RGB out in the AV control panel. It works similarly to the "Full Range HDMI" control on the display settings in XMB, but this is something you have to specifically set for BluRay while playing a BluRay.
Also look in the advanced image settings of the monitor and see if you do get Fill, Aspect, 1:1 as options. You don't want the monitor stretching the image to 1920x1200, you want it at 16:9, 1080p.
Good luck.
Originally posted by: MrAK
ok I'll try to make it more understanding, ex: staircase scenerio, when a line is supposed to look straight, it just looks crooked. Since the PS3 doesn't have aliasing, what is the problem with the image? As for screen tearing, thank God I dont see that since everything runs smooth with the exception of the crooked edges *jagged/rough* thus crooked lines/angles
And when I change the output resolution in the PS3 display option I uncheck everything and leave 1080p check marked yet 420p is stuck at check marked, i cant seem to uncheck it for some unknown reason, why would that be so?
And I go to custom scaling, in the monitor's onscreen options and enabled fit aspect ratio, but the "crooked" problem is still there
SORRY for the somewhat negative reaction i gave out earlier, its just I am new to these kind of things when it comes to troubleshooting problems such as this