Until more knowledgeable people come along, I'll take a stabe (<<-- [Later] Holy cow! The correct spelling for word back there was censored!!!! Oh Brave new world!)
I just read your message over again, but am still not sure. Is the install failing right in the beginning when it tries to go into the (now default) graphical install mode, or later when it is trying to set the final resolution for the card and monitor?
This is a fairly new monitor right? I would have thought it would have been detected, so the first thing I would go is to get on google and search for your monitor +Linux. See if anyone else is having problems--and if you're lucky--a solution.
The next thing to try when you get to the "OUT OF SYNC RANGE," is hold down the Control and ALt keys while hitting the plus or minus key. This steps up (plus) or down (minus) through whatever table of resolution values Redhat has in its default setup file. That might get you going. If this didn't work (and maybe even if it did) proceed with the next steps.
If it is failing at the start, I would go ahead with the text install and then go back later to setup Xfree86. If later, then just tell it not to test it, BUT also don't tell the program to start you in graphical mode. (You'll have to go back later and change your /etc/inittab file to startup in graphical mode.)
Now, with the installation done, and rebooted in text mode, you can try to configure Xfree86. With RH8.0, the configuration program is now redhat-config-xfree86. Make certain you have your monitor specs around, mainly the ranges for the horizontal and vertical scan rates. There might also be an optimal resolution (usually the case on LCD screens)--maybe even a whole table of resolutions with refresh rates. Using the configuration program, you should be able to step through the monitor, mouse, keyboard, graphic card setup. I haven't used the program recently, so I'm not sure how well it picks up the correct info by itself. (Obviously not well enough or you wouldn't have the problem.)
In any case, when you are all done, go open up the XF86Config file to look at it. In spite of what many of the HOWTO say, the file is name XF86Config. RedHat 8.0 doesn't have a XF86Config-4 (for XFree86 4.x) and XF86Config (for earlier versions). It just has the XF86Config file (sitting in the /etc/X11 directory). Unless you know your way around this file, don't start editing, just make certain the monitor section corresponding to the name you gave your monitor (there can be many monitor sections) has the values that match what you saw in your manual.
If it looks OK, then use the startx command. If things aren't really screwed up, a failure will land you back on a command line (and scratching your head). If it works, see one of the HOWTOs on changing your inittab file. (I don't remember if the respawn line is simply commented out when the default runlevel is 3 or if you have to type in the whole line. Besides that, you should only have to change an earlier line from id:3:initdefault to id:5:initdefault).
Goog luck...and remember my first warning.