Did you touch the heatsink and was it actually that hot? Maybe the thermometer is off. If it is that hot, look at cooling it down (check the thermal paste, add a fan), as temps that high will contribute to instability.
Yes, increasing the FSB speed will cause system instabilities just like overclocking any other component. Just like you need to add voltage when you increase the frequency of the CPU, you also need to add voltage to the northbridge (MCH voltage) when you increase the frequency of the northbridge to stabilize it.
At this point, I think you'd want to start "dividing and conquering" in order to decrease variables and find weak points in your overclock. By this I mean that since you're forced to overclock your RAM when you increase CPU frequency past 3.2GHz, find out just how far you can stably overclock your RAM by removing the CPU from the picture. Do this by simply decreasing your multiplier to 6x while keeping voltage the same. Providing your temps on your northbridge aren't as bad as they seem (touch test is crucial

), give it some MCH voltage (+0.2V is usually sufficient) and start increasing your RAM. Using a bootable USB key with Memtest86+ on it is a quick way to test stability (boot, let it run for five minutes, if no errors, reboot, increase frequency, run again, etc.):
http://www.memtest.org/ . Give your RAM up to 2.0V for daily use and see how high you can take it. After you have a ball park range of its capabilities, consider them when overclocking your CPU so that all components are working within their capabilities.
If your video card can handle your games fine, don't upgrade. If you feel like you want to push it further, both the GTX 260 216 and HD 4890 are great bang-for-your-buck solutions that will be significant upgrades over an 8800GTS 320MB