You will know you've gone too far by the smell of an electrical fire, rather distinct.
Seriously, i have a Sapphire 9500np > 9800 card with 3.6ns memory. Like those have suggested above, try and overclock the gpu and the memory separately to find there limits (usually the introduction of artifacts). I first left the memory at 270, and was able to overclock the gpu to around 385, no artifacts. Then I ran 3DMark2001 for around 1/2 an hour to get things heated up. Artifacts started to show. Then i backed off in 5mhz steps until no artifacts and then 5 more for a safety factor. I settled on 370 mhz for the GPU. I then turned the gpu back to 270mhz and quickly ran the memory up until artifacts and backed off slightly. To my supprise the 3.6ns memory would go to about 315mhz (not bad for memory that has a theoretical speed of 278). Again, fed the card a continous dose of 3DMark2001 to heat things up, backed off in increments and settled on 310mhz. Then, I ran all the GPU up to 370mhz and the memory up to 310mhz and gave it a dose of continuous 3DMark2001 all was ok for a while, but artifacts started to appear after a while. I then backed each one down separately to find out which was too high, and the memory proved to be the culprit. Set the GPU to 370 and the memory to 305 and all has been well ever since.
Just my two cents, but when overclocking anything, I try to overclock just one parameter at a time leaving all other in their stock state. Then test them all at the same time after I've have reached each individual limit. Then tinker from there if there are any problems running in combination.