First of all what type of ram where you using? Brand and rating...
The known issue that Ive seen before is when the default speed the motherboard sets for the ram is incompatible, AND you overclock too far.
When you overclock too far, and have to set it back to the default it doesnt boot, and you need to put a stick of ram in that doesnt have that issue.
Thats basically it, you should be able to clock up to find your limit for all three sticks. Keep in mind, the weakest link makes the chain fail. If you test each stick individually you can find out what each one overclocks to, but with all three in at once, you cant run any faster than the slowest stick.
So then its a judgement call, if you need the 768 ram, you just go a bit slower on it. If you can handle 256, pop in the fastest one and fly.