Are you looking to overclock just your memory, or your entire system?
Will you be replacing your PC5300, or adding to it?
I am going to assume that you want to overclock just your memory and that you will be adding to your existing memory.
Be aware that different memory does not always play well together, so your results may be underwhelming.
The first thing to do is to find the maximum speed of your current memory. Do this with only the PC 5300 installed. Go into bios and manually set your memory timings. I would start with loose timings, say 5-5-5-18-24-2T. Set your memory standard to DDR2 800. Put your CPU multiplier at 10. Set your CPU frequency at 170. Change HTT multipliers to 4. Set memory voltage (vdimm) at auto. Boot your computer.
Your CPU should now be running at 1700 mhz and your memory at 340 mhz. Run memtest to check for errors. If everything is okay then go into bios and raise your CPU frequency to 175. Boot and test, memory frequency should be 350. Keep going (max of 220 or you will be overclocking your CPU) until you get instability, this will be your maximum memory speed at loose timings with standard voltage.
Increasing memory voltage may allow you to reach a higher speeds, but can be dangerous. I would check the specs on your memory before doing so. Bear in mind that standard voltage is 1.8v and that performance PC6400 memory is rated 2.1 or 2.2v. Increase voltage at your own risk.
If you are running PC6400 at standard speed the memory would actually be at 366 mhz (2200/6) If your PC5300 will do this then you could add your PC6400, set memory standard to DDR2 800, CPU multiplier to 11 or auto, HTT multipliers to 5 or auto, and CPU frequency to 200 or auto. You would now have your computer running at normal speed except the PC 5300 would be overclocked to match your PC6400.
You might be able to run tighter timings on your memory, you would have to experiment with it to find out.