What version of Memtest86 are you using? I experienced this very same rebooting thing when I was using Memtest86, latest version 3.5, with 2 x 2GB of G. Skill, on an Asus P5Q-Pro. Searching the Internet, I found a Memtest86 Wiki page that describes an incompatibility with 4GB of Ram.
I then tried Memtest86+ version 2.11, and it worked fine, recognized the P45 chipset, timings, everything. I ran it for 24 hours at full spec speed without errors.
The Memtest86 problem could just be 4GB on a P45 chipset. I tried the same program on my P5W-DH (975 chipset), which has 4GB of G. Skill as well, and it ran without incident.
I think the Memtest86+ is more up to date than the original and hence more compatible with modern hardware.
Good Luck