You're not just burning the "iso" file, are you? You need to use a burning app like Nero that supports burning images and go to "file>burn image" and browse to and select the "memtest86 3.0.iso" file. Nero will burn the image of that file and it'll work fine, assuming that your system can boot from the CDrom drive.
If you already burned the image correctly, ignore what I said above and consider this a bump.
