i am a hardcore overclocker and looked into all those programs. To make this thread short:
The best program for testing your system is "OCCT"
http://www.ocbase.com/downocct.html
dont let you turn off by the french webpage...the program itself is in english.
The OTHER (only) program which is halfway comparable/as good for testing as OCCT is prime95, that is, the "torture test" function in prime95.
I however prefer OCCT because it is far easier to use and *far* more sensitive than Prime95. As a rule....with OCCT enter about 70%-80% of your available memory to test (i got qgb, i have 700MB)...and then let it run (torture mode)...for one, two, three or more hours. Or even better overnight.
memtest (IMHO) is only good for testing the memory but does NOT give an idea how stable your system would be in withing windows. (Except you let it run maybe 24hrs or so)....