I know nothing of FRAPS, but I can offer a bit of info on the Quicktime thing. Might not be what you want either. Ulead Videostudio can output to .MOV format. But it's $100. I think that Quicktime is a somewhat proprietary format, or at least Apple has some legal wing over top of it. You probably need a license to output to it, and free software can't do that, otherwise it wouldn't be free.

Note though, Xvid, Divx, and Quicktime are all variants of the same thing - MPEG-4.
Virtualdub - what compressor are you using for video? It's going to be a huge file if you leave it at default (raw frames, uncompressed). Install
Xvid and use that to compress the video file. You can get really good quality and small files. Granted, using heavy compression can make the process incredibly slow - I converted a movie to Xvid recently using the best possible quality (Quarterpel, motion quantization, chroma motion, all the quality settings I could find), and it processed 3 frames per second on a 2GHz Thoroughbred XP-M. The output though looked the same as the original.