CQ=constant quality. I agree that the manual shows an RF20, but considering if your making a serious encode its likely your going to be scaling it up to 1080P making artifacts much more apparent, especially banding. At resolution, RF20 is fine. In my experience, at RF of 15 or 16 gives a much cleaner picture at "somewhat reasonable" file sizes. I say somewhat, but it is still half of source usually, unless there is a lot of pixel noise. For the most part if the dvd has a clean mpg2 encode, then that will translate to a lower filesize using CQ for the same quality.
Please, don't upscale your encodes. There is no reason for this on PC systems. The only time you should upscale an encode is if your target device has a particularly crappy upscaler. Upscaling increases the needed bitrate for quality for no gain.
talimer did you ever get the problem with the .0024 sized files fixed? That sounds to me like something in your encoding chain has become broken.
Also, those settings are pretty extreme (Do you want this encode to finish in your lifetime

). And the deblocking settings are WAY too high (this does NOT improve quality). I recommend putting your deblocking settings down to 0 and possibly even -1 for live action footage. It should only really be a 1 or a 2 for animated footage. 6's are just a little extreme.
As for the ref and bframe settings. Those are really pretty high as well. after 6 reference frames the quality gain is essentially 0, especially for live action footage. There is a larger benefit in animated footage, but that is mostly from bframes (the difference between 16 and 6 reference frames, even in animated footage, is somewhere around 0.1%) Decreasing the reference frames will significantly increase your encoding speeds.
Your motion estimation is also WAY too high. This can actually decrease the quality of encodes. A motion estimation of about 24 is the most anyone should ever do. In fact, that may be the issue. The size of your motion estimation may be causing x264 to consume WAY too much memory.