I've used TMPGEnc a lot, and have never had problems with quality. You most certianly have some sort of conflict in your settings or maybe a driver issue. It's also possible that there is something incompatible about your source video.
i have also used ffmpegX (mac only), although i found it more difficult to use.
If you just want to encode to a small file size, try Dr. DivX. It's not that expensive and very easy to use with excellent results. I've been using it for more than a year now.
But i still think you are fine with TMPGEnc. There's a program called DVD2AVI that I use, which takes a ripped DVD and converts it into an audio and video file that are then converted by TMPGEnc into a DVD-burnable file (audio and video.TS or something) ... which you then burn onto DVD. Excellent results. But time consuming.
If you reply with your specific goals, I might be able to give you more tips.