I don't think more RAM will help you unless maybe your system is unusable while encoding.
If your goal is to increase your video encoding performance, I don't think you'll be able to squeeze much more performance out of your current build without upgrading. I was going to suggest overclocking higher, but I don't know if your power supply or motherboard can handle much more, and you may need a better CPU cooler since I don't know what you're using now.
You could probably get away with upgrading to a socket 939 AMD X2 processor (X2 3800+ is $152 at newegg at the moment) without upgrading anything else and see higher performance.
If you really want to go all out, get a Conroe CPU (E6300 is cheapest), a new mobo ($150-200 range probably), new power supply (should be doable for under $100), a big HSF if needed ($35-60 there), and DDR2 RAM ($100-175) and overclock the hell out if it 😀