I'm not really too sure why your ATI card has lesser quality than the Nvidia card.
I can only assume there is a range of quality and perhaps your card is at the lower end of that spectrum?
The Radeon that I have is made by Sapphire and is pretty much one of the cheapest options - it is a 64MB card that actually came with 128MB on board. The card also played Soldier of Fortune 2 far better than my GF4 Ti 4200 (much smoother) which is really against popular opinion (there was one other guy on these forums that thought his 8500 performed better than his GF4 too) much as your DVD playback quality seems the opposite.
It is hard to say from the still shot image how good or bad the DVD playback is compared to the Nvidia card - if you could do a side by side we might see more differences.
All I can say is that with my Radeon the image quality was sharper compared to the Nvidia cards (GF2 & 4) - I'll see if I can take a screen shot of a DVD movie on my GF 4 but I doubt it would really show anything as the image quality is already very good. When compressing the file to .jpeg there will presumably be some loss of image quality also?
On defense of the Nvidia cards though I did not see any great difference (if any at all) in 2D compared to the ATI card.
I certainly would never change to one brand from the other simply based on the DVD or 2D performance differences that I have observed for sure
Is your Radeon a genuine BBA?