is 480p the max DVDS have, or would a normal progressive scan dvd player work better?
A proscan DVD player will play at the same resolution that the XBOX will. Some people use PCs or video scalers to play DVDs at higher resolutions, but it's just scaling... DVDs don't actually have additional video information to use.
You can buy DVD players with better video quality, but I think the XBOX looks pretty good. I have both a proscan DVD player and my XBOX and have compared them directly.
Since you already have your XBOX modded it won't cost anything to get DVDX2 and edit it to play DVDs in proscan. So just give it a shot and see if you like the result. If not, you can then start looking at standard proscan DVD players.
My Xbox plays at 480P but the Picture Quality is horrible
Are you sure you have everything calibrated correctly? Mine seems to play back with pretty good quality.