Yeah, it all depends on your main usage. 5D sucks for sports, is good for people shots indoors (things like cheap 50mm lenses become much more useful) and landscapes. The 5D has been my main camera for 3-4 years now. The AF is pretty bad; if you're comfortable with using the center focus point 100% of the time, it's okay. Really the main benefit of the move to FF is how it transforms your lenses. As for the body itself, it will be a step backwards in many respects.
Compared to a D80? I wouldn't say so.
The 5D's AF is not the best, and to be honest it is the main reason I plan to upgrade to a Nikon D600/D700 later this year.
However, the D80's AF isn't anything special, either. 11 points, 1 cross point in the center. The 5D I/II have 9 point AF systems with 1 cross point in the center, and also 6 additional "invisible" assist points that come active during AI servo. AF is certainly comparable between the two. In terms of speed, both will do 3 FPS continuous shooting.
The 5D will offer more than just better image quality compared to a D80. One other big difference is how much larger the viewfinder on the 5D is compared to an APS-C camera like the D80. I have a hard time using APS-C cameras after being spoiled by the huge viewfinder of my 5D.