The fx5800 was arguably more competitive in performance because it could take the lead in a few benchmarks. The 2900xt couldn't.
No it wasn't competitive for anything that mattered- the FX5800 was an utter disaster for anything which required DX9.
At that time Nvidia had been generally ahead of AMD for a while with the Geforce 3 and Geforce 4 series,and it came as an utter shock when the FX was released. The 9700 PRO came out of the blue - nobody expected ATI to launch such a card. ATI by that point had been written off as not being able to beat Nvidia at the high end.
The 9700 PRO was the card that actually proved that ATI could actually compete in the high end market and it changed people's perception of what they could do.
The non standard DX9 implementation in the FX series meant it could not even run HL2 properly - Valve had to not only made an alternate DX9 path for the FX5800 but had to degrade image quality too. You undersestimate how big a title HL2 was at the time and even Valve basically told people to get an ATI card for the game.
It was a title people had waited six years for and the premiere graphics card company at the time,Nvidia,could not make a card which could run it properly in DX9.
That is from a Valve presentation at the time - they basically could not recommend using the FX series for HL2.
This was the first time many people(including myself) actually bought an ATI card,and friends who bought an FX card were so dissapointed they ended up buying an ATI card the next time.
Plus the backlash for so bad for the FX series,the follow up X800 series apparently was the only time ATI managed to get more marketshare than Nvidia even though the Nvidia 6000 series was technically better. Not even AMD could do that with the HD5870 launching six months before the GTX480 and the HD5850 launching nine months before the GTX460.
Nvidia even mocked the FX5800 with the following video:
https://www.youtube.com/watch?v=H-BUvTomA7M
The 2900XT was comparative less of a dissapointment and it was pretty crap - ATI never had the same superior run Nvidia had prior to the FX for two straight generations as Nvidia had literally dominated ATI at the high end before the FX.The FX was a shock to the system,the 2900XT was more an utter dissapointment but so was the X1800 series too. Even though the X800 series outsold the 6800 series,the fact is the latter were technically superior,and it was only really the X1900 series which had an edge in reality. So in the three series between the FX and the G80,Nvidia had the edge in two of them technically.
No body had written off Nvidia like ATI had been written off just before the 9700 PRO was launched.