Simple answer: driver development and a lack of commitment on ATI's part.
ATI made platform-specific boards for the Mac because of endian issues and supporting those boards became harder and harder. So when the Rage128Pro or Radeon came out, it came out for both Mac and Windows, but you couldn't just buy a Windows card and put it in the Mac and just use the Mac drivers. This was especially frustrating since the Windows card was generally half the price of the Mac card. Also, the performance of the drivers for the card generally lagged behind that of the Windows counterpart. And recall that this was back in the day of ATI writing crappy drivers for Windows, so just imagine how bad they were on the Mac.
At the same time the Radeon was being developed, the GeForce cards were becoming the card of choice for gamers on Windows. This was happening at the same time that Jobs returned to Apple and was trying to reinvent the platform. One of his goals was to push gaming on the Mac. Some of you may recall that Quake 3 was first demoed on a Mac running a Radeon card. NVIDIA saw the Mac platform as an opportunity and began internal development.
In short succession, the Radeon was included on the G4 tower. The Cube was developed and would include the Radeon, but the Cube was still a top-secret project. A day (or was it two) before the unveiling of the Cube, ATI announced that the Radeon would be in the Cube. This announcement stole Job's surprise and as punishment he stripped the Radeon out and released the Cube with only the Rage128Pro. The customer was the victim. At the same conference, NVIDIA announced that starting with the GeForce2MX chip, every new architecture would have Mac support built in.
GeForce2mx was included on the Mac about six months later. This was Apple's "high end" graphics solution. Strange, since it was the "MX" so it still couldn't compete with the Windows gaming market. When the GeForce3 came out, NVIDIA stood by it's promise to support the Mac and the card was actually debuted on the Mac before it was seen for the PC. The game used to demo the card was Doom 3 also seen for the first time.
That's it in a nutshell. ATI sucked. NVIDIA stepped up. Of course, not much has really changed on the Mac. Gaming hasn't surpassed Windows. Drivers still aren't updated. The coolest cards still aren't available retail, etc...