I've used Windows XP64 as my primary OS for about a year or so, before Vista came out.
With the usual Intel, AMD, nVidia, Realtek, Marvell hardware there's no problem getting drivers.
The main problem with gaming is that especially older games often use CD/DVD protection which works with special low-level drivers. These drivers won't always work in 64-bit OSes.
Another thing is that quite a few installers/applications have buggy OS detection. They will just refuse to run, because they think your OS is unsupported. Sometimes that can be fixed with the compatibility settings. Sometimes it helps to just extract the files manually, or install them in a different OS, and just run the files directly from their installed folders in XP64.
In general it's about as good as 32-bit Windows for gaming, as long as you can install and run your games. Most big titles will just run (I've played various games, like Doom3, Far Cry, Crysis, various Need For Speeds, Half-Life2 etc).
I don't expect any current or future games to have problems with 64-bit anymore, since it's become a lot more common with Vista x64 out.
Having said that, Windows XP64 is a bit of an 'orphan'. It's a cousin of XP, but not quite. And although it's related to Vista x64, it's not quite Vista either. Even Microsoft no longer supports XP64 with a number of their applications. You cannot install Live Messenger, Mail etc on XP64 for example. And ofcourse you don't have DX10 either. So I think as Vista and Windows 7 will get more popular, you'll quickly find yourself 'abandoned' with XP64.