Did you precisely follow the install instructions for 98 on pg.70-72 of the manual? they are a PITA

but if you failed to install something like the RAM controllers it could explain the behavior. I would have questioned the settings you're using if you are running newer 3D games but since they run OK under XP with the same settings, or so I presume, That leads me to surmise you're missing some drivers. Also, I and many others have found that this board has a distinct preference for w2k and XP so I'd recommend keeping XP on there if you weren't just using a copy already on another system for testing purposes.