I would use USB over built in sound, but not if you have a decent sound card. If you have a good sound card, USB will bypass it and use itself, which will more than likely give you inferior sound. With the USB headsets, they typically add virtual surround sound like dolby headphone. So a step up from built in sound, and below discrete sound cards. Also seems to be less noise than the standard analog jacks, but thats not universal. USB typically adds $15-$20 to the price, so it depends on how much those features are worth to you.
Downside is if you decide to get a sound card down the line, you will need another head set. You can buy a regular analog headset and buy a USB to 3.5mm adapter that is also a sound card, for under $10. Or buy a headset that uses an adapter, so then you would have the best of both worlds, but that option seems less and less these days, probably to reduce costs.