A regular headset will utilize your computer's sound generation capability (i.e. your soundcard or your onboard audio, which you shouldn't be gaming on anyway).
A USB headset is essentially its own soundcard. It's useful for laptops, or if your onboard sound REALLY blows chunks. It can also produce clearer sound, since the analog circuitry is not stuck inside the radio-frequency noise-infested computer case, but rather next to your ear.
Setting up a USB headset might be a little more challenging - since Windows will essentially have two soundcards on file all of the sudden (your regular sound card plus your headset.) You'll need to set up Windows to use the USB headset as the preferred device for input and output.