It depends on the license you want to get. A basic VFR, single engine, private license (i.e. no charging your friends for flights) takes a minimum of 40 hours not including ground time, exams, etc., and the place I went to charged about $60 an hour. So, that's at least $2400 right there, not including books and other equipment. Other licenses go up from there. Training includes basic flight, stalls, spin awareness (I never actually did spins), ground reference (flying around a point), radio navigation, and a lot more.
You only need to practice recovery from a spin to get a CFI (certified flight instructor) license (and maybe an airline transport license). Otherwise they just tell you how to recover without actually doing it (although some sadistic instructors will make you do it anyway). I got up to my third or fourth solo flight (half of those 40 hours are time flying by yourself) before I had to move away to college, so I don't know much about taking the exams or doing cross-country.