You don't need to spend $500 if you're just doing gravel trails. More like $350 should be enough. If you're going to be doing real single-track then you're gonna want to spend in the $500 range and get a good bike. Also remember to buy a little below your budget so you have money for accessories such as water bottles and holders, locks if needed, speedometer if you want one, mud guards if you want them, better off-road tires, etc.
Also make sure you get a frame that fits you well. I am 6'0" with fairly long legs, and I have a 19.5" frame on my Trek 4500. It's a good size for road riding (which I do quite a bit of), but I would probably be better off with a 17.5" frame for off-road riding.
Oh and whatever you do, DO NOT GET FULL SUSPENSION. There are no good full suspension bikes for $500 unless you are buying a several year old used bike, which I wouldn't do unless it is in great shape.