I did your exact shopping two years ago. I test drove a Mazdaspeed6, speed3 and a stock Legacy GT before buying my car & driving it home from Michigan to NC.
The speed6 is a nice ride. Interior quality is comparable to the Legacy IMO, but the seats are far better. I found the lower trim level with the cloth seats and such to be lacking; I wanted all the amenities, but didn't get to test drive a loaded speed6.
The handling is excellent and far better than the Legacy out of the box IMO. Stiffer chassis, less roll. The engine is more responsive at lower RPMs and the gears shorter than the Legacy's 5-speed, making for a more responsive package. I've not driven a 6-speed Spec.B, but I imagine the gearing helps a bit. FWIW, the 5-speed Legacy actually hits 0-60 faster since it doesn't have to go to 3rd gear. The speed6 will also get better gas mileage from what I understand. It also has more interior room IIRC.
I would find a Mazdaspeed forum and look into the long term reliability of the cars. The engine and tranny should be good, as they're nearly identical to the MS3 AFAIK. I would be concerned about which parts are unique to the MS6, as the car was only made for a couple years. The only problem I definitively heard about was leaking rear differentials.
IMO, the Spec.B suspension isn't anything special. You get aluminum control arms and Bilstein struts - big whoop; saves you about $1200 in aftermarket parts. You will still want to put stiffer springs and sway bars on there to get properly controlled ride. I rode in an 06 Spec.B with STi Pink springs and Cobb sways - flat cornering, great grip and excellent ride quality. That is what I'm going to do with my car.
The only thing really special about the Spec.B IMO is the 6-speed that 07+ models got. I've not driven one, so I can't comment on the 6-speed vs 5-speed. The 5-speed is nothing special - the factory clutch is crap, the factory dual-mass flywheel is expensive & can't be resurfaced, shifting is notchy and clutch pedal vague. If I really get into keeping this car for the long haul, I will have the 5-speed rebuilt to tighten everything up or swap in a 6-speed.
The Legacy's seats are not very comfortable if you're very tall or very short. The rear seats don't have enough legroom for tall people in front & in back. Gas mileage is pretty decent; I get 24-27 on the highway, ~20 mixed, and 17-18 mixed if I'm really beating on it (which I usually do).
Common problems on the Legacy's are wheel bearings and clutches. Warranty on 05-06 models for the rear wheel bearings was extended due to 100K miles for this; mine was replaced under warranty. Also the DRL's burn out the headlight bulbs regularly; an easy fix to disable them. Piston ringlands are the weak spot of the Subaru motors unfortunately. Most people, whether stock or with exhaust + tune (i.e. Stage 2), seem to do okay with the motors. But you still see a handful of reports on the forums of ringlands breaking, even at surprisingly low mileage and with no mods. These represent the minority for sure, but are a concern.
The Legacy has a wider aftermarket. There are a number of suspension options and while the turbo mounting location was unique to 05+ Legacy's, now the 08+ WRX uses the same mounting and it's not a big problem to go with an STi-style turbo. If you dropped in the VF52 turbo from the 09+ WRX (~$900), an intercooler (~$700), injectors, full exhaust and a custom tune, you'll be over 300 HP/TQ to the wheels. Sadly you'll also need a clutch with that combo, but it's a reliable and stout power number.
I mainly chose the Legacy over the speed6 because (A) I found one in my price range with several nice mods done that I liked, (B) top safety ratings from the IIHS, (C) I was having trouble finding any speed6's I wanted and (D) the higher trim level speed6's were pushing into the low-mid 20K's.
P.S. Get an 07+ Legacy - folding rear seats FTW.
I'm happy to answer any other questions about the Legacy.
As for other cars to look at, the G35 is a thought. I was shopping in your same price range 2 years ago and the only G35's at that price had high miles and were extremely common. Plus used models with the sport pack seemed impossible to find.
You could probably find an 05-06 Mitsubishi Evo for that price, but insurance will be sky high compared to the Legacy and you will have to be very wary of how the car was driven. The cars are pretty bulletproof, but given how many teenage dbags beat those cars to hell I would be concerned. A good friend of mine had an Evo VIII MR and now has an IX; great cars if they're taken care of.
As for 4 doors, manual and RWD/AWD...I think that's about it my friend. The Mazdaspeed3 isn't a bad ride and unless you are trying to clip every apex, the FWD part isn't too damning.