I guess it depends on what you're going to do with it once you buy. If you might tow a boat or a trailer from time to time, you probably ought to look at something other than the Lexus. The Lexus' towing capacity is only 2000lbs. The 2004 XC90 T6 was 5000lbs. I haven't seen the specs for the V8 2005 model but I assume it's at least as good.
For the record, I'm in the same boat. I'm looking to replace my '97 Honda Passport but I'm not seeing any really attractive SUVs out there. Seems each one has at least one glaring problem whether it's limited cargo capacity or towing capacity or fuel economy. I don't go rock hopping but truck does see dirt so I need at least modest off-road capability, too. That last critera probably knocks the X3 and maybe the XC90 off my list.