So let me get this straight - you went from a 306HP V6 to a 210HP 4cyl Turbo A4. You do realize the IS350F Sport is a S4/33p/C350 competitor right?
The S4 is far more powerful. You assume I was looking solely based on performance also... In a corner, I saw no real difference. In AWD function, the Lexus was a reactive system...I easily spun the wheels. In the A4, I couldn't spin the wheels (and in the end, I've only spun wheels on ice, or when I turn off the traction system and am on gravel or ice. And on ice, I WANT to spin tires sometimes and just let the mechanical differentials do the work.)
Did not like the interior of the 350 - passenger arm rest was the cup holders (ow!), transmission stuck into the driver side a fair amount and I'm 6'4". Nav system used a *** backwards mouse/trackball type thing. I didn't dare think of touching it unless I was parked and had 5 minutes to attempt to click what I wanted.
Modding - yeah, Audi wins this one...
Dealership I'd have to deal with: Huge minus for Lexus. I'll be happy to never go in there again.
Transmission & fuel economy: 8 speed straight 4 beats a 6 speed V8.
Oh, and for performance, the 350 is what, 0-60 in 5.7s with AWD. An S4 is around 4.9 and an A4 AWD is 6.7s (using a few 2013 numbers here; the A4 in 2014 got a new intake manifold and gained some HP in general, and a bunch of HP for E85 which is only supported in 2014 A4s with Quattro due to a 5th injector being added so the car can cold start using E85.) If I feel like it I can easily mod my A4 to gain some extra boost here and there, put in an HFC to remove any spool up time and do a stage 1 tune from APR (which doesn't need a new exhaust and intercooler) for not too much cash and gain quite a bit of power.
If my choice was between an IS350 and a S4...I'd make the same choice I did before: Audi.