Aero Student here.
I could probably type out a laundry list of stuff here.
You missed the big one: air density. Subsonic airliners cruise at about 35000-40000 ft. The Concorde cruised at 50-60k ft. AFAIK they never "officially" released numbers for the SR-71 but it's speculated that its top speeds were reached at 110-120k ft or higher.
The atmosphere at sea level is almost 5x more dense than the atmosphere at 40k ft. Trying to run a vehicle at sea level means more drag, more heating, more fuel consumption, etc.
