I would not say I don't like it, it's more that I rather spend money on things I can experience for longer. But it does kinda stress me out a bit too, depends on the kind of trip though. I've been on trips where we had zero schedule and zero places we HAD to be, we just explored. That was actually fun. But trips with a big itinerary are just stressful, constantly having to check the time and figure out where you're going etc. Almost all the trips I've been on I was not the one worrying about any of that mind you, I just followed the others around but even then the whole "we need to be here for this time" thing makes it less enjoyable I find. So many times where I actually wanted to see or do something, but couldn't because we had to be somewhere else soon. I also have paruresis and it's actually gotten worse as I get older, so that does make travel less fun as well.
Basically if I have several grand to spend on a trip, I'd rather spend it on my own property or on something nice I can enjoy for years. Though if money was no object and I had unlimited time off I would probably do a 1 month trip at a cuban resort or something though just to get a break from winter. I tend to get fed up of winter around April so I'd probably just take a trip mid April so I can arrive back mid May when things are hopefully starting to warm up. Something like a resort where you just lounge around would be my kind of trip.
I totally understand the appeal of people who do like to travel though, it's just not that big of a thing for me I guess.
Take trips that don't have firm itineraries. Just get a travel book, and do things at your pace. Those are also my favorite types of vacations, just a list of shit you want to do, without someone telling you where to be, and check them off as you will, judging the time you have left with what you want to do. I feel you on making yourself a nice bat-cave. I have a nice little home base here in my little 2BR condo, pretty nice TV setup with sound system, nice audiophile headphone setup, good gaming setup, all the interior amenities I could want. And I can keep wanting to upgrade all this stuff and put more and more money into it, but it's not worth it as much as traveling. Honestly, I don't need a damn thing more except to see more stuff outside my little area, and a traveling partner to do it with. It's a tough realization to come to.
I understand you may not like cities much, so go to all the natural beauty destinations you can. Alaska, Colorado, Utah, Arizona, New Mexico, the Pacific Northwest, California - this is stuff you don't even need to leave the continent for. Trust me, you will be happier if you see these things in real life more than just on a 4K television. A couple good trips and you just might get addicted.
Finding the right travel buddy is the hardest part though. I'm not a solitary traveler as much as I like to share the experience.
The only reason I keep my sanity is being near NYC, which is like having an insane amount of stuff to discover and see on a daily basis without having to go far at all.