I also waited as long as possible to get my license. Too many people pass their driver's test JUST to get the license. They don't actually want to learn the rules of the road, all they want is to own the road, which they never will anyway. People get arrogant and impatient too when they're driving. I was with my roommates recently, and they're occasionally yelling at other drivers for certain behavior. Then I notice that my roommates are driving in exactly the same manner, and yelling at other people for being so stupid in response. Driving does make people stupid. They're piloting a large piece of metal at considerable speed, and treat it like a toy.
Sorry if that wasn't good for confidence.
I got my license at 18, when my parents said that I had to learn how to drive. It scared me then, and it still worries me today, but I do do it. I've had a few close calls, such as when people change lanes without looking, but no accidents. A tactic I use when driving too - I assume that other drivers are likely to be arrogant, impatient d!cks. And drive with patience. If you try to speed ahead when the sign says "MERGE HERE, MORONS!!!" you might risk getting either just stuck in a lane that ends, or perhaps smashed between two tractor trailers, all because you tried to save a few seconds by gambling with your large metal vehicle.
Yes there is a lot to stay alert for, but it is possible to adapt to.
Now, all that said, I might have a different opinion of cars if I had to drive in a city on a regular basis. I'm from a rural area, and even now at the college campus I'm at, I don't encounter any real city driving. The times I have had to drive in small cities, I usually prefer to stop afterwards for a little break to just calm myself down. People get goddamn insane in dense cities.