The body adapts over time, physiological and psychologically. To give an anecdote: I was one of those people who said "Caffeine doesn't bother me, I sleep fine". Well, from that frame of reference, it is was true (to me) because I didn't know anything better. Once I cut caffeine out for a significant period of time, I realized just how horrible my life was on the stuff. I feel it is often that way with people in many things.
We also make the same mistakes, over and over again, often with slight differences (unless we learn nothing). I, for a while, thought that after getting clean off caffeine for a year, I'd be able to moderate my use and gain all the benefits from it, with none of the drawbacks. It worked, for a while, but slowly and surely my QoL started to deteriorate until I cut it out again.
It also takes much longer to withdraw than people think. The major symptoms are only a week ,but there is often a persistent exhaustion that occurs for much longer.
I also feel that caffeine feeds into other addictions. I noticed that it increased my desire to junk food, which in turned increased my desire for getting high on whatever it is I wanted. Is this all caffeine's fault? Of course not. Is it a gateway drug? No, of course not. Does it cause changes in the brain? Yes, it does. It does mess with brain chemistry. Life is all about balance, if you can make it work for you, great, but often times when we think works is from a faulty frame of reference anyway... Even in my current stable condition, I could be totally unstable, dependent and not know or perceive it. I guess when things get bad enough, people will start some self exploration and figure out for themselves that what we put into our body can influence our moods, feelings, etc... Then we have an "ah ha" moment, and change some things up, sometimes for the better, sometimes maybe for the worse. Live, learn and apply. I guess that is all we can do.