There are so many good hot drinks rich in beneficial phytochemicals where caffeine comes as part of the package (green/black/white tea and coffee). I usually end up having some caffeine about 2/day as part of green/black tea and coffee.
I find it beneficial for getting me jogging. I pretty much always have coffee and/or tea before I go jogging.
I do find I don't sleep as well as when I went caffeine free, but I limit my intake to 3 small serving/day max, and usually all before 6PM so the impact is small.
Overall the benefits outweigh the downsides for me. YMMV.