lol yeah I would not call golf an athletic sport. Being athletic may very well help with proper swing technique and such, but it's more a game of skill than athleticism.
It's kind of like pool or darts, you don't really need to be all that fit to play those games either. Being fit will of course help, as it will help with anything, but it's not really a requirement as far as how good you'll be at it. They are more games of skill. Like how well can you throw the dart or how well you can plan your shots in pool etc.
To be competitive with the best golfers athleticism is a big factor, particularly on difficult courses that are long and have major hazard issues. Even on easier courses, athleticism is helpful.
What is athletic is something of a semantic issue. Personally, I feel that physical coordination, eye-hand coordination is part of it, as well as strength, flexibility, cardio is fundamental too. If you don't have those things you will suck at golf, period. If you just lack the strength, you can be good, very good. I was told of a golfer who was probably in his 70's and could virtually play par. He'd hit his tee shots less than 200 yards but straight and had a great short game. However, I'd call that guy athletic, definitely. Just not real strong. But he was doing this on my home course, where top level golfers would have bested him by 1/2 dozen strokes.