Yeah, it starts with 2 or 3, here and there.
Take it from someone who has smoked for years, quit for 13 months (not a single puff during), and now smokes again. Don't ever smoke another cigarette again.
Seriously, please, don't. I've done some drugs in my day, but I have NEVER, EVER, done anything even remotely addictive than smoking cigarettes. You cannot imagine not only the difficulty of quitting, but also just getting enough desire and determination in trying to do so.
And just remember, most women don't smoke, and choose not to be with someone who does.
If you won't do it for your lungs, nor your peace of mind, at least do it for your penis.