Being small I was picked on most of my life. I've been in a LOT of fights in my younger days. When I was in the Marines I was ALWAYS getting in fights with other jarheads. WHY? Because they keep you pissed off and wanting to fight. (The grunts anyway) I was Air Wing, so that's different. However I was all the time running into pissed off Grunts, they wanted to blow off steam, they wanted to fight, but not get hurt, so they would single me out since I was 5'6" and around 145 Lbs. what they DIDN'T know however was that I had been fighting most of my life. (Never started one though) and most of the fights I had been in were 2-3 guys on one (ME). I always tried reasoning with these Neanderthals that felt fighting was a way to solve things, but talking seldom works in the real word.

At any rate, I lost very few fights. I would usually get inside their reach, climb up them and get them in a head lock then start working on the face. (Worked for me, I have a LOT of upper body strength) I guess the point I'm trying too make is, just because someone is smaller than you, don't expect it to be an easy fight!
The only time I ever threw the first punch in a fight was the ONE time I thought the guy that wanted to fight MIGHT have a reason. To make a long story short, one night in High School I was being chased by someone in my car. I ran a stoplight and got pulled over; the cops knew me and asked what I was doing. I told them someone was chasing me. About that time another car came around the corner and I told the cop that that must be the car that was chasing me. He pulled the guy over, searched his car, found beer, and took him to the cop shop. The next day at school the guy said his dad took his car away, and that he WASN'T the guy that was chasing me. HE WAS PISSED. I still thought it WAS him, but didn't know for sure, so I told him that if fighting would make him feel better, I would meet him after school. We did, we fought. And it was about a tie. (Though I had the upper hand when the fight got broken up. I had him in my patented "Head Lock" and he wasn't going anywhere)
Any way since I figured he MIGHT have a beef, and I HAD to fight, I might as well get the upper hand and get in the first punch. Always a good idea if you HAVE to fight!
BTW: I saw this guy after 30 years at my High School reunion last year. And as I suspected all these years, He WAS the guy chasing me that night!
It is rare that one MUST fight. In your situation I would say just let it be. Barring that, tell him if he wants to talk smack, to meet you at the GYM. Put on some gloves and go at it that way!