Actually that is a good summary of your point.
First, you defined the alternative.
The negative attention they are getting now has less potential impact than not switching the announcer out.
This implies that the option they took has less of the impact vs the option they could have taken. Your logic is that had they chosen the other option, the negative attention would have been greater.
Next you have this...
Right now the only impact is a bunch of fringe jerks offs complaining on the internet about ESPN.
This is vital to understanding your claim. This part means that right now only the fringe are the ones making an issue out of this.
Fringe definition - adj, not part of the mainstream; unconventional, peripheral, or extreme.
So, all things coming from this is from the non mainstream. So the logic follows that had they taken the option of leaving him, the negative attention would have been greater. That coupled with the next part about how right now its all fringe means that the other option would brought negative attention from non fringe. Because you implied that the 2nd option is the cause for only the fringe to react, you inherently implied that the first option would have caused the non fringe to also bring negative attention.
So, if you had intended to say something else, you failed because your claim is explained above. Its quite possible you misspoke but he is not wrong in his understanding of what you actually said.