That's not all that serious of an offense, but it could have been handled better. In general if someone complains about getting an email that they feel is inappropriate for some reason, just apologize and let it go instead of trying to convince them they were wrong to complain.
And on a side note, I've noticed it's not a good thing to be the guy who runs the football pools (or organize the company picnic, or other similar things). They could be the ones to get the axe early when layoffs roll around and I can think of three different cases where that happened. The most recent was the football pool guy who everyone liked. Turned out that when some people had to be cut, some executive (who was in the football pool, by the way) decided that guy must have too much time on his hands to run the pool so well, so he was gone.