He was smiling, but I've always thought humor was a way to mask the truth.Apologies, I could only go by the context of the OP. Does your wife talk about it in front of him a lot?
How was he acting when he said it? Joking, genuinely annoyed?
Most guys wouldn't make a comment like that in a 100% negative manner, unless he is the kind of person who usually mutters these sorts of passively aggressive comments or generally doesn't care for you, I don't think it's anything to get too worked up about. Let it go, joke about yourself, or bust him back depending on the kind of guy he is.
I'd bust back with a "how's that foreclosure working out for you, mr. money bags? The IRS still garnishing your wages? Props to you, wish I could measure up to the standards you've set!" but I've been known to go for the jugular a bit too often