Honestly, it's not the cold that's really the problem for me -- it's the snow, and the result it has on traffic.  Any measurable amount of snow and my commute goes from ~20 minutes one way up to usually 1.5 hours one way, sometimes 2 hours.  
With the cold, there gets to be a point where it just doesn't feel colder anymore.  My wife moved up here from FL in mid-January a few years ago, and even she hasn't been complaining about the cold.
We've talked about moving someplace warmer, but we always end up at the same conclusion -- we get paid well enough, have relatively secure jobs with good benefits, and the schools here are very very good when compared with some of the warmer out-of-state areas we've looked at.  That being said, we WILL be moving within the next couple years, but just closer to work, not out of state.