Honestly, it's not the cold that's really the problem for me -- it's the snow, and the result it has on traffic. Any measurable amount of snow and my commute goes from ~20 minutes one way up to usually 1.5 hours one way, sometimes 2 hours.
With the cold, there gets to be a point where it just doesn't feel colder anymore. My wife moved up here from FL in mid-January a few years ago, and even she hasn't been complaining about the cold.
We've talked about moving someplace warmer, but we always end up at the same conclusion -- we get paid well enough, have relatively secure jobs with good benefits, and the schools here are very very good when compared with some of the warmer out-of-state areas we've looked at. That being said, we WILL be moving within the next couple years, but just closer to work, not out of state.