Hehe....I remember driving a brand new BMW roadster with the top down on a mojor highway in January. It was back when the first started building them and noone had one on the road yet.
So my friend and I, in all of our infinite wisdom, decided we'd look much cooler with the top down. I'm still not sure if people were staring at us because of the car, or our stupidity!
To answer your question, it wasn't that bad with the top up (of course I'm in SC, winters rarely get severe here). I've been in older drop tops where age on the top was an issue with the weather.