generally, comfortable headphones are not suitable for gym because they trap too much heat on your head. Beyer Dynamic DT-990 offer good sound quality and are very comfortable (in my opinion) but are definately not suitable for gym - especially if you try carrying around a headphone amp to enable the "drowning out" of the crap gym music.
IEMs are probably the best to isolate your ear drums from the gym music. They typically can lower the surrounding noise by an average of 27db. Unfortunately, I have found IEM to not be very comfortable for running etc because everytime my feet impact the floor, i get microphonics through the IEM.
Right now, I am using a 40$ pair of Koss PortaPro. They offer a nice bass "hump" in the ca. 80hz range that puts more "kick" into your favorite gym songs -- a place where you dont really care about overall musical accuracy. The Koss PortaPro is pretty lightweight and are decently comfortable.