Listening to music is very passive, you just listen, sometimes not even that, when talking you're actively participating.
I can actually notice this with phone calls too, once in a while I've talked on the phone while in the car, and if it's just some babbling on about stuff I don't care about, I drive just as well as I do without the phone since I don't actually listen, I just say "Uhu" once in a while.
That said, I make it a point not to talk on the phone while in the car these days, since I believe that to be reckless and respectless to your fellow drivers.