can you give a little bit more detail as to what you would use this for?
For instance, in my use, switching headphones is not instantaneous due to the time it takes to remove the cans from my head and place a different pair on, so I've always just manually switched the headphone connection as well rendering a headphone A/B/x switcher a non-need.
Even speaker switchers (good ones) have been shown to change the sound quality in a measurable and perhaps audible way. Depending on the change of impedence w/r to FR of the headphones, their sound quality could change in different ways due to the switch.
Also, most switchers that I know of (affordable) do not level match. So if the sensitivity of your headphones varies, a direct comparison is difficult, and you would have to rush to change the volume knob.