Even if Vaping is completely harmless in a second hand inhalation sense, it is still rude to be pumping plumes of vapour in peoples faces. Think of it this way, it would be like using mouthwash and making sure everyone around you could smell your minty freshness by blowing your breath at them. No one wants that from some random stranger.
The fact that Vapers(I'm trying to switch to it myself) thought they could vape anywhere is, IMO, based upon a misunderstanding. They thought that the particular toxins in tobacco smoke was what the non-Smoker objected to and in fact at one time that would have been true. However, what has transpired in the last few decades is that non-Smokers have become acclimatized to not having visible exhalations invade their space, as it were. So even if vaping has no harmful affect to their health(yet to be determined how harmful or not), they still have a sense of being intruded upon. As such, Vaping should be treated the same way Smoking is, by Vapers themselves.
All that said, it should be noted that anyone who breathes in deeply and exhales in public is basically affecting the air in the exact same way a Smoker/Vaper is, in the Physical sense. Just something to consider, especially if one contracts a Contagious ailment and insists on going to Work/School/Public locations.