Pussy...
No, you just get used to it. I was the exact same. My first few months wearing contacts, it probably took me 10 minutes to get them in. My very first two or three times EVER, it took me a good half hour. That's why the optometrist had me do it in the office the first time.
Over time, you just get used to touching your eye. Your natural reflex is to close and look away to protect the thing. And you've been using that reflex properly for a probably a decade or more.
Just remember that you're not touching your eye. You are only touching a piece of plastic resting on your eye's surface. What helped me was I used to stick my finger in my eye to readjust the contact -- actually, it was attention whoring and I'm lucky I didn't get it infected.