HamburgerBoy
Lifer
My response would have been to smile and perhaps nod my head at the other person. I'm kind of like the OP in social situations. I never "naturally" know what to do like so many other people seem to. It's taken me a long to time to reason out the proper responses that I do know now. I've determined that smiling and nodding fits in most any situation where there's doubt as to the proper response.
The only really proper times to touch other people in response to their touch are real hugs (not your situation), handshakes, and kisses. There are situations where it may be acceptable for you to initiate contact. Congratulatory pats (like the other guy was attempting to do), high fives, etc all have their places. You have to be able to identify when those times are though. I can't ever seem to identify them except in retrospect though, so I almost never initiate contact.
Yeah, the smile-and-nod is my usual technique as well. I guess part of the reason I reacted that way is because I know my advisor knows that I'm a terrible public/group speaker and really awkward and quiet overall, so in the back of my mind I had this idea that he told the other members that I had issues and that they were nicer/more lenient because of it, which would make a quick pity side-hug make more sense to me.