People click with me all the time. It's just that I'd rather not socialize that much. I have a friend that used to call once a week just to chat. What kind of guy does that? He even has a wife.
Eh, some people just enjoy talking for the sake of talking, while others don't. I personally am not one of those talkative people, but I don't mind individuals who are.
That being said, for many socially anxious people, I'd imagine that one of the ways they come to terms with the condition--and/or one of the consequences of always being hyperattentive to how others perceive you--is to attach negative feelings to all aspects of social situations, including the people involved.
It's obviously not enjoyable to be sitting in a room, anxious that everyone's looking at you, and having to force yourself to talk because your nerves have clamped down on your ability to spontaneously think of something to say. Thus, the act of socializing becomes associated with all of these negative emotions and physiological reactions. Eventually, the thought of socializing itself starts to elicit them; one of the ways we may attempt to "explain" this avoidance of socialization to ourselves, then, is to--again--attach negative meanings and characteristics to everything social: the people, the conversation topics, the locations, etc.