It's always been like that for me. I just have to refresh the server I want a bunch of times, or open the console and type "connect <IP>" and "retry" until it works.
With persistence it almost always succeeds.
Oh, and also, when I exit the game, if I don't also exit Steam, it ties up my connection such that I can't easily access the web for a minute or so.