SF has its neighborhoods. If you were there for a weekend or small trip, it's likely you were only "DT" SF, which is a complete shithole. It's really shittier than any shithole I know in this country....but it isn't the whole city. There are of course some uber fancy neighborhoods that you likely wouldn't recognize as the same city, but you kinda have to be there for reasons, like, living there.
I'm no SF fan, and I'm completely guilty of judging it by 1 or 2 square miles of market street, the wharf, and Powell areas, but that is such a tiny part of the city. ...OK, the Haight is still pretty gross, but that's like the gross parts of any other interesting city. Totally acceptable because it's kinda worth it for all the stuff that is there. Just keep your head down and to the side to dodge the live piss streams when they appear.