I live in SF, near the Panhandle. I work in Downtown Oakland. I don't own a car right now. I live with three other people - two own cars, two us don't.
For me, it's not really a big deal. Parking is kind of a hassle near my flat. I take the bus and BART to work every day and while that's not the most enjoyable thing in the world, it's not that bad. Many, many people do the same. I find that the MUNI trams, buses, BART, the occasional cab, and just walking are fine for getting to the places I want to go. I don't get out of the city a whole lot, but when I do, BART and Caltrain are decent options. You can of course also rent a ZipCar or use City CarShare if you need to.
All that to say, I don't think having a car is necessary or even preferable, depending on your lifestyle and where you will be living in SF.
One note: there's a lot of cool stuff in my immediate neighborhood. If I lived in the Outer Sunset or something I might feel differently.
Edit: you're right that the city isn't really that big. Although I've never tried, Google tells me you can walk from Ocean Beach to the Ferry Building, across the entire city, in about two and half hours.