It's called planing, and it's not simply for image. When a city grows the demands on infrastructure grow as well. Public transit, parking, roads, electrical grid, sewer system and water supply all play into those decisions. Parts of SF's sewer system are nearing a hundred years old. The trucks that deliver everything a city needs to survive are very hard on streets, many of which were never built with eighty thousand pound trucks in mind. It's not as simple as "build more sky scrapers".
All that said, most city's do have rules to control how much of anything gets built, if SF doesn't want to become a sprawling metropolis they get to make that decision. It's not "running it for everyone" it's protecting the interests of those that already live there. As well as limiting growth, they also do things like preventing slaughter houses from being built within city limits and controlling garbage dumps.
We all have the right to live where we choose, as long as we can afford it.
All of the things you listed can be upgraded.
Second, who do you think is being benefited by the policies that limit housing supply? Right now SF has the most expensive $/sqft prices in the country. Only the very rich can own homes, and its causing huge inequality. They city is not protecting the interests of its people, its protecting the interests of a very few elite at the top at the expense of everyone else under them.
Also, you started out by saying that you knew of no such policies that limited supply that was driving up prices. You sure seem to know about them, so I'm going to assume that you were flat out lying.