You might have to check around for small electronics shops, though they might be tough to find.
There's a little hole-in-the-wall place near me - from the road, there's no sign of it. It's around the back of some kind of run-down industrial building. But they stock a big rack of Jameco parts, and they have a better component selection than Radio Shack. I don't even recall seeing any signs indicating the name of the place.
Other than that, you're probably going to be buying online. Mouser, Digikey, Newark - they've probably got what you need.
What sort of thing are you switching with it?