A related question:
What's the difference between a neutral wire and a ground wire? The neutral wire is tied to ground at the breaker box anyway.
Is the ground wire basically just a sort of failsafe, or secondary, path for electricity in the event of a problem?
You are right that the Neutral should be connected to true Ground at the breaker box in virtually all North American systems. The difference is in the current being carried, and thus in the small non-Zero voltages at various locations.
If you are at one outlet in a room in which several outlets are fed from the same breaker (this is the normal scenario), the Neutral line is already carrying back to the breaker panel the total current being supplied and used by other devices plugged in in the room (circuit). Now the Neutral wire, being a real-world piece of copper, has a small resistance per foot. Suppose for illustration's sake that the total resistance of the Neutral wire from the room back to the breaker panel is 0.01 ohms (10 milliohms), and the total load on the circuit is 10 amps (typical breaker rating would be 15 amps). Then the Neutral wire at the room end is at a voltage of 0.1 Volts (100 mV) compared to Ground. That is NOT zero, and it is enough to be a noise source to some electronic circuits. It certainly is NOT a dangerous voltage to people.
Now suppose, instead, that there is a poor connection in the line at one of the junctions, making the line's resistance 1 ohm. That 10 amp load now means the Neutral line is at 10 Volts above Ground. That still is not dangerous to people, but it's getting into a worrisome range. Something in the 25 to 40 volt range can be lethal! As an aside, if there's a 1-ohm resistance in a junction inside a box in the wall with a 10 amp load running through it, the power being dissipated by that resistance is 100 watts. That much heat being generated inside a small junction box would quickly build up to a real hot spot, and maybe a fire start. So let's face it, that is not a common situation.
But by extension, you can see that a Neutral line has voltages on it that can be a problem for certain types of circuits, although not usually a shock hazard for people. BUT if any really odd situation creates an unusually high resistance in that line, its voltage actually can get high enough to give you a perceptible shock, or even possibly a lethal one. On the other hand, any circuit should be designed so that the Ground (bare) or Bonding lead in the wiring NEVER carries any current under normal conditions. So it is guaranteed to be at zero volts (compared to true Earth Ground) anywhere in the circuit layout. Thus it is ideal for getting rid of ("grounding out") small noise signals. It also is an absolutely unimpeded route for carrying big currents in the event of some major wiring failure safely to Ground, which accomplishes two things. First: it allows enough current to flow that it causes the breaker to open, stopping current flow. Second: during the brief time it takes for that breaker to react and open, it ensures that the exposed exterior of a device (to which the Ground lead is attached) is at so low a voltage that it is NOT dangerous to anyone who touches it.