Well, from what I can tell, there's two ways to blow speakers: 1) Blow them and 2) Clip the amp.
1) involves having way too much power for the speakers. The speakers don't have enough range of motion to take all of the power they're being given. They bottom out the magnets and the surround. This is the way to blow a woofer. I think it's pretty hard to do and most decent speakers have overload circuitry designed to protect against this. And you'd have to have like 3X the rated power to do it.
2) involves having the input signal to your amp bumping up against the maximum headroom of your amp. So what happens is like this. You have an imput signal (say, from your CD player) that's curvy:
................./^\............../
\............../.....\............/
.\............/.......\........../
------------------------------ etc. This is a nice, smooth curve with rounded peaks, although I can't draw them.
...\......../...........\....../
....\....../.............\..../
......\_/................\_/
Then, your amp basically wants to take that same graph and draw it a million times bigger. BUT, it runs out of headroom. Say the input signal peaks at, on some imaginary scale (I don't know all of the exact units involved), a 2. Then you want to amplify it by 1,000 (not unrealistic, I think) and that'll power your speakers. But your amp only has enough power to output 1,500 -- not the 2,000 required to draw the whole thing. So your amp outputs the following:
\............../^^\............./
.\............/.......\........../
------------------------------ etc. This is the same graph as before, but with the ends chopped off.
...\......../...........\....../
....\___/.............\_/
At the extreme levels of clipping, you get a square graph:
.........|^^^^^^^|..........|
|........|............|..........|
-----------------------------
|........|............|..........|
|___|.............|____|
It's this that can really wreak havoc on the speakers. Normally, speakers have a nice, smooth motion: they go one direction, then slow down, come to a nice, smooth stop, and accelerate in the other direction. (A few thousand times a second.) With clipping, the smoothness is taken out: They are in position X, they suddenly (almost instantaneously) accelerate and move to position -X, where they stop (almost) instantaneously, and they stay there for a split-second, when they suddenly move back to X, again coming to a screeching halt from a very fast movement. The forces put on the drivers are way beyond what they ordinarily take. This has destroyed several of my tweeters (when I lent my setup to some friends

I had warned them but they still turned it up too loud).
The problem of clipping essentially comes down to not having a powerful enough amplifier. If the levels between the inputs of the amp and the outputs aren't really matched well (say you have a CD player that puts out way over-spec signals) then you can get this problem even if your amp is very powerful. The headphone jacks on portable CD players clip a lot when you turn them up to full volume. They simply can't play the signal loudly enough.
Anyway, this is like the sum of my knowledge on this subject. I may be wrong, but I have read from good sources that it's *MUCH* easier to damage speakers with an under-powered amp than with an over-powered amp.
[edit]: trying to pretty up my graphs ERG it's hard b/c they don't look the same when I hit "preview" as when they're actually posted... okay good enough now.