dust can attract moisture. Most speakers (or at least the good ones, if I remember correctly) are made from some sort of heavily treated paper composite material, lots of speakers are some sort of plastic, and cheap speakers use paper and the foam stuff in there is fragile.
All that can be damaged by moisture... Dust can also promote mold.
Now all this stuff may not ruin a speaker by themselves, but in a humid enviroment or over a long long time it can probably do some damage and weaken it.
Other then that I can't think of a whole lot.