<< the first thing to remember is that with a residential line you do not have a service level agreement, and your 640/320 is the maximum speed you will possibly reach, but even if you never got over 400/100, they would still, legally, be providing you exactly what you paid for. >>
No, DSL providers promise 384K minimun. It's unacceptable for not getting 384K at least. If it's because you're too far from the central hub so that you can't get 384K, don't use DSL.
There must be something wrong with your tel line or your DSL provider.
My DSL provider - Pacbell - often down once or twice a month, often suddenly become very slow, just call the customer support, tell her it's down, listen to her BS instructions, pretend you're trying, tell her that you try it and still doesn't work, get a ticket number, after few hours to next day, it'll back to be working, and you'll feel faster than before. Then after some days, repeat the above.
Also, when your DSL is abnormally slow, try to pick up the phone and listen if there is noise. I had a 2 months experience of having noise in tel line during night prime time that made the DSL almost not working - extremely slow, like a 300bps modem.
Moreover, if you don't live close to the central hub so that you can get 1.5M, don't use DSL too. Afer using DSL at 1.5M for a year, I can't accept anything lower than 1.5M.