The problem may not be on your end. DSL is quite sensitive to moisture. i.e. when it's downpouring, the speed may be slower than when it's sunny.
However, if the symptom happens on a regular basis, the problem may be on your end.
Have you tried another client like vuze?