If your download speed is below the bitrate of the file being played, your media player will have to buffer the file while playing. That will cause the noticeable pause. For example, that would happen while trying to view a streaming Real Video file on a 56kbps connection.
See if playing back local files causes the same problem.