When using remote desktop internet connection drops!

[Intelman07]

I have an odd problem. It seems everytime I remote desktop into a computer on my network while connected for a minute internet connection drops on every computer on the network. The connection will just "pause", my instant messengers go offline and are forced to reconnect and downloads time out. This started happening when I was forced to replace my router and only seems to happen when I am using remote desktop. I have a SMC2804WBR. Has anyone heard of an issue like this and or have a possible solution or any thoughts at all, thanks!