Go Back   AnandTech Forums > Hardware and Technology > Networking

Forums
· Hardware and Technology
· CPUs and Overclocking
· Motherboards
· Video Cards and Graphics
· Memory and Storage
· Power Supplies
· Cases & Cooling
· SFF, Notebooks, Pre-Built/Barebones PCs
· Networking
· Peripherals
· General Hardware
· Highly Technical
· Computer Help
· Home Theater PCs
· Consumer Electronics
· Digital and Video Cameras
· Mobile Devices & Gadgets
· Audio/Video & Home Theater
· Software
· Software for Windows
· All Things Apple
· *nix Software
· Operating Systems
· Programming
· PC Gaming
· Console Gaming
· Distributed Computing
· Security
· Social
· Off Topic
· Politics and News
· Discussion Club
· Love and Relationships
· The Garage
· Health and Fitness
· Merchandise and Shopping
· For Sale/Trade
· Hot Deals with Free Stuff/Contests
· Black Friday 2014
· Forum Issues
· Technical Forum Issues
· Personal Forum Issues
· Suggestion Box
· Moderator Resources
· Moderator Discussions
   

Reply
 
Thread Tools
Old 11-13-2012, 02:14 PM   #1
alfa147x
Lifer
 
alfa147x's Avatar
 
Join Date: Jul 2005
Location: Georgia
Posts: 29,844
Question FTP transfers failing due to unstable internet. Another transfer protocol?

We use a VPS for temp storage for transferring video footage. My problem is that my internet connection at my apartment is quite unstable and the FTP client (CyberDuck) will throw up a error message. Then I can hit resume and it'll download the rest of the file. Sometimes the file will come down with no issue but 75% of the time it'll have errors. Is there a better way that I can get these files down?

The internet is bundled with rent and they block all P2P traffic so I couldn't even use a torrent setup (if even possible) plus my VPS host would not be fond of that.

These files are not too large. It's mostly video shot with iPhones / iPads and very few from a 5DII so total size is approx 5 - 10gb per transfer.

Any ideas?

Thanks,
Alfa
__________________
.
alfa147x is offline   Reply With Quote
Old 11-13-2012, 02:31 PM   #2
skillyho
Golden Member
 
skillyho's Avatar
 
Join Date: Nov 2005
Location: Columbia, SC
Posts: 1,322
Default

Is there a specific reason you're using a VPS? I'm assuming your ISP provided it? Just curious...

I would probably setup a Filezilla server on the virtual machine and send the files in broken RAR chunks (2GB) and let them reassemble on the other side. This wouldn't be such a pain if something timed-out/disconnected and you could pick back up easily.

Why it's happening with the system you have is difficult to troubleshoot without knowing more. Does your internet connection drop, but you maintain network connectivity? Does this happen consistently with large file transfers?
skillyho is offline   Reply With Quote
Old 11-13-2012, 02:37 PM   #3
Nothinman
Elite Member
 
Nothinman's Avatar
 
Join Date: Sep 2001
Posts: 30,672
Default

Sounds like you just need a client that will automatically retry which I thought most did these days. I would probably use a tool like wget which lets you set how many times to retry and how long to wait between them.
__________________
http://www.debian.org
Nothinman is offline   Reply With Quote
Old 11-13-2012, 02:58 PM   #4
mikeymikec
Diamond Member
 
mikeymikec's Avatar
 
Join Date: May 2011
Location: UK
Posts: 4,332
Default

With FileZilla you can specify the number of auto retries IIRC.

Yup, in preferences, specify the number of retries and the number of seconds between retries.
mikeymikec is offline   Reply With Quote
Old 11-13-2012, 03:06 PM   #5
alfa147x
Lifer
 
alfa147x's Avatar
 
Join Date: Jul 2005
Location: Georgia
Posts: 29,844
Default

Quote:
Originally Posted by skillyho View Post
Is there a specific reason you're using a VPS? I'm assuming your ISP provided it? Just curious...

I would probably setup a Filezilla server on the virtual machine and send the files in broken RAR chunks (2GB) and let them reassemble on the other side. This wouldn't be such a pain if something timed-out/disconnected and you could pick back up easily.

Why it's happening with the system you have is difficult to troubleshoot without knowing more. Does your internet connection drop, but you maintain network connectivity? Does this happen consistently with large file transfers?
I got the VPS to experiment with and then we needed a way to transfer large files so I had the other guys dump files on to it. I still want to keep it to mess with. The reason I think our internet sucks is because the backbone wasn't setup to handle so many people. Normal usage it's fine but for transferring files that take 4 - 5 hours to download at some point the internet breaks and the transfer fails.

Quote:
Originally Posted by Nothinman View Post
Sounds like you just need a client that will automatically retry which I thought most did these days. I would probably use a tool like wget which lets you set how many times to retry and how long to wait between them.
I'll look into using wget. I didn't realize you could use it to pull FTP files. Thanks. When a client automatically retries does it start from 0 or picks up where it left off? If it picks up would that possibly create errors?
__________________
.
alfa147x is offline   Reply With Quote
Old 11-13-2012, 03:07 PM   #6
alfa147x
Lifer
 
alfa147x's Avatar
 
Join Date: Jul 2005
Location: Georgia
Posts: 29,844
Default

Quote:
Originally Posted by mikeymikec View Post
With FileZilla you can specify the number of auto retries IIRC.

Yup, in preferences, specify the number of retries and the number of seconds between retries.
Sweet. I'll have to use that. When it retries does it start at 0 or does it pick pup where it left off?
__________________
.
alfa147x is offline   Reply With Quote
Old 11-13-2012, 03:10 PM   #7
imagoon
Diamond Member
 
imagoon's Avatar
 
Join Date: Feb 2003
Location: Chicagoland, IL
Posts: 4,787
Default

rsync can also be useful since it does rolling checksums on the data so an invalid chunk gets discarded.

The corruption issues is harder to deal with when working with FTP. Some servers can be configured to drop a few k from a file on a lost connection so that the resume in theory start over prior to the junk data.
imagoon is offline   Reply With Quote
Old 11-13-2012, 03:21 PM   #8
mikeymikec
Diamond Member
 
mikeymikec's Avatar
 
Join Date: May 2011
Location: UK
Posts: 4,332
Default

Quote:
Originally Posted by alfa147x View Post
Sweet. I'll have to use that. When it retries does it start at 0 or does it pick pup where it left off?
It would be bloody silly if it started again at zero, so I assume it doesn't

Seriously though, I think that resuming transfers over FTP requires server-side support, but I think that support came in to mainstream FTP servers more than a decade ago.

Last edited by mikeymikec; 11-13-2012 at 03:29 PM.
mikeymikec is offline   Reply With Quote
Old 11-13-2012, 10:05 PM   #9
alfa147x
Lifer
 
alfa147x's Avatar
 
Join Date: Jul 2005
Location: Georgia
Posts: 29,844
Default

Okay. Setup ProFTPD on the VPS to allow resuming downloads. Now I'm pulling down a 8gb file @ 500 KB/s ETA 9 hours from now. I also setup cyberduck to retry the download after waiting 40 seconds and 9 total retry attempts.

Thanks for the help!
__________________
.
alfa147x is offline   Reply With Quote
Old 11-14-2012, 03:16 PM   #10
alfa147x
Lifer
 
alfa147x's Avatar
 
Join Date: Jul 2005
Location: Georgia
Posts: 29,844
Default

Just checked the files. The MD5 checks out. Thanks for the help!
__________________
.
alfa147x is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 02:49 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.