Best way to copy massive ammounts of data across network? (from Linux sys to OpenBSD sys)

[Electrode]

Hello again one and all. I think I have fixed what was, at the time, a hair-loss-inducing networking problem I encountered when I first tried to set up OpenBSD on my server, and I'm about to give it another try. However, I do have a slight problem I will have to overcome: how to get my /pub stuff (50 GB altogether) from the ReiserFS partition it is on now to an FFS partition that OpenBSD can comprehend. The simplest way to go about it that I can think of is to remove the /pub drive from the server and put it in a workstation, fire up some sort of filesharing daemon, pull everything over the network, then put the drive back in the server, mkffs (?) it, and move everything back to it. A number of issues arrise here, however:

1. The only network filesystem I know anything about is SMBFS, and it doesn't appear to preserve all of the file attributes. Plus I don't even know if OpenBSD includes Samba or not.
2. I could use FTP, but I don't know any way to mass-copy files. I don't think it preserves attributes either.
3. Some of the files are very large, exceeding 2 GB. I know from past experiences that nothing likes files this big.

And one question that doesn't really relate to the above, but is there any way to make OpenBSD request a specific IP when it does a DHCP update? I've gotten used to this method of getting a semi-static IP, and I'm not willing to give it up.


Thanks in advance!

 

[nuttervm]

1. The only network filesystem I know anything about is SMBFS, and it doesn't appear to preserve all of the file attributes. Plus I don't even know if OpenBSD includes Samba or not.

If you want to share a dir on your openbsd box, you can use samba like any other unix variant. f you want to access files via smbfs and get files from a smb share on teh network over to your obsd box, then you need a port/package called sharity-light. its the openbsd way of mounting remote smb shares.

2. I could use FTP, but I don't know any way to mass-copy files. I don't think it preserves attributes either.

that would be an option, but i don't know if it preserves attributes either. usually you can specify your ftp client to preserve them, but i am not sure. your best bet here would be transfer a few files with different permissions/users/sizes etc and see if everything is as it should be.

3. Some of the files are very large, exceeding 2 GB. I know from past experiences that nothing likes files this big.

i'm not really sure about this one. I'm looking around and if/when i find something i'll post it.

And one question that doesn't really relate to the above, but is there any way to make OpenBSD request a specific IP when it does a DHCP update? I've gotten used to this method of getting a semi-static IP, and I'm not willing to give it up.

why not just hard code the IP address? problem solved. if that is not an option, in my experience, you will almost always get teh same ip address when using dhcp. also, generaly speaking, once you have your obsd box set up you will never reboot it, and therefore you will always keep the same ip. i knwo this is not a universal truth, but it has held true.

 

[drag]

NFS? I think Networking File sharing protocal was created to solve these problems, You should be able to mount a remote NFS file system on your local directory stucture like any other partition/cdrom/floppy thingy.

 

[Tiger]

Ditto NFS. It's only going to be as fast as the slowest NIC but it should work.

 

[Electrode]

I was thinking of NFS too. The thing is, I don't know anything about it. I've heard you mount it the same way as SMB, just with -t nfs instead of -t smbfs, but I'm not sure how to set up NFSD on either Linux or OpenBSD. Any tips?

 

[n0cmonkey]

ftp

mput is the command to put multiple files somewhere.

mget is the command to get multiple files from a location.

prompt will tell the stupid ftp server not to ask your permision to download the file you already told it to send you.

 

[n0cmonkey]

And one question that doesn't really relate to the above, but is there any way to make OpenBSD request a specific IP when it does a DHCP update? I've gotten used to this method of getting a semi-static IP, and I'm not willing to give it up.

man dhclient

 

[manly]

NFS if you're willing to set it up (not too much work).

I could be wrong w.r.t. file attributes, but rsync would probably be my first choice.

 

[Tiger]

Here's the Howto I used to get 20 student work stations and 1 server cooking in 2 hours time.
The students were all Linux vigins. Ignore the urpmi and RPM crap if you aren't using one of those distro's.
Linky

 

[Electrode]

Thanks for the link Tiger!

 

[TonyRic]

Electrode, if you have sshd running on one of the machines you can use it to make the transfers.

scp -rp tony@my_machines_ip:/pub/* /pub

This will copy ALL files from "my_machines_ip" /pub dir to the local /pub. If you want to push the files instead of pulling them (the example above is a pull) just reverse the 2 file directives.

man scp will help you more if you need it. This is a VERY efficient way of doing this. I redo my web/mail/file server every year and I move about 30GB of data and always do it this way.

 

[Electrode]

Didn't know about that. I would do that, but the last time I tried to SCP a 4 GB file, it didn't work. I'm pretty sure my only option is NFS, so that's what I'm going to use.

 

[nuttervm]

when using scp, make sure you are using the blowfish algorithm. by default ssh uses 3des, which is a higher level of encryption and therefore slower performing. i know with my p133 firewall i can transmit at about 175 kb per sec with 3des on and 500-700 with blowfish. i am running on a switched network so this is entirely because of the slow computer and the encryption going on.

 

[TonyRic]

As I recall, rcp and scp do not have file size limitations inherent to the programs themselves. The limit you may be referring to is a filesystem limit of 2GB per file.

I have used scp and rcp to copy multi-GB files with no issues.

 

[n0cmonkey]

scp can be slow and machine intensive, thats why I recommended ftp. On my local network I usually use scp, but if I want to transfer a lot of files or a large file I use ftp.

EDIT: ftpd is only run on my local systems on local interfaces (not facing the world) and promptly turned off after transfer is complete

 

[TonyRic]

n0c, I would normally agree, however there are some (like me) that refuse to install any protocol that authenticates in the clear. SSH will allow sftp, scp and so forth and should support ftp and rcp as a fall through if you config it properly. However, the performance gained imo is not worth the security risks if one is on any connection except dialup.

 

[n0cmonkey]

Originally posted by: TonyRic
n0c, I would normally agree, however there are some (like me) that refuse to install any protocol that authenticates in the clear. SSH will allow sftp, scp and so forth and should support ftp and rcp as a fall through if you config it properly. However, the performance gained imo is not worth the security risks if one is on any connection except dialup.

I agree with you to an extent, but there are reasons I feel "safe" using ftp on my home network. First of all, Im the only user. Second, I dont use it often. Third, if my machine is cracked and someone is sniffing my internal network, Ive got worse problems than them getting the password to a non-root, fairly unprivledged user

 

[TonyRic]

n0c, point taken.

 

[n0cmonkey]

Originally posted by: TonyRic
n0c, point taken.

Thanks, but you definitely brought up good points. On most networks I would definitley use scp and "eat" the lost time. My network is pretty "clean" though.