Best way to verify successful transmission of many files?

[kstu]

I'm pretty new to this stuff and figured I would pick the collective AT brain and hope you can teach me a few things. Also, I wasn't even sure what forum to post this in, so hopefully I chose correctly.

Anyways, we have a project that will involve the migration of up to tens of thousands of files at a time. I was wondering what tools and/or programs are available for file verification of that many files.

Let me know if you need any specific information and hopefully I'll be able to provide it. Like I said, I'm pretty inexperienced, so you might need to hold my hand a little more than usual.

Thanks.

 

[Cogman]

MD5Sum will do the trick for you (just what it was built for). to generate an MD5Sum file you would run the command (in the folder you are transferring)

md5sum * > files.md5

To check the files after the transfer you would run

md5sum -c files.md5

hope that helps. I should also note that this doesn't go through the directories, it only generates a sum file for the directory you are in. Someone might know a better command to iterate through sub-directories.

 

[Modelworks]

Zmodem

 

[SleepWalkerX]

Generating an md5 is a good way to check for validity. Depending on what exactly you're working with, I would just set up a torrent. It takes care of transferring + checking file validity. I've done this with Azureus over a lan several times and it works quite well, especially with multiple people. But I guess it really depends on what you're transferring.

 

[QED]

How do you plan on transferring the files? Are the two systems which are sending/receiving the files on the same operating system?

The md5sum utility Cogman mentions works great with individual files, or a directory of files, if both systems are on a Unix platform.

If the files are scattered through a directory in many subdirectories, you can still use the md5sum utility by "tar"-ing" or "cpio"-ing the whole directory structure you wish to transfer first, and then running the md5sum utility against the resultant tar file or cpio file. After the files are received by the second system, repeat the tarring or cpio-ing process on that host, and run the md5sum utilityi against that file and compare. You might want to consider just transferring the original tar or cpio file since you've already generated it, and it's typically easier to detect an error in the transmission of a single (albeit large) file than it is to detect an error in the transmission of one of tens of thousands of smaller files.

 

[Cogman]

Originally posted by: QED
How do you plan on transferring the files? Are the two systems which are sending/receiving the files on the same operating system?

The md5sum utility Cogman mentions works great with individual files, or a directory of files, if both systems are on a Unix platform.

If the files are scattered through a directory in many subdirectories, you can still use the md5sum utility by "tar"-ing" or "cpio"-ing the whole directory structure you wish to transfer first, and then running the md5sum utility against the resultant tar file or cpio file. After the files are received by the second system, repeat the tarring or cpio-ing process on that host, and run the md5sum utilityi against that file and compare. You might want to consider just transferring the original tar or cpio file since you've already generated it, and it's typically easier to detect an error in the transmission of a single (albeit large) file than it is to detect an error in the transmission of one of tens of thousands of smaller files.

The only thing I have against taring the whole directory is
1. It can take up a lot of spaces.
2. It doesn't allow for a modular transfer IE with all the files separate an MD5 sum will tell you that files 1, 2, or 3 is broken. and those will be the only files you have to transfer. With a giant tar you get told that you need to re transfer the entire tar.

A torrent might be a good alternative as it does allow for both checking and transferring of the files. I haven't do that myself though.

 

[Netopia]

What about rsync on both sides?

Joe

 

[sourceninja]

rsync is a very good option imho.