Any way to upload video clips and avoid corruption?

[fuzzybabybunny]

I'm not sure which forum to put this in.

I have a videographer shooting video clips for me, and then she FTPs each clip to my server where I download them and cut them into a video.

Last night we had 3 clips out of 33 show corruption artifacts while played. I asked her to check her original clips and she says they had no corruption, so it seems that something happened on the transfer. The clips still play, but it has those artifacts in about one second of the video, making the entire clip useless.

What should I do to ensure no corruption when she's transferring her video files to me?

 

[Markbnj]

Artifacts in video are usually a result of compression. I won't say it's impossible for an FTP transfer to complete successfully and result in corrupted data, because people keep claiming it happens, but it hasn't happened to me. FTP over TCP should utilize the TCP checksum throughout the chain, so in that case specifically it just shouldn't happen.

To make sure, have your videographer take an MD5 hash of the file before it's sent, and then run the hash on your end to make sure you get the same value.

 

[KLin]

How big are the clips? maybe try dropbox to see if you get the same issue?

 

[Train]

Artifacts in video are usually a result of compression. I won't say it's impossible for an FTP transfer to complete successfully and result in corrupted data, because people keep claiming it happens, but it hasn't happened to me. FTP over TCP should utilize the TCP checksum throughout the chain, so in that case specifically it just shouldn't happen.

To make sure, have your videographer take an MD5 hash of the file before it's sent, and then run the hash on your end to make sure you get the same value.

ya I was thinking the same thing. I think it is more likely different codecs than an actual file corruption.

 

[DaveSimmons]

It could be compression or codecs, but to be safe you could run the video through 7-zip or WinRAR to place it in an archive before uploading. If that takes too long just use a program that generates an MD5 hash from the file.

When you de-archive it the 7-zip or WinRAR program will do error checking and can tell if the file was corrupted during the transfers up or down.

 

[fuzzybabybunny]

There are a number of issues:

There is no compression going on.

1. She shoots the video clips and just sends them straight to the FTP. She doesn't touch them. She said her originals did not have artifacts, and the originals were what she put on the ftp.

2. We were having serious issues with sending zips files. The files are like 4GB in size, and the chance of them being corrupt after upload and rendering the entire file non-operational was very high, which is why we decided to send individual clips. If one clip was bad, at least I had other clips that I could use and she would only have to upload one clip instead of the entire 4GB archive.

 

[DaveSimmons]

Why can't she zip each clip then?

On my PC, I just mouse over a filename, right click, WinZip > Add to (filename).zip -or- 7-zip > Add to (filename).7z

Repeat for each file.

 

[fuzzybabybunny]

Why can't she zip each clip then?

On my PC, I just mouse over a filename, right click, WinZip > Add to (filename).zip -or- 7-zip > Add to (filename).7z

Repeat for each file.

I think the goal is to make sure that corruption will never exist in the first place on transfers. I don't want to download a zipped clip only to find that it's corrupt, and by that time it's like 3AM her time and I've got a deadline to meet during that morning. I won't have time to ask her to re-up the clip.

Basically, I need a bombproof way to transfer files that ensures that they're not corrupt in any way by the time they reach me.

 

[DaveSimmons]

If you control the server (dedicated or VPS instead of just a hosting account), you could upload the zip and have the server run a script to scan it for checksum errors. Linux "zip" or PHP's zip library can scan an archive for errors.

You might also be able to install some custom FTP server program on the server that does some custom client - server communication to test files of any kind for corruption.

If you leave your PC on you could use some site - to - site transfer program on both ends that uses checksums. Maybe there's a LapLink version (eleventy) that does this?

 

[Ken g6]

A private bittorrent tracker occurs to me as a simple way to do it. She can set up a tracker with a bittorrent client, and you can be the only one downloading from it - or even knowing where to look for it. Any corrupt chunks should get rejected and re-requested.

 

[TheRyuu]

Is there some binary vs ascii transfer modes going on here or what? I'm not saying it's not possible for something to get corrupted but TCP is supposed to protect against this sort of stuff and the fact that you have not 1 (which I might believe) but 3 makes me think there's something else fishy going on.

Short of forcing certain ftp clients or server side scripting or something long and convoluted, the easiest way is probably to just make some par2 files over an entire set and upload that at the end. Then you can download that and run it with quickpar or whatever just to make sure it's the same on both ends.

Thanks to the way par2 works it doesn't matter if that stuff gets a little corrupted when it gets uploaded (or downloaded). Plus if anything is corrupt you can just repair it right provided you have enough recovery blocks (I can't see you not either, it sounds like it's just a small little bit somehow getting corrupted).

To make sure, have your videographer take an MD5 hash of the file before it's sent, and then run the hash on your end to make sure you get the same value.

It sort of goes along the same line as this except with par2 you get the chance to repair the problems as well. Another thing you might consider is throwing like the crc32 in the filename if you just want a quick check but if you're already breaking out par2 then there's no reason to do either of these things.

 

[fuzzybabybunny]

I had her transfer the 3 corrupt files to me again via the ftp and this time there was no corruption.

The old files compared to the new files had different file sizes. Something happened during the FTP transfer...

I feel like there's gotta be an easier way than creating PAR2 files. Creating PAR2 adds 10 minutes to each project that we do.

 

[TheRyuu]

I had her transfer the 3 corrupt files to me again via the ftp and this time there was no corruption.

The old files compared to the new files had different file sizes. Something happened during the FTP transfer...

I feel like there's gotta be an easier way than creating PAR2 files. Creating PAR2 adds 10 minutes to each project that we do.

Well that's the small price you're going to pay for the ability to repair files (which you said was sort of important). Without knowing more specifics such as any issues with the transfer mode (which could cause difference in file sizes), although I'm not sure how that would manifest itself (or if such files would even be playable).

It's certainly easier and quicker to do:

md5sum /path/to/videos/* > MD5SUMS

On her computer and compare it with what you get after you download it if you at least want to check but you said that the ability to not fix the issue at the same time was sort of make or break.

The only thing I can suggest is to diff the corrupted files with the non-corrupted ones and see what's different (for issues stemming from binary vs ascii transfer modes with ftp look for the hex codes for line endings LF and CR respectively: 0x0A 0x0D). Perhaps we can try and narrow down the issue here based on that. If she did everything exactly the same then I suppose you could rule that out. Maybe it was just a one time issue, and if so, maybe just making some md5sum's are the way to go since it's very unlikely that it's going to happen again.