How to protect backups - correct errors

[toad096]

Hi,
I am trying to define a backup strategy, where I store full backups on a set of external disks (inside a set of truecrypt containers). But i want to cover the eventuality (which will happen sooner or later) that a particular truecrypt container will become unreadable due to bitrot, defective sectors, whatever...

I came across multipar:
http://http://en.wikipedia.org/wiki/Parchive

which generates parity data, in order to be able to correct corrupt data.

Can someone recommend any alternative approaches? Share their experience?

Any alternatives to multipar utility? I could not find any reviews.

My platform is powerful Windows 7 and Windows 8 desktops, using a set of external USB3 disks and SATA disks in removable frames for backups.

Thanks.

Toad

 

[Mark R]

RAR/WinRAR offers a similar system to PAR files, where the recovery data is integrated into the compression tool.

If you just want to store the files "loose" in folders (rather than in a RAR file) then I really haven't found an alternative to PAR2 (for which multipar is the best solution).

Multipar is what I use on my archival backups (e.g. photos, home movies, other stuff being stored for posterity, tax/business records, etc.)

 

[toad096]

Thanks, Mark. This is exactly the info I am looking for. But did any of your datasets ever get corrupted and you had to use Multipar to recover?

 

[Elixer]

PAR2 works great, and yes, I had some DVDs that I had to use par2 to recover from the broken files, and it worked out fine.

Note, you need to make sure you have enough par2 blocks to be able to recover from very bad situations. This is also why when you do par2, it could end up being about twice the space of the original files (if you go 100% coverage).

 

[toad096]

Thanks Elixer!

Will try out both WinRar and MultiPar.

 

[Mark R]

Just bear in mind that due to a limitation in the PAR2 format, each file you use must take up a minimum of 1 "data block". "Data blocks" can be any size you want, but there is a max of 32k "data blocks" in any one PAR2 collection. So if you have large number of small files (e.g. JPGs) and a bunch of huge files (e.g. videos) then PAR2 will give very poor space efficiency. The algorithm also gets super slow once you go over about 1 GB of files.

In general, PAR2 works well if you give it a reasonable sized data set where all the files are vaguely similar sized (don't have 10,000 text files in a folder full of 30 megapixel photos).

Multipar is better than any other PAR2 system. It is the only one that supports subdirectories - PAR2 officially only supports 1 folder, and can't store sub-folders. Many of the others also have subtle bugs in their algorithms - the files and recovery work, but the degree of protection is decreased from the theoretical maximum (in one case, catastrophically badly decreased).

PAR3 is offered in Multipar (PAR3 is a prototype algorithm invented by the author of multipar which fixes a number of limitations of PAR2 and uses a more advanced algorithm) - but it is still only prototype code, and is changed regularly. Because of the risk, that future multipar versions may not be able to read old PAR3 files, you should not use it for any important data.

I've never had any data corrupted since I've started using multipar. I started it a few years ago, when I had some photos corrupted by a flaky USB hard drive enclosure. However, since then, I've been lucky and not had any "bit-rot". However, I have done some basic testing by deleting files, and using the PAR2 files to recover the deleted file, or corrupting files with a hex-editor and successfully repairing them.

 

[toad096]

Thanks Mark, I appreciate the detailed reply!