Reliable long-term storage

[angry hampster]

I'm a photographer who is slowly starting a business, and would like to keep backups of my work. Would a 3.5" external HDD be a reliable way of storing files? When not backing up files immediately after a shoot, it would sit in my safe in a static-free bag. Also considering a 3.5" internal HDD and a rosewill or thermaltake dock. I would just burn to DVDs but I can populate 4.7GB of space very quickly. Archival DVDs are also very pricey.

 

[sxr7171]

I would personally get a Windows Home Server to have local duplicated data for easy access and also use the plug-in for Amazon S3 for data that is really valuable to you.

You will keep moving the data to newer systems over the years and you won't be caught off-guard with disks that 25 years from now you might easily find a reader for.

If Amazon ever goes out of business I would hope that they would let you know to retrieve your data before they take the servers down.

 

[RebateMonger]

How much data do you have now? How much a year or two from now?

 

[TXAngel08]

How important is your data, and how much is it worth to you? For many people, an external hard drive works, however they make the mistake of not keeping copies of their data in multiple places. An external hard drive can fail just as quick as one inside your computer.

There are many ways to go about this, all of them have a cost. First, storing your backup locally (even in a safe), is not secure long term storage. A bank safety deposit box is a good start for off-site storage and it goes up from there.

You might also consider something like Mozy or Carbonite as another backup solution. For $50 a year, it gives you a second backup in addition to the hard drive storage. Personally for really important data, I wouldn't use such a service as my only option, but it is a good start and a decent backup to your backup if you're on a budget.

One issue to keep in mind is that you have no guarantee that a hard drive (or DVD-R, or tape, or anything else) stored for several years will work when powered back on, so how many copies of your data do you want to keep? Do you keep multiple versions, or are these static files? Do you reuse your backup medium, or do you archive the backup forever and make new backups?

No matter what option you choose, they all have failure points. Do you have a backup of your backup, and do you test your backups to make sure they work? Nothing is worse than thinking you have a backup, only to find out that you really don't.

 

[Blain]

I would trust no single method for archiving important data. I'm a firm believer in overkill...

* External HD - only connected to PC and wall power when backing up
* NAS - connected to network and wall power 100% of the time
* DVD+R - don't worry about gold archival disks, go with high quality Taiyo Yuden 16X (burned at a slower 8X)
* Online - cloud storage

 

[43st]

My wife worked in a art conservation lab at NYU for a bit (now with a independent conservator), this is a massive concern and not much is being done about it just yet.

Here's an article on digital decay. Also this Sterling piece, and this NYU piece.

Your best bet is a properly stored and protected hard copy. If you're less concerned then the above methods will probably work fine. I use a combination of active hard drive, CD backup and Carbonite. I have lost a handful of photos over the last 20 years to copying, unreadable CD, etc..

 

[TXAngel08]

And of course, if you have the money, there are services like this:

http://www.undergroundvaults.com/offerings/itemsstored/moviefilmstorage.cfm

They have underground mines 58 stories below ground in Kansas, Missouri and Kentucky.

Of course, as someone else has already said, digital decay is an issue. Making sure that you can read the information again in the future is an issue, so multiple forms of storage are required with backups of the backups...

 

[sxr7171]

I think cloud storage is good because it puts the onus on someone else to manage the data and usually those people are experts on managing that data. However anything can happen even at the best firms. So it wouldn't hurt to have 2 such providers.

As for externals I think they fail much more frequently than fixed HDDs. People think they can buy an external and be safe and then they cry when it fails. I mean you have a device that operates at micrometer tolerances that you carry around in a backpack. How would you really trust that?

IME external drives fail left and right. You pretty much have to manage your data yourself or have someone do it for you. Or both. If you manage it yourself, you have to keep moving it to new drives over the years and trust yourself not accidentally delete data while copying.

You cannot just put a HDD in a safe deposit locker either because they degrade over time. So you must actively manage your data.

Physical optical disks sound good with their promises of 99 year storage but how do you really know? How are you going to find a DVD reader in 2049? You may be able to or you may not be able to. We don't know.

Which is why again I believe you must manage your data actively. You must keep transferring it to newer systems or pay someone else to do it for you or both.


I just looked up Carbonite. OMG! $55 unlimited? How? I could put 4TB on that thing and still pay $55 a year? Wow! Much cheaper than S3.

 

[TXAngel08]

I have 6TB backed up to both Carbonite and to Mozy, yes it really is unlimited, yes they really will let you back up that much, and yes I've had to use them to restore data I deleted by mistake (it happens), they work just fine.

Mozy is faster on the upload, but Carbonite has a better backup design (it just runs all the time backing files up, Mozy runs at scheduled times.)

This is, at the end of the day, why I moved from Time Warner Cable to FioS. TWC gave me 1meg upload, FioS gives me 25 meg upload.

 

[RebateMonger]

Some good points have been made. I, too, don't believe in a single backup. Depending on how much data there is, I think that Mozy or Carbonite are a good choice as a secondary backup. I'd keep local backups as my primary backups because they are much faster for backup and restores.

As noted, any backup should be periodically tested. I test my client's hard diisk backups monthly, to ensure that they are actually readable and restorable. As long as they are actually BACKUPS, finding a problem with a backup system isn't a disaster by itself. You replace the disk or other media and rebuild your backups. But you need to test the backups so you know whether they are working like you think they are. When your primary data storage fails isn't the time to test your backups.

 

[taltamir]

if digital decay is an issue, then use parity.

ZFS has build in automatic parity and can most easily be gotten using opensolaris from genunix.org
search the genunix wiki for "getting started with CIFS" to share files to windows clients.

Or you can also use par2 (parchive) which is ideal for reconstructing damaged files (due to digital decay).
Store your items in RAID1 arrays ONLY. And use online backup in case of a fire or theft.
be warned that fire safes protect PAPER from COMBUSTING, it still reaches a temperature hot enough to damage all digital storage media.

 

[sub.mesa]

You would want checksums so you know when corruption occurs. Otherwise; your files may be accessible but silent corruption may kill the files. And you will overwrite your backup with corrupted versions of the files. That's why you need something like ZFS for any serious storage setup. Try to read and see if its something for you.

 

[Emulex]

par2 and par are good for adding extra parity to your files.

 

[sub.mesa]

Only those are not updated automatically when the file changes. Also, ZFS is able to automatically correct corruption when it is detected (on the fly) and will use redundant data sources to correct the file to uncorrupted version.

 

[TXAngel08]

if digital decay is an issue, then use parity.

That works, assuming you're concerned about unrecoverable read errors on a hard drive or DVD.

It doesn't work if the media itself fails (breaks down beyond readability). If you leave a hard drive in a vault for 50 years, I'd be quite shocked if the platters contained anything on them.

Transfering the data to new hard drives every few years and keeping more than one backup set is a better option. As someone else said, if you want real long term storage, it requires active management.

be warned that fire safes protect PAPER from COMBUSTING, it still reaches a temperature hot enough to damage all digital storage media.

This is a really good point. While there are no standards for fire testing safes, generally they are considered to have failed when the internal tempature reaches 350 degrees which is the point where paper starts turning brown. Most digital equipment will fail long before then.

Of course, for the right amount of money, safes can be purchased to do this job, but it would be cheaper I think to rent space in a bank vault.

 

[TXAngel08]

You would want checksums so you know when corruption occurs. Otherwise; your files may be accessible but silent corruption may kill the files. And you will overwrite your backup with corrupted versions of the files. That's why you need something like ZFS for any serious storage setup. Try to read and see if its something for you.

Yet another good reason to not overwrite your backups.

A friend of mine has a public data business, they make incremental backups every day and full backups every week. The weekly backups are kept forever and never overwritten. If memory serves, either every 3 or 5 years (forget which), they copy the backups to new hard drives to maintain data integrity.

Of course, their data is worth millions of dollars, so the cost of all that is justified.

 

[latch]

I firmly believe that if you want "reliable" storage, you need to think "not in my house". There's a ton of things that can go wrong even in a NAS with redundancy - the most likely being that you realize something is wrong when its too late.

You should at least be _looking_ at online solutions. You can get a flickr pro account for $25/year, which lets you restrict access, and lets you store and upload an unlimited amount of pictures. There are a lot of other online storage solutions.

I use jungle disk as an online storage for my most important files. I have about 10gb there.

I use unraid, which uses a parity approach for everything else. I find raid1/replication (as in windows home server) to be hugely cost ineffective past the 5TB mark. These are files that I can live if I lose (I can always ..uhmm...reacquire them if need be).

 

[43st]

One can also use zip compression in archive mode to help prevent data loss as well. But you are still constrained to the media and hardware of the day... so don't store it on an 8" floppy. ^_^

 

[Dorkenstein]

Sorry to bump, but does Carbonite allow you to back up anything at all? With the exception of your active windows system stuff, that is.

 

[43st]

Sorry to bump, but does Carbonite allow you to back up anything at all? With the exception of your active windows system stuff, that is.

Carbonite adds a right click option that's basically 'back this up'. I have my photos, documents, and design work folders selected and it backs them up. It adds a little dot on the file icon that is either orange (has not been uploaded yet) or green (has been uploaded).

It might exclude .exe files, but I think you can manually add them.

 

[Emulex]

the problem is the compression and deduplication datastores increase the risk of corruption so you need to consider that in all practice of backup.

why i break up rar files or .7z files to 4GB even though i'm moving from 5tb to 2TB disk to disk - limiting the potential loss if a sector goes bad (offsite is kinda abusive).