What best to way to backup of large amount data?

[unseengundam101]

I need method to back up TBs of data, I will be starting out with about 2 TB of data. Usually the amount of my data grows 1 TB to 500 gb a year. Eventually, I will have tons of data to back up. I am trying to think of effective way to backing that much of data. Eventually, I plan to build a dedicated Data Server for it. I just need figure out he hard drive configuration so I know how back up the data.

I was initially thinking about just using Raid 1 for all data. However, people here tend not advocate using raid especially the software kind. I can see problems with software raid, however I can't think any other way to quickly back up tons of data. What other ways are there to easily back tons of data?

 

[pm]

I'd recommend a NAS drive, and rsync (or it's Windows equivalent) and compression. Incremental backups are pretty fast with rsync (or the Windows versions which use the same method).

 

[RebateMonger]

Redundant RAID isn't a reliable backup method. There's too many things that can go wrong.

Your choices for this much data are tapes or hard drives. Suitable tape drives and tapes are likely more money than you'll want to spend unless this is for a well-off business.

The usual approach for small businesses is to use large removable or external hard drives: USB, SATA, or e-SATA. Up to a point, you can use single full backups with incremental updates. Be sure to periodically test the integrity of your backups or you might get a bad surprise when you need to restore your files.

Personally, I don't trust a single backup of really important data. Ideally, you keep two or more backup sets, keeping one or more offsite in case of fire, flood, theft, etc. It all depends on how much risk is acceptable to you.

 

[Nothinman]

However, people here tend not advocate using raid especially the software kind. I can see problems with software raid, however I can't think any other way to quickly back up tons of data. What other ways are there to easily back tons of data?

It's not that we don't recommend software RAID, it's that we don't recommend using those crappy onboard controllers and RAID1 isn't a backup solution. Frankly I love Linux software RAID, along side LVM it can do just about anything. RAID1 is fine even in Windows, higher levels of RAID aren't usually recommended because of the extra load they add to the CPU but that's a judgement call.

 

[MalVeauX]

Heya,

An easy and fast solution would be to get an eSATA controller into the system. Then just buy 1tb external eSATA capable drives. Connect. Backup. Disconnect. Store. To be safe, like others mentioned, do more than one backup set so that you double your failsafe in case something happens to one of them and don't store them both in the same place either. Doing this, HDD's in proper storage can last longer than you should ever need (ie, longer than your life time). Since you claimed about 1tb a year, you can easily get two 1tb eSATA drives each year. Right now, they're barely $100 each. $200 for a full double backup each year is pretty sweet considering it all. Think of it like $17 a month (the equivalent of an online MMO subscription). I mention eSATA because it's easy to connect, but mainly because it's so much faster than USB options and keeps you from getting inside the case (where you could potentially do something on accident and do damage). eSATA is as fast as if the drive was internal, so it will do very well for you.

You don't even have to get external drives. Another option is to just get a good eSATA capable enclosure, and buy internal 1tb drives. Just install the drive into the enclosure; backup; remove drive; store it. Replace with next drive in the same enclosure, and repeat the process. You could probably get better warranties this way as warranties on basic internals are longer served than warranties on external 3rd party enclosures with a drive inside.

I would not use RAID at all for this. There's no point. RAID is not about backing up, it's about uptime. RAID1 will not help your data either, again, that's just uptime.

If you plan to serve it all from a source one day, I would not serve it from your physical backups. Instead, just copy the backups to your own NAS and serve it that way, again, keeping backups separate from that. The NAS can run RAID all you please (RAID5 would be a good way to do that) and serve your data perfectly well. But again, this is not your backup, that's just your server. Keep your backups separate. It is more costly, but trying to keep years of data on any RAID array is asking for you to lose data. So make sure you do a real backup and keep it separate. Any thing you keep in use should be separate from backups.

Here's an example of the external eSATA (and USB) external 1TB drive: $120 @ NewEgg

Very best,

 

[unseengundam101]

Thanks for the replies. I however still have more concerns/questions.

One main thing I am want consider is easily automating it. Main reason I was liking raid, is basically you don't need to do anything extra. Also, would like note most of my data is non-critical. I usually sync my important backup data to my laptop + burn on DVDs once in awhile.

For example, Physically pluging in a 2 esata (for 2TB) every week (maybe everyday), running sync software (probably kill take long time), and then removing the drive will take a a lot of time. Eventually, I am sure I will get to 10 TB or 20 TB of total data, then whole process will get even more cumbersome. Then also the problem is how to spilit up the data. Say I have 10 TB Raid 5 on server, how do I automatically split up the data into 10 different drives? Probably some software for it somewhere?

Maybe, what I need is have a "Data Server" and then a separate "Backup Data Server". The Backup Data Server will automatically back from Data server every night or something like that? However, having both a Data and Backup data server seems like of expensive (maybe?)

I think maybe doing Linux Raid 0+1 or something might be easier solution for the tons of non-critical data. Any other suggestions?

 

[RebateMonger]

Originally posted by: thureen
One main thing I am want consider is easily automating it. Main reason I was liking raid, is basically you don't need to do anything extra.

Redundant RAID is great until it stops working correctly. At that point, without backups, you've got a serious problem. If your data is important, you should have a separate backup system. If the data isn't important, then don't bother backing it up. In any case, I wouldn't use RAID striping (like RAID 5) without backups, because a RAID 5 failure may very well cause you to lose EVERYTHING at once. It's bad enough to lose a single drive. Losing an entire array at once is worse.

Maintaining offsite backups of a 20 TB array is a big deal. Probably more than you'll want to tackle. So you are going to be pretty much stuck with onsite backups using a separate backup server. That doesn't have to be costly. Certainly no more so than RAID 0+1/1+0.

 

[SickBeast]

If you're adding critical data to the drive constantly on a daily basis, I would go with RAID-1, plus I would back that up on a weekly basis to an external drive. Ideally, you'll want to purchase 3 identical drives for this. For your needs, it sounds like you'll want 3 1.5TB drives as a minimum.

Otherwise, just back up the drive to an external drive daily; the worst case scenario is that you lose a day's work.

In terms of automation, there is software that will automatically backup your files in whatever increment you want.

You may want to go straight to a NAS (network attached storage) box if you need 10-20TB of storage initially; the external drives are simply too small for what you're doing. You could build a NAS box yourself.

 

[Team Spicoli]

Originally posted by: MalVeauX
Heya,


Here's an example of the external eSATA (and USB) external 1TB drive: $120 @ NewEgg


Hey Mal,

I was reading online that these external drives may house refurbished drives. Any suggestions to help narrow down the selection to non-refurb's?

Thanks as always