Storage Planning (NAS + RAID + backup plan)

[sao123]

So about a year ago I bought I Synology Diskstation DS212j to server as our primary shared fileserver/FTP, placing 2 1TB drives in a Raid 1 configuration, in case of drive failure, and weekly backups to a 1TB External USB Drive.

Originally intended to hold our Family pictures, home movies of the kids, Music Collection, financial documents etc...It appears a year later, I woefully undersized our storage needs, as as soon as we began using it... we began saving things never before thought of... Including Acronis Images/TV & Movies/Game Libraries/Etc...

So now I intent to build something a little more right sized.
I believe I am going to get a Synology DS414 or DS1513 4 or 5 bay NAS, and probably start out with 3-4 3TB drives.

I've been reading NAS forums and articles and there is so much conflicting information, I just need some best direction.

I recognize that there is no *best* NAS solution available, and as with everything there are tradeoffs... but having trouble making some decisions.

ITEM 1... I don't trust hard drives... I've been using WD / Seagate / Maxtor desktop drives for almost 16 years, and I have had a failure rate of about 30%.

Which drive is the right drive... I have had far more failures in seagates than WD's, so im leaning toward either WD Reds or WD SE's. I like the 5 year warranty of the SE, and 3 TB is just under $200 ea.

ITEM 2... So to protect against drive failure... I need help picking the right RAID (raid 1 under the 2 disk NAS was the only choice).
* RAID 5 seems very common, and also very susceptible to 1 drive failure plus 1 read error = failed volume.
* RAID 6 fixes this, with a heavy performance penalty, really only beneficial if HDDcount >= 5.
* RAID 1 or 10 you lose half of all your drive space.
* Truthfully, I wish Synology would have a drive scheme like the UNRAID platform (Parity but not striped)... Which I also considered instead of the Synology, but that has its own set of problems. - I like the Synology interface better and the UNRAID has its own performance penalty.

Obviously all this is just theoretical... in real life experience how often is RAID 5 not- rebuild able? How much of a performance epenalty really is RAID 6? I know I want something which can recover from a drive failure, without having to do a full restore from the backups... I just need to understand the REAL factor of cost vs reliability vs performance...




ITEM 3, how to backup a 12 TB NAS? Backing up 1 TB of data on a USB drive is pretty straightforward. Barring some sort of heavy compression, backing up 4TB of data on an external drive is not going to fit on a single USB drive... Backing up a NAS to another identical NAS hardly seems like a cost effective scenario, and not easy to switch offsite every week. My DSL isn't the fastest, so a cloud service most likely is not a possible choice.


I know Synology has plugin capability on their new models... But I'm not familiar with any of the plugins they currently offer for download, as my DS212j has limited abilities.

I'm only building file/storage... I don't plan to do any transcoding or media center stuff on this box... I just need a large reliable primary storage facility.

HELP!

 

[BonzaiDuck]

Well, you may be a relative newcomer to this, I don't know. While some, such as yourself, feel comfortable with "NAS" units which you fill with your own hard-drives, I'm comfortable in my "server-rut." I like getting extra use from older equipment matched with selective new-tech items like SATA controllers, good gigabit NICs, etc.

There must have been a time when my server 'capacity" was maybe 60GB? 160GB? Then with 160GB HDDs -- maybe 480GB?

I've kept all my "professional" files, business-professional-academic e-mails, anything I've ever written, e-mail archives, a software library, personal-financial, document facsimile archives -- since maybe 1985. I regret losing some research I did in the mid-'80s on "rational expectations and the business cycle" -- with econometric work done with a software package I paid a couple hundred Franklins for. The files were on 800KB 5.25" floppy disks, and I never thought to keep the drive that read those disks, or transfer them to 3.5" -- or whatever.

I've used a tape-backup system, too. That's a lotta trouble and time.

RAID1 and RAID5 are great, but you can't just break up the disks or move them to another controller. You have to keep track of which disk is -- well -- which in the array. I like drive pools better. I like file duplication within drive pools even more. With that, you have duplicate copies of folders you choose which will always be on separate disks -- in case one of them goes south.

Let's suppose your "serious" files -- financials, document facsimiles, spreadsheets, correspondence, "taxes" etc. are even as much as 100GB without duplication. I'd be surprised if they're more. But suppose they are. You may have a large Picture collection, and the resolution chosen for your photos then determines file size and the size of the archive. Maybe those, too, are important.

If there are DVR recordings or movies -- those you really want to save can be moved to a USB drive. It's your choice whether you want duplicated redundancy for those things on a server.

It boils down to this for me -- maybe but not necessarily, for you: I can get all of the important stuff on a 1TB HDD by simple file & folder copying. I can get the important stuff plus the movies on a 500GB drive using compression and drive imaging.

If you want to run a lot of spinners adding to your power-bill, you can have "server" or "NAS" backup in the background. You can sync the files with the backup server.

I figure I can afford to lose up to a couple weeks of e-mails, recent files or data -- but no more. So I first have the file-server duplication for redundancy. I do backups of the important folders at least every two weeks.

I've had about five "file-servers" since the late 1990s. The first was just an older computer which became a collection point for used hard disks, first with NT 4.0 server, then with the Windows 2K Pro OS. Then I had another older computer -- same thing, really -- with Win 2000 Server. After that, XP was an interim server OS with a RAID5 array. Eventually, newer "older" computers configured with WHS v.1 and then WHS-2011.

Sometimes, it scares me no less, when some disk goes south, and I begin to worry about "data loss." But I have a tiered backup strategy. And by this time next month, server capacity will be either 6 or 8TB.

I'm not going to worry too much about the DVR captures or movies. So I can still do the same folder-backups and image backups to drives of 1TB or less.

 

[sao123]

Well, you may be a relative newcomer to this, I don't know. While some, such as yourself, feel comfortable with "NAS" units which you fill with your own hard-drives, I'm comfortable in my "server-rut." I like getting extra use from older equipment matched with selective new-tech items like SATA controllers, good gigabit NICs, etc.

There must have been a time when my server 'capacity" was maybe 60GB? 160GB? Then with 160GB HDDs -- maybe 480GB?

I've kept all my "professional" files, business-professional-academic e-mails, anything I've ever written, e-mail archives, a software library, personal-financial, document facsimile archives -- since maybe 1985. I regret losing some research I did in the mid-'80s on "rational expectations and the business cycle" -- with econometric work done with a software package I paid a couple hundred Franklins for. The files were on 800KB 5.25" floppy disks, and I never thought to keep the drive that read those disks, or transfer them to 3.5" -- or whatever.

I've used a tape-backup system, too. That's a lotta trouble and time.

RAID1 and RAID5 are great, but you can't just break up the disks or move them to another controller. You have to keep track of which disk is -- well -- which in the array. I like drive pools better. I like file duplication within drive pools even more. With that, you have duplicate copies of folders you choose which will always be on separate disks -- in case one of them goes south.

Let's suppose your "serious" files -- financials, document facsimiles, spreadsheets, correspondence, "taxes" etc. are even as much as 100GB without duplication. I'd be surprised if they're more. But suppose they are. You may have a large Picture collection, and the resolution chosen for your photos then determines file size and the size of the archive. Maybe those, too, are important.

If there are DVR recordings or movies -- those you really want to save can be moved to a USB drive. It's your choice whether you want duplicated redundancy for those things on a server.

It boils down to this for me -- maybe but not necessarily, for you: I can get all of the important stuff on a 1TB HDD by simple file & folder copying. I can get the important stuff plus the movies on a 500GB drive using compression and drive imaging.

If you want to run a lot of spinners adding to your power-bill, you can have "server" or "NAS" backup in the background. You can sync the files with the backup server.

I figure I can afford to lose up to a couple weeks of e-mails, recent files or data -- but no more. So I first have the file-server duplication for redundancy. I do backups of the important folders at least every two weeks.

I've had about five "file-servers" since the late 1990s. The first was just an older computer which became a collection point for used hard disks, first with NT 4.0 server, then with the Windows 2K Pro OS. Then I had another older computer -- same thing, really -- with Win 2000 Server. After that, XP was an interim server OS with a RAID5 array. Eventually, newer "older" computers configured with WHS v.1 and then WHS-2011.

Sometimes, it scares me no less, when some disk goes south, and I begin to worry about "data loss." But I have a tiered backup strategy. And by this time next month, server capacity will be either 6 or 8TB.

I'm not going to worry too much about the DVR captures or movies. So I can still do the same folder-backups and image backups to drives of 1TB or less.

Unfortunately, while I considered building a home server... I simply have no leftover hard available to build with (that's what I built my hyperspin arcade out of). Building a home server would require buying all the components (mobo/cpu/ram/power/drives/case)... It just seems a more straightforeward choice to buy an out of the box solution.

Looking at my data: About 1.5 TB of it is what I would consider irreplaceable... This is what I definitely want to make and offsite backup of incase of fire.


Critical:
102 GB of family photos and home videos
1.5 GB Personal Documents (Email, Resumes, Finances, Poems, Recipes, Design Drawings & Diagrams, Application Development Files, etc)


Non Critical but difficult/impossible to replace:
55GB Music
10 GB Audio Books
30 GB e-Books + PDF product manuals/service guides
200GB TV + Movies
900 GB Hyperspin Emulator / Rom Files


Non Critical / Replaceable
50 GB Software ISO backups
40 GB Device Drivers, Downloaded Install files & Software/Game Patches
2 GB Random MISC stuff (Saved XKCD comic, demotivationals, etc)
1TB Acronis PC backup images for Main PC, Wifes Laptop, Tablet, etc.



I used to host all this as several raid 1's in my main PC, but offloading it to its own separate box has made rebuilding my main rig (happens more often) cheaper, less complex, and easier to fit into micro-atx...


I suppose I could use the DS212j to backup the most critical of stuff from the new NAS, but that still is problematic wanting to keep offsite copies.

 

[alzan]

My RAID5 knowledge is limited to Dell small business/workgroup servers with dedicated RAID controllers and a 50%/30%/20% mix of Seagate/WD/Fujitsu enterprise drives; I replaced equal numbers of each brand. That being said:

RAID5 didn't crap out a lot and when it did it had an 85% success rate of re-synching the array after drive or controller failure. The RAIDs were 3 drive arrays in the ~317GB range. Array rebuild took between 6 - 7 hours, YMMV.

With the NAS you have in mind you could do a 4-5 drive array with 1-2 drive(s) as the hot spare(s); in certain circumstances the array could survive 2 drive failures and keep running (depending on the firmware's user/configuration settings).

You might consider BlueRay discs or WD Blacks in an enclosure for archiving.

 

[Red Squirrel]

I find those 4 bay NAS boxes are quite expensive considering you can't easily expand further, but you can't really build a server/PC to match it at their price, so if 4 drives is going to carry you over for quite some time they do make sense.

Personally I'm a big fan of building my own and using Linux mdadm raid and making the host serve files via NFS, SMB or other protocols. That part is whatever you decide, since it's just a standard Linux system. The nice thing with mdadm is you can easily expand by adding more drives.

You can get drive cages that let you hot swap drives, and they go into cdrom bays. That's how my first setup was. I had two of those in my server for a total of 8 drives. Eventually changed them out for 5-disk ones for 10 drives total.

Once you get into more than 8 drives it gets very expensive though because you now need to buy controllers to get more sata ports.

For your case I would probably suggest trying to find the mobo with the most ports at a decent price, a case that has lot of cdrom bays and then buy one or two of these:

http://www.tigerdirect.ca/applicatio...7958&CatId=285

Supermicro makes 5-bay ones too, but I can't seem to find it but if you search deeper you might find it. They use trays.

As far as raid, raid 5 is fine, I never had issues with it. I find it's the best bang for your buck. Though after like 8 drives I would start considering raid 6 or raid 10. Raid 5 rebuilds do take a while, typically it's an overnight thing, and may even still be going the next morning. But the system is fully usable during that time and the data is available. The beauty of raid is that you never lose access to your data and never have to rebuild your file system from backups. Backups are basically your 2nd resort. I tend to do local backups of files I'm working on such as coding so I can go back a few days, weeks or months. (my method wastes lot of space though but it works, I just rsync it to a month/week/day folder)

With today's very large drives, raid 10 makes more and more sense though. I have one array that consists of 4 3TB drives in raid 10. Even though I'm "losing" half my drives that's still 5.4TB of usable redundant space. I plan to add 2 more drives at some point to get a small performance boost and increased space.

 

[BonzaiDuck]

[Implied quote acknowledgement for Red Squirrel and the OP . . . ]

Went to the doctor this morning for my checkup, then to complete errands. On the way back, stopped in a computer store which was next door to the one which used to be there (but isn't). I was looking to obtain a 2.5"-to-3.5" SSD bracket, instead of waiting another two days for a package of three to arrive. Only guy in there -- a Korean fellow named Dave, and we had a long conversation.

We were discussing "Mainstreamers." Mainstreamers wouldn't think of either buying a server or NAS or building one -- they've all bought into the notion of "cloud storage." Maybe-- they'd buy a NAS device. More likely, they'd obtain a USB3 external HDD and do backups that way.

I can sympathize with the OP if his only server option requires a new computer build. I've been recycling my older computers into server-duty for some time. I'd have to think twice about building a server with all new parts.