DIY SAN question

[sdifox]

I actually mentioned to her a few weeks ago that I wanted to build my new office in the attic over the garage and she told me I was nuts.

I could possibly rework an upstairs closet with venting into the attic though.

draw up some plans then show her... show her a colour swash and have her come up with paint scheme.

 

[dave_the_nerd]

n/m

 

[XavierMace]

You do realize this is for home use and experimenting, right?

It's hard to tell what exactly you are using it for as you haven't really said beyond you're looking for 100TB-200TB of space with very good performance. Most people consider the 30TB-40TB that people like me and sdifox have to be excessive (and they're probably right). You'll also note we're both running ours on enterprise hardware. So it's really hard to picture 200TB just for "experimenting" especially using consumer equipment which eliminates a lot of "experimenting" aspects. Unless it's an experiment in data loss. More importantly if I had enough personal data to fill 100TB-200TB, I'd sure as hell want a backup but given you're wanting to cheap out on the build, I'm assuming that's not in the picture either. In short you've told us you want to build a race car but you don't want to spend much money, you don't want it to be loud, and you don't want to use parts built for racing. The odds of a good outcome are low.

You may need to better define what exactly you're trying to do and what you consider GOOD performance to be. Because you're not getting 200TB of fast storage for cheap regardless how you build. Servers don't HAVE to be loud. Yes, premade ones generally will be. But you can build quiet servers. It's just not cheap. You said you want cheap, yet you're talking $3k-$7k in drives. You don't want loud, but you're talking about cramming 50 2.5" drives in the system. That's going to require a decent amount of cooling. If you're doing these in hotswap bays, that's either talking a small fan on each bay (small means loud) or larger fans with enough CFM to pull air through the largely restricted/blocked front. Which again means loud. Using consumer hot swap bays rather than an enterprise expander backplane means you're going to need multiple RAID controllers. So far, everything about your build is in conflict with itself.

Here's a rough idea of what I'd try to do to check all the boxes other than maybe the undefined "cheap" one.

Supermicro Xeon-D board. Probably X10SDV-4C+-TLN4F-O.
Max out the memory.
SATA DOM boot drive.
NVMe drive for caching.
LSI 9211 for storage pool in the one available PCIe slot.
SC836BE1C-R1K03B Case (or similar), pull the power supplies
Pony up for the 920w SQ (Super Quiet) power supply

Hook the case fans into a PWM controller and see how low you can go and still keep the system sufficiently cooled. The CPU is passively cooled so that eliminates one of the other usually noisy parts on a server. Potentially replace the fans IF you don't run into cooling issues with that many drives, a controller capable of running said drives, and 10GbE NIC's. FYI, the hottest component in my servers is the 10GbE NIC, followed by the RAID controller. That's pretty much going to determine your noise level. Of your listed OS choices, FreeNAS would get my pick. I would also be running two for replication because there's a limited number of ways to back up 100TB+ of data. You mentioned "parity" in one of your posts. I and most storage admins will tell you not to run RAID5/6 (or a ZFS equivalent) on an array of this size. But obviously running a RAID10 (or equivalent) is going to require a higher drive count to get the same amount of space. And the traditional statement of RAID is not a backup.

 

[IndyColtsFan]

It's hard to tell what exactly you are using it for as you haven't really said beyond you're looking for 100TB-200TB of space with very good performance. Most people consider the 30TB-40TB that people like me and sdifox have to be excessive (and they're probably right). You'll also note we're both running ours on enterprise hardware. So it's really hard to picture 200TB just for "experimenting" especially using consumer equipment which eliminates a lot of "experimenting" aspects. Unless it's an experiment in data loss. More importantly if I had enough personal data to fill 100TB-200TB, I'd sure as hell want a backup but given you're wanting to cheap out on the build, I'm assuming that's not in the picture either. In short you've told us you want to build a race car but you don't want to spend much money, you don't want it to be loud, and you don't want to use parts built for racing. The odds of a good outcome are low.

You may need to better define what exactly you're trying to do and what you consider GOOD performance to be. Because you're not getting 200TB of fast storage for cheap regardless how you build. Servers don't HAVE to be loud. Yes, premade ones generally will be. But you can build quiet servers. It's just not cheap. You said you want cheap, yet you're talking $3k-$7k in drives. You don't want loud, but you're talking about cramming 50 2.5" drives in the system. That's going to require a decent amount of cooling. If you're doing these in hotswap bays, that's either talking a small fan on each bay (small means loud) or larger fans with enough CFM to pull air through the largely restricted/blocked front. Which again means loud. Using consumer hot swap bays rather than an enterprise expander backplane means you're going to need multiple RAID controllers. So far, everything about your build is in conflict with itself.

Here's a rough idea of what I'd try to do to check all the boxes other than maybe the undefined "cheap" one.

Supermicro Xeon-D board. Probably X10SDV-4C+-TLN4F-O.
Max out the memory.
SATA DOM boot drive.
NVMe drive for caching.
LSI 9211 for storage pool in the one available PCIe slot.
SC836BE1C-R1K03B Case (or similar), pull the power supplies
Pony up for the 920w SQ (Super Quiet) power supply

Hook the case fans into a PWM controller and see how low you can go and still keep the system sufficiently cooled. The CPU is passively cooled so that eliminates one of the other usually noisy parts on a server. Potentially replace the fans IF you don't run into cooling issues with that many drives, a controller capable of running said drives, and 10GbE NIC's. FYI, the hottest component in my servers is the 10GbE NIC, followed by the RAID controller. That's pretty much going to determine your noise level. Of your listed OS choices, FreeNAS would get my pick. I would also be running two for replication because there's a limited number of ways to back up 100TB+ of data. You mentioned "parity" in one of your posts. I and most storage admins will tell you not to run RAID5/6 (or a ZFS equivalent) on an array of this size. But obviously running a RAID10 (or equivalent) is going to require a higher drive count to get the same amount of space. And the traditional statement of RAID is not a backup.

I understand many of my initial requirements seem at odds with one another and that's one reason I posted here - to see what the options are. I guess at the end of the day, I will spend if I have to because if the solution is easily expandable, I can start low on storage and add more later and divert the funds now to doing a better core build. Maybe I should go with a Qnap or Synology and call it a day, who knows.

Anyway, I'm currently running more than 40 TB of storage on a server I built in 2013 (Supermicro board, dual Xeon 6 core, 192 GB ECC RAM, LSI RAID controller, mirrored SSD volumes, conventional drive arrays, etc). It has been an awesome piece of equipment but my main storage array was not built optimally due to both cost constraints and the need to bring the box online quickly. The plan at the time was to eventually add an additional SSD array(s) and offload some of the disk intensive stuff to those arrays (I run several SQL servers) and the slower volume would be used for file data. Unfortunately SSD prices haven't dropped as far as I hoped and the 2 TB and 4 TB models are just too much.

I have clarified my requirements since beginning this discussion - I was shooting for 100 TB to start with maybe an easy expansion capability of 200 TB. I'm probably going to back down a little from that requirement and give up storage density and go with 10 - 16 TB drives (I won't be building for awhile in all likelihood and 12, and possibly 16, TB drives may be available).

I understand very well that RAID isn't a backup and currently have local and cloud backups running and would continue to do so for critical data. The backup strategy depends on your data set - there is no need to run regular full backups on relatively static data like movie rips and that comprises a large chunk of current and future data. You back it up to the cloud once, maybe have a local archive drive to copy them to, and then back up additions or delete movies which have been removed, and go from there.

 

[dave_the_nerd]

The NAS/SAN build thread drinking game:

backup

::drink::

*car analogy*

::drink twice::

LSI 9211

::drink::

FreeNAS

::tequila shot::

*Parity RAID FUD*

::drink twice::

(caveat*)

And the traditional statement of RAID is not a backup.

::drains glass::

*Parity RAID is fine as long as the individual RAID groups are <=12 spindles and you use dual parity (RAID6 or equivalent) for drives >1TB. That's how most storage vendors do it, anyway. You can aggregate/virtualize groups together into storage pools if you want, otherwise, just manually take care of which raid group you put your stuff. It's still slower than RAID10, but it's usually fine. If you want to get fancy, mix your RAID types and migrate data between them as needed.

 

[dave_the_nerd]

Anyway, I'm currently running more than 40 TB of storage on a server I built in 2013 *snip*

That's a good build. Do another one like it.

 

[sdifox]

If you have gigabit fiber internet, BB backup may be an option for you.

Assuming you don't have a lot of changes daily of course.

And go for 3.5" drives. 2.5" is useless for massive storage. You can still mount 2.5" ssd in 3.5" trays and connector will line up, you just have to be careful on insertion/removal.

As to movie rips, just backup the index. Internet is your backup :awe:

 

[IndyColtsFan]

That's a good build. Do another one like it.

I've seriously thought about building another like it but with bigger drives (main array is eight 3 TB drives in a RAID6 array on an LSI 92xx - I think it was a 9261 IIRC). That was one area I skimped on - I used WD Red NAS drives rather than enterprise drives. But in fairness, the drives have run 24/7 since February of 2013 with no issues. I've got some additional drive space connected to the server via iSCSI on a Synology NAS too.

If you have gigabit fiber internet, BB backup may be an option for you.

Assuming you don't have a lot of changes daily of course.

And go for 3.5" drives. 2.5" is useless for massive storage. You can still mount 2.5" add in 3.5" trays and connector will line up, you just have to be careful on insertion/removal.

As to movie rips, just backup the index. Internet is your backup :awe:

Crashplan unlimited FTW. I also have a local USB "vault" drive which I also copy the rips to when I'm done ripping.

 

[XavierMace]

Your current build is a perfectly acceptable build in my book with the possible exception on the array setup. It would certainly be easy enough to update that to meet your expanded storage requirements. What are you looking to gain over that existing box? I mean if you're looking to build another just for the sake of another, that's fine. But to put something in perspective, your current build has more grunt than most of NetApp's FAS lineup. You're not going to get anything close to that with consumer gear, including the Qnap/Synology route. That's not a slight against those options, that's just not the market they're going for.

*Parity RAID FUD*
::drink twice::

I felt that way until I had a second drive fail during a rebuild. Now I give that stance more consideration. How many large (> 8TB) rebuilds have you done? If you're running ZFS, there's also expansion considerations that make it easier to run mirrored vdevs rather than RAIDZ. I'm not quite sure if you agree or disagree with the rest of the statements, I'm assuming disagree, but I'd love to hear why.

 

[dave_the_nerd]

I felt that way until I had a second drive fail during a rebuild. Now I give that stance more consideration. How many large (> 8TB) rebuilds have you done?

I'm a SAN admin, at a company that makes SANs, so... approximately one grillion?

I have about one drive failure a week. Sometimes two. Drive population is... maybe 500? A lot of them are older. I went about six weeks in 2014 with no failures. It was pretty awesome.

At least if you mean a 8TB RAID group size (individual drives in the machines I run are <=4TB, which means, for instance, 36TB raid groups plus 2 parity drives and 1 hot spare.) It's all automated. Usually, by the time I req. the new drive and swap it out, the RAID restripe is already complete, without me having had to touch anything.

I mentioned double-parity, because that's REALLY important. Losing a second drive while restriping your group is common enough that everybody knows somebody who's had it happen, even if they haven't had it happen themselves. But losing a third (and actually hosing your array) is basically "death by lightning strike" odds. Basically, it doesn't happen, hence my comment about FUD. (And if you have 3 failures that close together, the other 9 in the shelf are basically time-bombs.)

Flipside is that in most mirrored configurations, if you lose one disk, you're fine, two disks, you might be fine, but if it's the wrong second disk, you're equally boned. The only saving grace is that rebuilds are way faster. But newer CPUs process parity soooo fast...

If you're running ZFS, there's also expansion considerations that make it easier to run mirrored vdevs rather than RAIDZ. I'm not quite sure if you agree or disagree with the rest of the statements, I'm assuming disagree, but I'd love to hear why.

I agree with most of what you posted, actually. It's all pretty common-sense stuff. I was mostly just riffing on the common themes - we do have these sorts of threads a lot, and the responses are pretty predictable.

I'm kind of sour on ZFS in general, since... well... you probably don't want to hear those stories.

::dave tears out some of his remaining hair::

Anyway, I don't have to use ZFS RAID management at work, but I'm familiar with ZFS and some of the "expansion considerations" from home use. I'd be curious what configuration of vdevs you prefer, but mirroring all your vdevs (and straight-up doubling your storage costs) strikes me as a hard sell - whether to the spouse or the purchasing department.

Plus, well, mo' spindles, mo' problems.

I guess it depends on the type of expansion you want to do. If you're just going to set something up and leave it alone for a few years, then RAID-Z(1|2|3) is fine. But it's not for people who want to grow an array incrementally (adding one or two drives at a time) and that's a REALLY common ask for home users or people on a budget. To do that, you'd need to be using mdadm or unRAID. (Or Synology OS, or...)

OTOH, if you deal with shelves of disks instead of individual disks and have a corporate budget, ZFS pool and vdev expansion limitations seem pretty easy to work around.

 

[XavierMace]

So in your case your setup is properly setup so it's less of an issue or even a non-issue. Most home users as well as the smaller businesses I deal with, a drive fails. Eventually they notice. Then at some point they pull the dead drive and insert one in it's place. No hot spare, no second parity drive. Obviously that's a failure of the build, not the technology itself. But it's what I generally see. I completely agree a second parity drive eliminates most of that risk. I agree the odds of a 3rd failure aren't great. But keep in mind we're talking a home user so likely consumer drives, all bought at the same time. Still not likely, but with the amount of data OP is talking about and the size of the drives, we're talking a VERY slow rebuild and I just hate slow in general.

The rest of your post kinda ties into the same reply and my response was geared towards a home lab setup, not a proper enterprise setup. Mirrored vDevs are becoming more popular. I personally went from RAIDZ2 to mirrored vDevs a year ago or so. Yes, people generally initially recoil at "now I need twice as many drives". However as you mentioned, home users often like just adding a new drive or two and that's basically a no go running RAIDZ. Running mirrored vDevs, I just add another pair of drives to the pool. They don't even have to be the same size. Unlike Storage Spaces, I don't need to add a cache drive for each vDev.

I completely agree that if you've got a corporate budget, that changes things.

 

[dave_the_nerd]

Running mirrored vDevs, I just add another pair of drives to the pool. They don't even have to be the same size. Unlike Storage Spaces, I don't need to add a cache drive for each vDev.

Hmmm... so you add all your drives to the same zpool? What happens if you want to retire some drives? (I've read that you can't remove a vdev from a pool once it's in there - am I right in thinking you'd have to "replace" the vdev by doing a rebuild/shuffle?)

 

[dave_the_nerd]

So in your case your setup is properly setup so it's less of an issue or even a non-issue. Most home users as well as the smaller businesses I deal with, a drive fails. Eventually they notice. Then at some point they pull the dead drive and insert one in it's place. No hot spare, no second parity drive. Obviously that's a failure of the build, not the technology itself. But it's what I generally see. I completely agree a second parity drive eliminates most of that risk. I agree the odds of a 3rd failure aren't great. But keep in mind we're talking a home user so likely consumer drives, all bought at the same time. Still not likely, but with the amount of data OP is talking about and the size of the drives, we're talking a VERY slow rebuild and I just hate slow in general.

It's true I'm spoiled.

Generally speaking, if I'm offering advice (like in this thread) I'm going to suggest making it look as "enterprise-ey" as possible, within budget constraints. That doesn't mean other approaches won't/don't/can't work though.

 

[[DHT]Osiris]

And don't forget about offsite backup prep for 200TB + churn

 

[IndyColtsFan]

And don't forget about offsite backup prep for 200TB + churn

I have Crashplan Unlimited, so no issues there. . Plus as I mentioned earlier, you have to intelligently back your data up. No need to size for nightly full backups when a big chunk is static data that rarely changes, for example.

 

[[DHT]Osiris]

I have Crashplan Unlimited, so no issues there. . Plus as I mentioned earlier, you have to intelligently back your data up. No need to size for nightly full backups when a big chunk is static data that rarely changes, for example.

Well, at any rate, you can probably mush together some kind of raid6, raid60, small subset of raid6's hammered into a larger pool, whatever, of 10TB platters, but TBH I'd just buy a dell server if you really need it. You don't 'play' or 'experiment' with a 200TB NAS. You buy that knowing what you're getting, and then use it for a specific purpose. I'd wager you don't actually have 100-200TB of useful data to actually keep a hold of (BD RIPs downloaded from TPB don't count), so I'd suggest you temper your expectations. If you want a reliable NAS that's not going to crap itself unexpectedly and take your billions of bits with it, you need to spend for it. If you don't, pare down your data or accept the potential for loss, and get yourself a handful of disks of whatever type and raid5 them in some consumer eggcrate solution.

 

[IndyColtsFan]

Well, at any rate, you can probably mush together some kind of raid6, raid60, small subset of raid6's hammered into a larger pool, whatever, of 10TB platters, but TBH I'd just buy a dell server if you really need it. You don't 'play' or 'experiment' with a 200TB NAS. You buy that knowing what you're getting, and then use it for a specific purpose. I'd wager you don't actually have 100-200TB of useful data to actually keep a hold of (BD RIPs downloaded from TPB don't count), so I'd suggest you temper your expectations. If you want a reliable NAS that's not going to crap itself unexpectedly and take your billions of bits with it, you need to spend for it. If you don't, pare down your data or accept the potential for loss, and get yourself a handful of disks of whatever type and raid5 them in some consumer eggcrate solution.

You're correct - I don't have that much important data. Sometimes, I like buying hardware and just experimenting and pushing the envelope. All critical data I have is backed up in 2-3 places currently and would probably only amount to a few hundred GB at most.

 

[XavierMace]

Hmmm... so you add all your drives to the same zpool? What happens if you want to retire some drives? (I've read that you can't remove a vdev from a pool once it's in there - am I right in thinking you'd have to "replace" the vdev by doing a rebuild/shuffle?)

Correct. You can't remove/replace a vdev, but you can replace the drives in the vdev. Since it's just a single drive mirror, the rebuild time is minimal as is the impact to the rest of the pool since it's just a mirror and not dealing with the parity calculations. This article goes over the advantages in more detail: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

 

[[DHT]Osiris]

You're correct - I don't have that much important data. Sometimes, I like buying hardware and just experimenting and pushing the envelope. All critical data I have is backed up in 2-3 places currently and would probably only amount to a few hundred GB at most.

Well, for what it's worth, we just got through buying a Dell 100TB backup server (12 bay, 10TB disks, raid5+1) for under $18k, original price was $35k however. That's 'enterprisey' though and includes stuff like idrac, warranty, etc. If you were to plus that up to their larger model, you could probably get away with a full 200-220TB (unsure on the density) like, maybe 50k? or so. Could maybe find an older model without drives and just buy the 10TB's, but those suckers aren't cheap unless you know a guy who knows a guy. 22 of them (basically the cheapest you could get away with) on newegg is $8360 for drives alone (21 raid5'd +1 HS, and I'd *never* recommend that configuration, it's just the cheapest).

Cheaper than that and you're going to be getting into unreliable territory. Like 'egg-crate with a JBOD of 1TB rust hooked into a half dozen reflashed PCIe2 raid controllers harvested out of old Dell 1950's duct taped to particle board' unreliable.

 

[sdifox]

Well, for what it's worth, we just got through buying a Dell 100TB backup server (12 bay, 10TB disks, raid5+1) for under $18k, original price was $35k however. That's 'enterprisey' though and includes stuff like idrac, warranty, etc. If you were to plus that up to their larger model, you could probably get away with a full 200-220TB (unsure on the density) like, maybe 50k? or so. Could maybe find an older model without drives and just buy the 10TB's, but those suckers aren't cheap unless you know a guy who knows a guy. 22 of them (basically the cheapest you could get away with) on newegg is $8360 for drives alone (21 raid5'd +1 HS, and I'd *never* recommend that configuration, it's just the cheapest).

Cheaper than that and you're going to be getting into unreliable territory. Like 'egg-crate with a JBOD of 1TB rust hooked into a half dozen reflashed PCIe2 raid controllers harvested out of old Dell 1950's duct taped to particle board' unreliable.

err, BB Pod cluster cost less than that.

 

[[DHT]Osiris]

err, BB Pod cluster cost less than that.

So, I googled, no results.

What's a BB Pod cluster?

 

[sdifox]

So, I googled, no results.

What's a BB Pod cluster?

https://www.backblaze.com/blog/open-source-data-storage-server/

get three of them, triplicate files.

 

[[DHT]Osiris]

https://www.backblaze.com/blog/open-source-data-storage-server/

get three of them, triplicate files.

Gotcha. Good price for the density.

 

[IndyColtsFan]

err, BB Pod cluster cost less than that.

This.

Guys, you are losing site of the scope of this project. I hate to beat a dead horse, but this is a HOME/FUN project. Telling someone to spend $20k-$50k on enterprise hardware for increased reliability for home use is insane.

 

[sdifox]

Gotcha. Good price for the density.

for archival purpose their approach is probably the cheapest one. of course your average home lab probably don't need/want that.