DIY SAN question

[IndyColtsFan]

Mods, please move this thread if this isn't the appropriate location.
***************************************************************************

Hello,

I'm in the beginning stages of planning a SAN for my home and wanted a few opinions. My goal would be to build something that could be expanded into the 100-200 TB range with very good performance at a relatively decent price. The uses would be to store media for streaming, VMs, and general data storage. I've toyed with the idea of booting PCs from the SAN too (possibly via an SSD shelf in the SAN) but I haven't thought about that in detail and that isn't a big deal if it doesn't happen.

Here are a few questions which I'm looking for feedback on:

1. What software do you recommend? Choices are Windows Storage Server, FreeNAS, or Starwind (they still make the free SAN version, right?)
2. I have an old Core 2 Duo E8400 in an x48 Asus Rampage Formula board with dual 1 GB NICs. I'd plan on possibly going with 10 Gb links to my server boxes via PCI NICs in the future. Is that CPU and board enough for this kind of build? If not, please provide recommendations.
3. So far, the best storage density appears to be the Icy Dock hot swap bays which install eight 7 mm, 2.5" drives into a 5.25" bay. I'd probably use multiple of these in "shelf" configuration and at least one of the shelfs might be fully populated with SSDs for applications needing super speed. For the other shelves, I'm looking for 2.5" 7 mm drives which are fairly cheap but decent. I *think* 2 TB is the max capacity on a 7mm 2.5" drive at this stage. Any recommendations? I considered Firecuda but I think that's overkill and can probably get something a little cheaper.

Thanks

 

[dave_the_nerd]

The more clients, the more CPU you'll want. If you're doing file and not just block, you'll want more CPU oomph too. An E8400 will basically boot, load an OS, and then fall over.

Most OSes will use RAM for cacheing purposes automatically, so MOAR RAMZ is going to make you happier, even with 1GbE. Like, 16GB+.

Your reference to drive bay adapters caused me to break out in a rash. 100TB of 2.5" drives would mean a minimum of 50 drives active (plus standby, parity drives for RAID, etc.) With consumer hardware, you're basically asking for a bad cable to hose your stuff. And at $50 minimum per drive, plus the adapters, you're not talking cheap.

Get on eBay and buy some old server gear. Something like this for a controller, and as many of these as you need to get the storage you want. (Although if you haven't got a LOT of clients, I'd actually go with a smaller number of higher-capacity 3.5" drives, since you'll need a lot fewer of them, which has benefits.)

Since it sounds like you just want an iSCSI box, you can use almost anything OS-wise. You could even go with an off-lease low-end commercial SAN like this. (Although I'm not sure how licensing and support would work.)

Otherwise, any OS you listed should work. I've used FreeNAS, but it's basically a pretty web interface for stuff you should be able to do from a CLI anyway. (Most of its whiz-bang features are stuff it inherits from the underlying FreeBSD, and which you can get just as easily on most Linux distros too.)

 

[IndyColtsFan]

Yeah, it makes me sweat too thinking about 50+ drives but I was looking for storage density. I don't have a ton of space so was hoping to do this in a form factor the size of a typical ATX full tower case. I guess I could go 12-bay with 10 TB or 12 TB drives and that probably would meet by needs even subtracting for parity, etc. I think I remember hearing that 16 TB drives are going to be released in the next few months too, so that might be another option.

Synology has a 12 bay model which can be expanded with another 12 bays but IIRC, it doesn't have a slot for 10 GbE. Otherwise I'd go that route and be done with it. I could look at a few other vendors such as Qnap I suppose.

The server gear is too loud and hot - that's my biggest concern there. I used to run several Proliants in the house and don't want to go that route again. Anyway, I also have a spare Ryzen 1500X and can add more RAM to that and use that as the basis for the system.

 

[dave_the_nerd]

Ah, I see.

Well, a full tower ATX case with the right brackets would support a lot of 3.5" HDDs.

https://pcpartpicker.com/products/case/#xcx=0&G=9,12&sort=price&page=1

Plus some combination of these:

http://www.istarusa.com/en/istarusa/product_speclist.php?series=HDDCage&sub=Trayless&Factor=3.5

At least if you want to be fancy. (If you can start with, say, 6x 12TB drives in the internal bays, you can get your system started and add additional cages/HDDs as your needs grow.)

 

[VirtualLarry]

I'm using a Source 210 case (I think that's NZXT). That has 8x 3.5" internally, and 3x 5.25", so you can fit some drive cages of various sizes into those as well.

I'm using an A85X mobo, with a ton of PCI-E slots, 8x SATA6G ports, and 4x8GB DDR3. Oh, and onboard video, so it doesn't use a PCI-E slot for that.

 

[Exterous]

Yeah, it makes me sweat too thinking about 50+ drives but I was looking for storage density. I don't have a ton of space so was hoping to do this in a form factor the size of a typical ATX full tower case. I guess I could go 12-bay with 10 TB or 12 TB drives and that probably would meet by needs even subtracting for parity, etc. I think I remember hearing that 16 TB drives are going to be released in the next few months too, so that might be another option.

Synology has a 12 bay model which can be expanded with another 12 bays but IIRC, it doesn't have a slot for 10 GbE. Otherwise I'd go that route and be done with it. I could look at a few other vendors such as Qnap I suppose.

The server gear is too loud and hot - that's my biggest concern there. I used to run several Proliants in the house and don't want to go that route again. Anyway, I also have a spare Ryzen 1500X and can add more RAM to that and use that as the basis for the system.

Both Synology and Qnap have models that expand via expansion racks (just put them on a shelf) and will go through 10g links but not sure what your price range is. Depending on your use Synologies have a problem with space reclaim on iSCSI volumes - basically you have to delete the volume or disconnect the iSCSI connection and let it sit forever. We've seen some QA issues with both in terms of software updates so we typically wait a few months to apply those unless there is an overriding critical fix thats needed. That said we've been generally happy with both. We have an environment running VMs off a Synology instead of an EMC because the EMC was more problematic

If you're looking for a pre-built solution maybe checkout http://www.45drives.com/products/ depending on budget. We're seriously looking at them for use with a few of our clients although we don't have any first hand experience other than a few conference calls.

For home built I'd say FreeNAS or similar. Windows Storage Server isn't bad but tends to come with a lot of extra annoyances like more frequent patching. That said its JBOD implementation is pretty nice and generally hassle free although some powershell experience would be good for when the GUI doesn't do what you tell it to do or it says it did.

I'd hold off on an SSD shelf until you see how the spindle drives work. We tend to run most VMs we setup on spindle unless we run into high IOPS applications (typically database driven like large sharepoint or exchange environments)

Can you stick it somewhere else (basement?) and run cables to it to avoid the noise issue of servery stuff? I haven't done a DIY of that density but I feel like there would be a lot of heat generation to deal with esp if you're wanting to run compute on it

 

[IndyColtsFan]

Thsnks guys. I think I'm going to run some tests on Hyper V with a few drives simulating what I need before committing to a larger build at this stage. Both of my current servers have the Icy Dock hot swap bays for 3.5" storage but I was hoping to get away from DAS at some point in the future and have a large, central storage pool from which I could allocate storage to different systems or VMs. One of the servers is a little older and I have thought about upgrading it and using it as the SAN, hence some of my earlier questions related to CPU and RAM.

 

[XavierMace]

A storage system of that size on entirely consumer hardware would give me nightmares.

 

[IndyColtsFan]

A storage system of that size on entirely consumer hardware would give me nightmares.

It won't have enterprise workloads. I'm not willing to pay enterprise prices or tolerate enterprise heat/noise levels at this particular location in my house.

I think I'm going to rip apart an old server I built, upgrade it, had an additional hot swap cage, and not be quite as ambitious as my original plan. 9-12 of the 10-12 TB drives would work. May also consider going the Synology or Qnap route with a model that support 10 GbE, though this server refit would be much cheaper.

 

[sdifox]

Are you building NAS or SAN?

 

[XavierMace]

It won't have enterprise workloads. I'm not willing to pay enterprise prices or tolerate enterprise heat/noise levels at this particular location in my house.

Make sure you post updates then the first time you have issues with your 100TB consumer NAS.

 

[IndyColtsFan]

Make sure you post updates then the first time you have issues with your 100TB consumer NAS.

You do realize this is for home use and experimenting, right?

 

[IndyColtsFan]

Are you building NAS or SAN?

Started off as an experimental SAN project but I think I'm reigning it in a bit and may move towards more of a NAS solution. I'm still debating a bit on how I should approach it. The more immediate need is for 8 to 10 TB of mirrored storage but I was trying to think longer term and perhaps build a larger storage solution for future and many of my current needs.

 

[sdifox]

I thought you run a server anyway, why not just add more das to it?

Or go hog and build three Backblaze pod :awe:

https://www.backblaze.com/blog/open-source-data-storage-server/

 

[IndyColtsFan]

I thought you run a server anyway, why not just add more das to it?

Or go hog and build three Backblaze pod :awe:

https://www.backblaze.com/blog/open-source-data-storage-server/

I do have a huge server, but to add more space to it, I'd probably have to replace the drives as the hot swap cages are fully populated. That's a ton of work and probably not worth the effort at this stage because I want to eventually move to centralized storage and carve chunks off of it to allocate to various machines. Although as I think through it a little more, I think I might have an idea to make it more feasible.

Trust me, I looked at the Backblaze pods.

 

[sdifox]

I do have a huge server, but to add more space to it, I'd probably have to replace the drives as the hot swap cages are fully populated. That's a ton of work and probably not worth the effort at this stage because I want to eventually move to centralized storage and carve chunks off of it to allocate to various machines. Although as I think through it a little more, I think I might have an idea to make it more feasible.

Trust me, I looked at the Backblaze pods.

no, there are DAS enclosures, you just hookup to your server directly with either fiber or 10gbt nic.

just a random google example, a bit old but you get the idea.

https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/

probably get better answers in storage subforum though.

 

[IndyColtsFan]

no, there are DAS enclosures, you just hookup to your server directly with either fiber or 10gbt nic.

just a random google example, a bit old but you get the idea.

https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/

I had looked at that option before and considered it, but was concerned with case size, heat output/noise, etc. Maybe I could revisit your suggestion and look for a smaller case or I could go with the old second server case I have, add backplane, and implement your suggestion.

 

[sdifox]

I had looked at that option before and considered it, but was concerned with case size, heat output/noise, etc. Maybe I could revisit your suggestion and look for a smaller case or I could go with the old second server case I have, add backplane, and implement your suggestion.

Garage, leave it in your garage and fiber or 10GBT to your server. Of course there are denser boxes so you are lowering $/GB.

4GB FC gear are aplenty on ebay and dirt cheap. load balance two adapters and you are golden


http://www.ebay.com/itm/Dell-QLogic...444441&hash=item2a79fa9c9e:g:J9EAAOSw0hlZMC-n

 

[IndyColtsFan]

Garage, leave it in your garage and fiber or 10GBT to your server. Of course there are denser boxes so you are lowering $/GB.

4GB FC gear are aplenty on ebay and dirt cheap. load balance two adapters and you are golden


http://www.ebay.com/itm/Dell-QLogic...444441&hash=item2a79fa9c9e:g:J9EAAOSw0hlZMC-n

Garage is too hot, down one floor, and on the far side of the house. It has its own attic too, so running cable back isn't trivial.

 

[sdifox]

Garage is too hot, down one floor, and on the far side of the house. It has its own attic too, so running cable back isn't trivial.

I can't help you on placement

no basement?

 

[IndyColtsFan]

I can't help you on placement

no basement?

I do have a crawl space and have considered it......

 

[sdifox]

I do have a crawl space and have considered it......

OR add ac unit to garage "rack space" One of the Daikin split units with switching power supply

http://www.daikin.com/products/ac/lineup/split_multi_split/index.html

 

[IndyColtsFan]

OR add ac unit to garage "rack space" One of the Daikin switched

Thought of that, but not sure the WAF would allow it to happen.

 

[sdifox]

Thought of that, but not sure the WAF would allow it to happen.

huh? what will she see? Thought you were going with the attic portion of garage?

just enclose it like a small storage space with the "wall" being pulley-ed.

 

[IndyColtsFan]

huh? what will she see? Thought you were going with the attic portion of garage?

just enclose it like a small storage space with the "wall" being pulley-ed.

I actually mentioned to her a few weeks ago that I wanted to build my new office in the attic over the garage and she told me I was nuts.

I could possibly rework an upstairs closet with venting into the attic though.