Building a Media Server/NAS

[Fallen Kell]

I am finally biting the bullet and building a media server. I plan on running FreeNAS with a Plex server for streaming/transcoding for devices that can not handle the original formats. My current hardware plans are as follows:

Case: Norco RPC-4224 with 120mm fan conversion (with a custom internal bracket for mount 2x2.5" hard drives)
Power Supply: Silverstone Strider Gold S Series 850W (70amp 12V rail)
Motherboard: Supermicro X9SRL-F LGA-2011 motherboard (has ipmi/kvm over ethernet support)
CPU: Intel Xeon E5-1620v2 Quad core 3.7GHz (3.9 turbo) with hyperthreading
RAM: 4x8GB 1600mhz ECC (not sure exact make/model)
SAS Controllers: 3x IBM M1015 SAS2 8xPCI-e 2.0(flashed with LSI IT firmware)
OS Disks: 2x128GB Samsung 840 PRO (mirrored)
Internal Recovery USB: 128GB USB 2.0 stick

I will probably start with either 12 3TB (or 6 4TB) WD Red drives. The case gives me the ability to expand to 24 drives (and I would do it in groups of 6 drives). These would be configured in 6 disk raidz2 vdevs (think of it as RAID 6) for 24TB usable storage (or 16TB with the 6x 4TB drives). Currently I plan to not use SAS expanders and get the full raw performance from the controllers, but if I used expanders, I could support up to 96 hard drives and use external bridges and expansion chassis to host the drives.

With the ipmi/kvm support over network I can manage this system as though I was physically on it from anywhere in my local network without the need of sitting near it (so it can go into a rack in the basement). I plan on teaming the 2x1GB ethernet connections to get more bandwidth to support multiple streams and/or recordings from my HTPC/DVRs.

The only real issues I see is that I do not know what CPU heatsinks will be supported on that board. It uses a "narrow LGA-2011 socket", which isn't the same as the standard socket (well it is from the CPU's point of view, but the mounting holes for heatsinks are not the same). Anyone have any experience with this board or other boards which use "narrow LGA-2011 sockets"?

Anyone else have suggestions? I thought about the E3-1220v3 CPU, but it has very limited PCI-e lanes as well as a 32GB memory limit, which is something I think I would easily exceed in future expansions (ZFS uses RAM for cache as well as other features, some of which I would turn off like data de-duplication, but it can easily exceed using 32GB as I expand the ZFS pool and want to maintain performance).

 

[dave_the_nerd]

All that for FreeNAS? Your "OS" disk can be a 4GB thumb drive.

Hell, the "five nines" NAS servers at work boot off of an internal USB key. It's not a liability wrt reliability. The SAN controller uses compact flash.

What else are you intending the server to do? because that seems like a lot of hardware to throw at a Plex server. (Hell, if it's just media files that you have backed up elsewhere, I wouldn't even bother with ECC memory.)

You must really, really, really like your movies.

 

[Fallen Kell]

It is also going to be my NAS, so not just media/movies. That is why the ECC is needed since ZFS caches to RAM for the writes. It will have anything important that I need. It will also be the backend storage for my unix/linux/bsd based systems (I have an older Solaris workstation that I have in my network's DMZ which is setup so I can securely transfer files to my home. I also used it to create some custom security software which does dynamic firewall rule changes based on live connection attempts which has been interesting watching it over time. I also have a couple linux systems and a macbook pro).

I have been debating the OS disks, and might just go with 2 WD laptop drives. Boot time isn't a factor as much for this (just that I havn't built a system in several years that didn't use SSD for the OS drive).

 

[smitbret]

It is also going to be my NAS, so not just media/movies. That is why the ECC is needed since ZFS caches to RAM for the writes. It will have anything important that I need. It will also be the backend storage for my unix/linux/bsd based systems (I have an older Solaris workstation that I have in my network's DMZ which is setup so I can securely transfer files to my home. I also used it to create some custom security software which does dynamic firewall rule changes based on live connection attempts which has been interesting watching it over time. I also have a couple linux systems and a macbook pro).

I have been debating the OS disks, and might just go with 2 WD laptop drives. Boot time isn't a factor as much for this (just that I havn't built a system in several years that didn't use SSD for the OS drive).

Agreed about the OS drives. Once the thing is booted up, it really will be something you don't notice. I've used FREENAS booting off of a flash drive and it isn't hampered by that.

Unless you plan on physically accessing the NAS and using it as a workstation on a regular basis, SSDs are pointless, although I certainly see the reasoning behind a RAID 1 setup for the OS.

 

[ethebubbeth]

I just redid my NAS inside a Norco RPC-4224

It has the following:
CPU: Opteron 6320 (8 core/4 module Abu Dhabi, 2.5ghz, 2.8ghz turbo)
Memory: 64gb ECC Samsung 1.35V CL9 (4x M393B2G70QH0-YK0)
Board: Supermicro H8SGL-F (IPMI equipped version)
PSU: Seasonic X-650 Gold
CPU Cooler: Noctua NH-U9DO (2x 92mm noctua fans)
Fans: 3x 120mm Panaflo L1A (internal fan plane)
2x 80mm Panaflo L1A (exhaust)
1x 92mm Panaflow L1A (pointed at the HBA)
HBA: Adaptec 52445 (Using it as HBA, all RAID features disabled)
Drives: 12x 3TB Hitachi HDS5C3030ALA630
6x 3TB Western Digital Red WD30EFRX
OS Drive: 16gb USB Flash Drive

I ending up going with the opteron since it was the cheapest platform for having access to 64gb+ of registered ECC memory. 16gb RDIMM are definitely the cheapest ECC memory per gigabyte at the moment. I am glad that I went with 64gb, and might consider expanding further in the future. I however am not using Plex so I don't really need to consider CPU horsepower all that much.

In terms of storage topology, the drives are in 6 drive RAIDZ2 vdevs in a single zpool (thank you AT storage forum for pushing me in the single pool direction). I get about 10.6 TB usable storage per vdev in this configuration. I am planning to get 6 more 3TB reds to build my last vdev and reach 24 drives.

I wouldn't worry about using a flash drive for the OS. My SuperMicro motherboard even has internal USB ports (I assume for USB dongles for workstation software) but they work great for keeping the flash drive safe from accidentally being unplugged. You could do the same with an internal USB header adapter if your board doesn't support it.

Once freenas loads, the root filesystem is mounted read only so you don't need to worry about write performance. Configuration changes are written to a /conf directory and moved to the root filesystem at next boot.

Let me know if you have any further questions regarding my build and experiences, since it seems like our systems are designed are to meet similar goals (outside of running Plex).