Storage Server replacement

[dtgoodwin]

I currently have my own custom built server comprised of a SuperMicro 24-bay hotswap 4U case, a SuperMicro X9SCM-F motherboard with an i3-2500T processor and 8 GB of Kingston ECC RAM in a 4x2GB configuration. I am currently using 12 drives 6 using the Intel ports off of the C206 chipset and 6 of 8 on a SuperMicro AOC-SASLP-MV8 (Marvell 6480) using two breakout cables. My drives are a OCZ Vertex2 60GB SSD boot (I've been lucky), 3 WD 3 TB RED drives, 3 WD 2TB Green drives, 1 WD 2TB Black, 1 WD 1TB AV, 1 WD 1TB 2.5", a WD 500 GB for client backups, and a WD 160 GB for OS backups.

I've been running Windows Home Server 2011 with Stablebit DrivePool pooling software. I've been happy with the performance and relative stability of this system. The ability to mix and match drives has been really nice as larger drives have consistently become cheaper, and also the ability to just add another drive when I needed more space. I'm using it's folder replication feature to make sure my critical files are on more than one drive. My very critical folders are stored on 3 different drives. I'm also backing up those same files to two external drives on a weekly basis. I had been backing up offsite until recently when TeamViewer suddenly became unstable for me and would lock up the remote system. I'm also running AirVideo HD Video server, Ubiquity Unify controller, iTunes, and OpenDNS dynamic DNS updater.

Recently I've been having filesystem errors on a frequent basis. Check disk corrects them and they are almost always security descriptor errors. I have the Stablebit Scanner software installed which monthly runs a surface scan on them and also checks the file system health. It finds no issues with drives during the scan, and SMART shows no errors at all on any drive. So far this week, I have had 4 errors across 3 different drives. Interestingly enough, all of them have been on the SuperMicro (Marvell) controller.

I'm very concerned about the integrity of my current setup. If cost were no object, I would probably switch to FreeNAS as I really like the protection ZFS offers. I know my hardware is supported with the possible exception of the drive controller. I currently have 17.3 GB of total usable space, with 13 GB used, and about 5 GB of that being taken up by multiple copies. This server stores everything digital I have. Photos, video (iso's of DVDs and blu-rays I own) as well as compressed versions, software installs, utilities, documents, music, etc.

To run FreeNAS, I know I would need to replace my memory and have at least 16 GB. I would also face an issue with my drive arrangement. I figure that I would need to purchase at least 2 more 3TB REDs to have the same effective space with protection since I cannot mix and match drives. I would probably want to run RAIDZ2. My challenge is that I hate to lose the flexibility I currently have in using whatever drive I want, and then the cost to change the hardware and then how to manage actually moving the data while reusing some of the same hardware. If I turned off file duplication, I might be able to remove the 3 3TB drives I currently have and run everything off the other drives. After the data is migrated, I could pull all the drives and put them in my PC one at a time to copy their contents to the new setup. There is obviously risk as I would only have one copy of the content, and powering down/moving drives that have been running 24/7 with rare exception, and as old as 6 years of on time.

What would you do? I know my data being so critical means that I should take whatever measures are necessary. It would be an extreme financial stretch to do what I think needs to be done. Is there another option to efficiently use what I have, but give me more protection against filesystem errors, silent corruption, and hardware failure?

I should add that I realize I won't be able to run things like the AirVideo server, Ubiquity controller. While nice, it's not a requirement to run on this box. I can run them on my desktop which is also on 24/7.

 

[dave_the_nerd]

The first thing you should do is back it up. RAID is not a backup. Crashplan is $60 a year for unlimited storage. Do it. Turn on QoS on your router and let it go - it'll take a few months, but that's okay.

The second thing you should do is stop worrying - the hardware you have will work fine with FreeNAS. (You don't need new RAM, you don't need more RAM.) Your SATA controller is also probably A-Okay.

Otherwise, your instincts are dead on, imo - ditch the old, small HDDs, and either:

1) Have two RAIDZ/RAIDZ2 pools, one of 2TB HDDs, one of 3TB HDDs. (You'd probably need to buy a couple more of each size, so around $350; ~$100 each for 3TB drives, and ~$75 Each for 2TB drives.)

2) Buy MOAR 3TB HDDs. You can probably find them on sale for <$100 each. Avoid WD Greens. Sell the 2TB drives in FS/FT or on eBay to defray the higher cost of 3TB HDDs. (Probably could get $50 each for them?)

Assuming RAIDZ2, you'd be looking at either 5x 3TB = 9TB, plus 6x 4TB = 8TB, (Buy two of each, 17TB usable, $350) or 7x 3TB = 15TB usable (buy 4 drives, $400, subtract back whatever you can sell your existing 2TBs for used, say, maybe $200.)

 

[Essence_of_War]

I currently have 17.3 GB of total usable space, with 13 GB used, and about 5 GB of that being taken up by multiple copies. This server stores everything digital I have. Photos, video (iso's of DVDs and blu-rays I own) as well as compressed versions, software installs, utilities, documents, music, etc.

Just to be super pedantic, these are all "TB" rather than "GB", correct?

I know I would need to replace my memory and have at least 16 GB.

Depending on how much performance you were hoping to see, I think you'd probably be OK with 8 GB.

I would also face an issue with my drive arrangement. I figure that I would need to purchase at least 2 more 3TB REDs to have the same effective space with protection since I cannot mix and match drives. I would probably want to run RAIDZ2

That may not quite be true either. Remember, ZFS introduces redundancy and capacity for the pool at the level of the vdevs. You could absolutely have 1 vdev for your 3x 3TB Reds in RZ1, another vdev of your 3x 2TBs in raidz1, and your 2x 1 TB (3.5'' and 2.5'') in a mirror. Each of those vdevs would have a single drive of redundancy, and your total usable capacity would be 6TB + 4TB + 1TB = 11 TB.

It may not exactly be best practice for an enterprise server, but it would probably be fine for your home use.

 

[dave_the_nerd]

I should add that I realize I won't be able to run things like the AirVideo server, Ubiquity controller. While nice, it's not a requirement to run on this box. I can run them on my desktop which is also on 24/7.

If you're accustomed to having a general purpose server, you might want to pass on FreeNAS and just install a Linux distro like Ubuntu. It'll let you use ZFS for storage pools, but there are AirVideo and Ubiquiti controllers available as well.

Also, IMO, the Crashplan engine works way better on Ubuntu than it does in a FreeNAS Jail environment.

 

[dtgoodwin]

Just to be super pedantic, these are all "TB" rather than "GB", correct?


Depending on how much performance you were hoping to see, I think you'd probably be OK with 8 GB.


That may not quite be true either. Remember, ZFS introduces redundancy and capacity for the pool at the level of the vdevs. You could absolutely have 1 vdev for your 3x 3TB Reds in RZ1, another vdev of your 3x 2TBs in raidz1, and your 2x 1 TB (3.5'' and 2.5'') in a mirror. Each of those vdevs would have a single drive of redundancy, and your total usable capacity would be 6TB + 4TB + 1TB = 11 TB.

Yes, TB, not GB. The thing holding me back on creating multiple vdevs is exactly what appealed to me with WHS and pooled drives. I didn't have to worry about what folder was growing and making sure that I always had space in the right folders. I just had...space.

I do have a backup, and certainly recognize that RAID is NOT a replacement for backup. I unfortunately, don't have space to backup everything, so I backup what cannot be replaced, and use duplication/RAID to keep everything working and to minimize having to re-rip disks/copy data if I lose a drive.

If I did this, I would be replacing the Greens with 3TB Reds so all drives would be matching. My 2 TB drives are really old and I don't trust them. They have had their idle timer set very high, so I don't have the head parking issue that most die an early death from in my use case.

I'm going to try to boot my server with USB and see if my controller is recognized. That will make a difference as I think I would like a 6-disk RAIDZ2 array, and still have nearly 12 GB of usable space. I've heard that using an SSD as an L2ARC drive isn't always recommended. I have a spare drive that would cost nothing. It's not super fast, but would easily saturate my teamed dual LAN. Would having it offset any issues with keeping my ram at 8 GB? I'm not super concerned about performance, right now my worse case performance is generally the speed of the slowest drive's minimum transfer.

 

[Cerb]

You could keep your memory. RAM recommendations tend to be based on business use cases with many users, and it scales by utilization.

Now then, being critical and all, how is the data backed up? And, how much of the data is actually critical (FI, ripped discs are not, as long as the discs are still good)?

With the problems all being on drives on that Marvell controller, the first I would do is hunt down a used LSI HBA on Amazon or eBay. Marvell cards are usually fine, but that one could be a dud. It happens. It shouldn't cost any more than another HDD, and I can't think of a better way, with no other errors showing up, to determine whether it is the controller or HDD(s). The Green drives or AV drive might not be happy, as well, but the controller could be it, as well. If it's the drives, rather than the controller, that changes things quite a bit.

While 3TB drives means more cost, 3TB is a good capacity/$ size, right now, so it could be worse, if that's what it comes down to. If you do that, my thinking would be to implement the largest RAIDZ2 you could afford to, and then plan to increase it later with an additional array of newer drives in the future.

 

[dtgoodwin]

I had been backing up to 2 locations. First is a local backup to external drives. My secondary backup was to another server at my dad's house (I also kept a backup of his critical files). Things went well with hamachi/logmein until they became a for pay service. I switched to TeamViewer, but even hosting the "server" component locally, I rarely can connect. When I do, it disconnects after about 15 minutes. I haven't had the time to find a suitable replacement that "just works".

My critical data does fit within 2 TB. Nice to have would be 4, with the rest being movies.

I did find out that my controller is not seen my FreeNAS 9.3, so I would need to go with something newer if I expanded beyond the 6 ports on the motherboard. If I didn't use my SSD as an L2ARC drive, then I would have exactly enough ports for the initial array.

I think I may have a second SuperMicro controller of the same type. I didn't see any system or app log errors that would make me think it had caused the issue, but of course that doesn't mean it's bad. If I can find it, I'll swap it in and see if it makes any difference. Long term I think I still want to end up with my storage on a more robust file system than NTFS.

Thanks for everyone's input.

 

[mfenn]

The thing holding me back on creating multiple vdevs is exactly what appealed to me with WHS and pooled drives. I didn't have to worry about what folder was growing and making sure that I always had space in the right folders. I just had...space.

I think that you don't quite understand how vdevs work. That's OK, because it seems really complex until you see it in action.

In ZFS, you have 3 main building blocks, one layered on top of another.

Layer 1 - vdev: the basic actual RAID building blocks for ZFS, and is where you choose RAIDZ1, RAIDZ2, mirror, etc. Vdevs should be matched capacity drivds, and can be upgraded in capacity by swapping out the drives one at a time.

Layer 2 - zpool: as the name implies, a pool of vdevs presented as one logical data space. You can add (but not remove) vdevs as needed.

Layer 3 - filesystem: actual mount points where you can put files and share them out. They transparently draw on capacity from the zpool. While you can apply quotas to them for space management purposes (useful in a multi-user environment), the quotas are just numbers which can be changed at any time.

That may not quite be true either. Remember, ZFS introduces redundancy and capacity for the pool at the level of the vdevs. You could absolutely have 1 vdev for your 3x 3TB Reds in RZ1, another vdev of your 3x 2TBs in raidz1, and your 2x 1 TB (3.5'' and 2.5'') in a mirror. Each of those vdevs would have a single drive of redundancy, and your total usable capacity would be 6TB + 4TB + 1TB = 11 TB.

It may not exactly be best practice for an enterprise server, but it would probably be fine for your home use.

Since data is striped across vdevs, the one downside to this arrangement is that it'll be as fast as the slowest drive (probably also the true in StableBit though). Given that the OP highly values modularity, I would probably actually go with the following vdev setup:

mirror-1 2x3TB
mirror-2 1x3TB + 1x2TB
mirror-3 2x2TB
mirror-4 2x1TB

That yields 3+2+2+1 = 8TB. That's just enough for the OP's data set (13TB used - 5TB overhead). Capacity upgrades can be done in 2 drive chunks, though the first upgrade could be to replace the 2TB drive in mirror-2 with a 3TB one.

 

[mfenn]

Recently I've been having filesystem errors on a frequent basis. Check disk corrects them and they are almost always security descriptor errors

This makes me think that you don't actually have a hardware problem, but are running into some sort of subtle bug in DrivePool since DrivePool operates at a filesystem level. It's not too surprising, since it's very common for "fancy storage software" products to work great in the lab, but then run into odd bugs when used over a long period of time in a real world (read: messy) environment. If you haven't already opened a support ticket with StableBit, I'd recommend doing so.

 

[dtgoodwin]

Thanks for the explanation Mfenn. I wasn't fully understanding what you were describing. I'm only moderately familiar with Unix/Linux/BSD, and then with only single volumes.

I have decided to go ahead and either go Ubuntu or FreeNAS. I'll be removing my 2 and 1TB drives. Newegg had a deal for 4 3TB RED drives, so I will have a total of 7. It was more than I wanted to spend, but I think this will be the best solution. That will give me plenty of space to grow, and all the drives will be matched.

I could open a ticket with Stablebit, but I don't know if that's the cause or not. The issues I've had are recent, but I've been on the same version of Drivepool for quite sometime. My data is becoming increasingly critical and I have more and more of it, so I want additional protection of a more advanced filesystem. I'm becoming increasingly nervous about silent corruption beyond the problems I am currently having, so it's the motivation to make the move.

I will also be reconfiguring my backup to my secondary offsite server. Any sugggestions to guides to setup OpenVPN, or another tool to be able to securely transfer files to my other server? It will remain a Windows-based machine.

Thanks again for all the advice.

 

[mfenn]

The easiest place to implement OpenVPN is at your router. Many of the popular aftermarket router firmwares such as DD-WRT and Tomato have OpenVPN preinstalled. Of course, more advanced routing platforms like pfsense also have it.

If that's not an option, you can set up an OpenVPN Access Server on the (Linux) file server, and then forward the appropriate ports to it.

In either case, the client side is pretty straightforward since there is an OpenVPN Windows client.

 

[dtgoodwin]

Just wanted to reply back. I got my drives, ran WD's long check followed by a long format. I pulled all the existing drives out except for the REDs. I ended up with 7 which created a minor issue. With only 6 Intel ports and my SuperMicro controller not supported I needed another port. I only had an Asmedia or Silicon Image 3132. I plugged the Asmedia in and everything was recognized.

I built a 7-drive RaidZ2 array. Everything seems to be functioning well. I created the volumes I needed, and shared them and copied everything from my existing drives. It had no trouble keeping up with multiple drives copying and was always fully saturating a single 1Gb link.

My ARC hit rate is extremely low, but it is climbing now that all the copying is done. I think that I will want to upgrade to 16GB of RAM when I can afford it. It also looks like I will be in the market for a new processor after learning that the Sandy Bridge i3 ECC support does not exist. I think I will move from the i3 2500T to a Pentium G2030 - the only Ivy Bridge processor that supports ECC and doesn't cost several hundred.

Thanks much for all the help. I will need to work on getting my VPN setup - I have PFSense routers at both locations which do have OpenVPN packages for them - it's just the challenge of getting it configured properly. For now, I'll continue to backup to my local drives. When I feel confident that everything has been copied over, I may use the old 2TB drives temporarily to create a second set of backups before I try to sell them.

 

[mfenn]

My ARC hit rate is extremely low, but it is climbing now that all the copying is done.

ARC stands for Adaptive Read Cache, so you would expect a low hit rate when you're mostly writing new data into the array.

You should see the hit rate rise as it learns what your most commonly accessed files are. Anyway, glad you got it all set up!

Thanks much for all the help. I will need to work on getting my VPN setup - I have PFSense routers at both locations which do have OpenVPN packages for them - it's just the challenge of getting it configured properly. For now, I'll continue to backup to my local drives. When I feel confident that everything has been copied over, I may use the old 2TB drives temporarily to create a second set of backups before I try to sell them.

Ah cool, pfsense at both ends is perfect. I think the best thing to do is to make sure that both sites are using different IP subnets, for example:

Your house: 192.168.1.0/24
Dad's house: 192.168.2.0/24

Then once you have that set up, you can do a site-to-site VPN, and traffic will be able to flow from one site to another without doing anything special on the individual PCs. To the PC's, everything will appear to be on one network.