Help me develop a storage solution

[themisfit610]

Hey guys,

So, I'm working on building a new storage system for my home.

I currently have a bunch of 250 and 500 GB drives all connected haphazardly to my desktop. An older Samsung 250 is my boot drive, and I definitely need to upgrade.

I want this silliness to end.

I have a lot of media spread across these drives, and I want to consolidate it all on a NAS, with some kind of fault tolerance. All told, I have close to 2TB of data at the moment, and want lots of room to grow.

Here's what I'm thinking:

1) Remove all the hard drives from the desktop, and replace them with a single 1TB Western Digital Caviar Black, partitioned with 300GB for the OS / apps, and the rest as a scratch partition to store work on (and my large RAW image library)

I may also keep a 500GB drive in my eSATA enclosure. It's handy as a backup device and for moving large files to other locations when I need to.

2) Build a NAS with FreeNAS, using a collection of new 1TB drives, and build a gigabit network with a cheap but decent gigabit switch. I was thinking of using iSCSI to mount volumes on my PC, and SMB to mount volumes on my laptop (since it would only need to basically stream media).

I'm not sure if FreeNAS can do both things at once without problems. iSCSI isn't necessary, but I understand it can be faster - since it works at a higher level.

Anyway, my primary concern is figuring out how many 1TB drives to buy - the Samsung F1's are about $110 right now at the egg - so more is certainly the operative word.

As I said, I have about 2TB of data right now, but as I rip my large BluRay collection I plan on filling up a lot more space. I also would like some sort of data redundancy, but the only truly important thing would be my photo library (which would be mainly on the desktop, but auto sync'd with the NAS once a month or so).

Other than that, some basic stuff like financial documents, and other personal data type stuff will be stored on the desktop - probably with a TrueCrypt volume. I will also sync this with the NAS.

I don't like RAID, as I've seen first hand the horribleness that is losing a massive volume. In fact, we lost a 30TB shelf at work the other day because of 3 disk failures

So, I think I want at least 3TB of space, and I'd like to abstract storage a bit. For example, mount mounts should be like:

\\storage-misfit\mp3
\\storage-misfit\movies
\\storage-misfit\hd-movies
\\storage-misfit\backups
\\storage-misfit\software

Now, all of these could be stored wherever, but I don't want to have to worry about where each of these shares are stored. In other words, I want to be able to easily add capacity to all the shares by simply adding another drive to the NAS.

Ideally, I'd like to be able to redistribute free space non-destructively, at any point.

If this isn't possible, I can just split up the content by expected size... like putting software and backups on one drive, movies and hd movies on another drive, mp3 and something else on another drive.

Anyway, I'm not sure exactly how to think about all these advanced storage concepts.

I'm sure someone out there is looking for a robust storage solution, like me - so I'm really loooking for any advice.

I'm also curious about hardware platforms. I'm assuming FreeNAS doesn't need much in the way of CPU and RAM - probably a single core and 512-1GB of RAM should be more than enough? I imagine Gigabit ethernet and a decent SATA controller are what really count.

Is there a known "recommended" motherboard for FreeNAS and 4-6 hard drives?

Thanks in advance,

~MiSfit

 

[RebateMonger]

You might also take a look at Microsoft's Windows Home Server. That will do exactly what you require, plus provide a web server, remote access, and automated backups of all XP and Vista PCs on your network.

You can throw any size drives into the drive pool and it'll make them available for storage. You can also enable drive redundancy on a folder-by-folder basis. You can add drives to the pool at any time. WHS will automatically balance the drives and set up the redundancy if you wish.

You'll need a 1 GHz or faster CPU, 512 MB of RAM, and a box to put the motherboard and drives into. You can add external USB and eSATA drives to the pool, too.

 

[Nothinman]

I'm not sure if FreeNAS can do both things at once without problems. iSCSI isn't necessary, but I understand it can be faster - since it works at a higher level.

Probably not, iSCSI exports the entire block device to be mounted and SMB shares a locally mounted volume. Unless you use a clustering filesystem to coordinate the two hosts you won't be able to mount the volume in both places at once.

I don't like RAID, as I've seen first hand the horribleness that is losing a massive volume. In fact, we lost a 30TB shelf at work the other day because of 3 disk failures

RAID is the only way to get any sort of fault tolerance. Losing 3 drives at once is a pretty rare event unless you have major heat, electric, etc problems.

Well, as RebateMonger mentions WHS has some strange per-directory redundancy but I have no idea how it works so I'd be hesitant to trust it.

So, I think I want at least 3TB of space, and I'd like to abstract storage a bit. For example, mount mounts should be like:

You should probably just have one big share and use directories inside of it to organize things. That's a lot more flexible because eventually you're going to want to change things up a bit.

Is there a known "recommended" motherboard for FreeNAS and 4-6 hard drives?

IIRC FreeNAS is based on FreeBSD so just check out their HCL.

 

[RebateMonger]

Originally posted by: Nothinman
Well, as RebateMonger mentions WHS has some strange per-directory redundancy but I have no idea how it works so I'd be hesitant to trust it..

Here's Microsoft's Technical Brief on "Drive Extender", which does the disk pool management.
Drive Extender is pretty amazing. It's, without a doubt, very complex technology that makes it very simple for the User to manage storage. If you need more space, you just toss in a drive and WHS manages the details.

WHS has been released for a year now, and Microsoft presumably reviewed the Drive Extender code with a fine-toothed comb when they had the "Data Corruption Bug", issuing a patch included in Power Pack One (released this summer).

I haven't read of any issues with the folder redundancy. But I wouldn't "trust it" either. Of course, I don't trust ANY form of drive redundancy with the only copy of critical data. I recommend backups of any data you can't afford to lose. WHS' Power Pack One includes a long-promised backup utility for backing up data shares to a separate hard drive.

 

[themisfit610]

Sound _very_ cool!

~MiSfit