Building a Home Storage Server

[skyhawk326]

Hello everyone,

I am starting on a new project and despite my best searching, so far I'm coming up short finding a good guide that can assist me.

I would like to build an independent, self backup-ed, networked storage server for my home. I've found that I have a rapidly expanding library of music, photos and digital movies that is quickly exceeding my capacity. Here's my dream, can you help me figure out the hardware I need to make it a reality?

I have a nice 19" rack system and the ability to make 1/2/3/4/5U boxes, if possible I'd like to have the solution be able to be mountable.

1) Expandable data storage (SATA I/II drives? USB? Removable? RAID Level?)
2) Network data transfer (Switch? Router?)
3) File Server (OS? CPU? MB? Memory?)
4) Optional FTP for away from home access
5) Firewall?

Budget is a concern but not a limitation, any experts out there that can weigh in?

Thanks for any assistance!
Skyhawk326

 

[Passingout101]

I wouldnt mind a bit if info on these as well!!

 

[elcamino74ss]

file server needs minimal. are you comfortable with linux and/or willing to learn? the other option is win2k3 server or try getting on the home server beta team. Im on it but havent played with it yet.

file server/ftp needs minimal cpu/ram and is more disk related esp if you want raid. Either build redundancy in your disks or have a good backup strategy in place.

for a first timer I'd suggest ubuntu server and google for some walk thrus on setting up stuff.

for a web front end to media files check out this http://www.jinzora.org/

I'm running a ubuntu server here on an old dell gx 150 p3 933 with 512mb ram and a 250gb drive. I'm running web, http, sql, file share, and content filtering for the kids internet.

 

[Madwand1]

"Old school" says that performance of the components of file servers don't matter, as you'll be bottlenecked by 100 Mb/s networking in any case, or that you don't need much performance to stream movies and music.

"New school" says file servers should be gigabit, and it's easy to get performance much faster than 100 Mb/s (e.g. 30 MB/s), but it's hard to get really high performance (e.g. 100 MB/s), and for that you'd have to get a number of things right (on both ends), and you couldn't afford to use any old CPU / parts.

"Green school" says that file servers should ideally be powered by solar panels with manual cranks for startup, and everything else, from performance to price, is secondary to that goal of low power consumption.

I think the most practical goal would be somewhere in between. Trying to achieve 100 MB/s is likely to be an exercise in frustration. Modern components can give high performance affordably with good power-saving modes, and devices which are turned off are equally the most power efficient.

"RAID alone is not a backup". Regardless of how good the RAID implementation is, it won't save you from things like user error, malware, and catastrophic failure of the PSU, and RAID implementations can add new failure opportunities of their own. If you have original data worth preserving, you should not just rely on a RAID implementation, but also have an external disconnected backup. I would suggest this factor be a constraint to the build. IMO, a cheap build + backup > a fancy build without a backup.

Storage capacity is an important constraint on a server build. Considering backup capability keeps this sane. The bigger it gets without a backup, the greater the data risk.

 

[tirouspsss]

my pc is a file server - as madwand1 said with 'newskool' - u want gigabit ethernet; for me OS choice was simple - it had to b soemthing i knew. As for back-up, well, i just dont have the income to buy two of everything (so no RAID 1 with storage hdds) so my hdds r just storage - thats it. besides most of it is 'just' media - some easily re-obtainable, some not - if something goes - tough luck! as for how to connect ur hdds (sata / usb etc) & how many u can connect - stay with SATA WHERE EVER POSSIBLE - it has the bandwidth that a hdd needs (unlike USB) As for connecting hdds to pc, well, lets just say that i have looked around & researched a lil... some of the THEORETICAL possibilites i have come across - phew! some things to keep in mind: power need (if all hdds r connected to 1 pc, it'll need good watts); if ur like me & wanna drag this thing to a LAN, consider weight & how ur hdds r stored (i have brilliant (imho) idea for a solution to this)... thats all i can think of for now...

 

[erwos]

Thoughts:
You should be able to find a 16-port gigabit switch that's mountable. I've got personal experience with the Linksys/Cisco SR2024, but that's 24 port, and massive overkill.

Don't put your drives on an external bus like Firewire or USB2. Even 1394b, which sounds like it's got a lot of bandwidth at 800mbit/s, actually gets bottlenecked fast with a software RAID and lots of modern hard drives. I'm personally feeling a lot of pain from my decision to do this, but it was all I could do with the computer hardware I had at the time. You want to use SATA, and preferably the 3.0gb/s variant. Gigabyte, I think, has a mobo out with 12 SATA ports - it costs good money, but so do PCIe SATA cards.

RAID5 is your best bet for performance/value, but only if you're using Linux. Windows Home Server is coming out late summer / early fall, and apparently uses a different redundancy method that's easier to add to. RAID under Linux is pretty trivial to set up, so don't get scared of moving in that direction. When you've got a couple terabytes of DVDs you personally ripped, you don't want to lose the time you invested into doing that, so don't use RAID-0 or spanning.

Conclusion:
For now, your best bet is a Linux system on a mobo with gigabit and as many SATA ports as you can get. You'll only need a gig of RAM or so, and the CPU doesn't matter much. Use Linux's software RAID subsystem (md) to create a giant RAID5 or RAID6 out of as many drives as you can afford (focus on $/gb value) - you can grow it as you add drives in the future, if they're of the same size ("mdadm grow"). You can use dyndns, ssh, and vsftpd to give yourself remote access as needed.

If you absolutely cannot handle Linux, wait for Windows Home Server. The betas are pretty impressive, and it'll provide you more of the remote access capabilities you seem to want.

Addendum:
If you're finding that you're getting killed by too much video, it might be time to convert to H.264, if you've got a machine powerful enough to the encoding in reasonable time.