Distribute file system or similar?

[AstroGuardian]

Hello everyone,

I need your help with this one. Let's say that i have 10 computers i a network. Each computer has 6 hard drives, 1Tb each. So i have 60Terabytes of storage on separate computers but i need to have a single network share of 60Terabytes.

Is there a way i can do that? What is the closest combination to this? Will Distributed File System help me with this?

 

[yinan]

The only way I can think of doing this is to have each computer act as its own iSCSI target using something like OpenFiler, then connecting to all of these volumes from one client and using dynamic disks to combine them into one drive.

But why would you do this? If you are talking about this much storage you should look into a much more reliable appliance.

 

[AstroGuardian]

Yes of course. i have considered some storage solutions from IBM but the main concern is the PRICE and not reliability. The combination is for web hosting services which do not require redundancy or 24/7 availability. So the primary focus is 130$ per terabyte price.

I also heard that Linux has some solution for this. But i don't use Linux. If there is a Linux solution i could employ a Linux specialist.

 

[Rifter]

why dont you just set up a big raid array, i know most of the high end controllers can support 32 or more drives, that way you would only need one computer and could save cost that way.

 

[AstroGuardian]

The computer price is irrelevant (100 - 130$). I could buy 10 computers for 1000$ but high end raid controller would support SAS and not SATA. SAS are expensive and have lower amount space than SATA. The RAID controller on the other hand would cost more than 2000$. So the cost per terabyte is higher than acceptable.

Scalability is also a concern. If i need to add storage space, it would be much more scalable to add just computers and disks rather than buying additional raid controllers which would be more expensive.

 

[Rifter]

that is true, but for $100 you will be getting slow computers and the disk speed of this array you plan to build will be very slow.

 

[yinan]

This is not a good idea. if this worked dont you think more people would do it? The project I am working on has needs for about 900 TB of data. They are spending multiple millions of dollars for the infrastructure to support it. If they could get away with only buying 900 drives at $100 each dont you thing they would?
If its just for web hosting, then it shouldnt take that much space. Why would web hosting not require redundancy or 24/7 reliability? If I am paying you to host my site and you lose my data and cost me money then I will make you get it back. A much better solution is to determine what each web server requires and design around that.

 

[Nothinman]

It could def work by setting up each host with a RAID array or LVM volume of all their drives and then exporting them via iSCSI or NBD and then connecting to al of them on the front end server and making one bigger RAID or LVM volume from those. Its not a pretty setup and you'd have to be very careful about data redundancy.

 

[Gooberlx2]

Originally posted by: Nothinman
It could def work by setting up each host with a RAID array or LVM volume of all their drives and then exporting them via iSCSI or NBD and then connecting to al of them on the front end server and making one bigger RAID or LVM volume from those. Its not a pretty setup and you'd have to be very careful about data redundancy.

But, you're (or you should be) sacrificing 20TB to redundancy right off the bat, assuming RAID5 per server and RAID6 for the cluster.

RAID5 per server = 10 drives lost.
RAID6 for cluster = remaining 10 drives from two servers lost. And really, NOT running in at least a RAID6 would be crazy.

Of course this also assumes the servers aren't all plugged in to the same switch, circuit, UPS, etc...to avoid a single point of failure.

Doable, but yuck...what a pain to manage. And God forbid something does happen and such an array needs to be rebuilt....over iSCSI. OP'll be older than Father Time when it finishes.

NOT running with redundancy would be suicide, IMO.

 

[Nothinman]

Originally posted by: Gooberlx2

Originally posted by: Nothinman
It could def work by setting up each host with a RAID array or LVM volume of all their drives and then exporting them via iSCSI or NBD and then connecting to al of them on the front end server and making one bigger RAID or LVM volume from those. Its not a pretty setup and you'd have to be very careful about data redundancy.

But, you're (or you should be) sacrificing 20TB to redundancy right off the bat, assuming RAID5 per server and RAID6 for the cluster.

RAID5 per server = 10 drives lost.
RAID6 for cluster = remaining 10 drives from two servers lost. And really, NOT running in at least a RAID6 would be crazy.

Of course this also assumes the servers aren't all plugged in to the same switch, circuit, UPS, etc...to avoid a single point of failure.

Doable, but yuck...what a pain to manage. And God forbid something does happen and such an array needs to be rebuilt....over iSCSI. OP'll be older than Father Time when it finishes.

NOT running with redundancy would be suicide, IMO.

Well you could also just export each drive as a single block device and let the front end host RAID them up so you only lose 1 or 2 drives to parity. Or LVM them up as one logical linear volume and then export and RAID those although recovery from losing a drive in that setup would probably be a huge PITA.

There's also distributed filesystems like CODA but I've never touched it.

 

[Crusty]

Originally posted by: AstroGuardian
The computer price is irrelevant (100 - 130$). I could buy 10 computers for 1000$ but high end raid controller would support SAS and not SATA. SAS are expensive and have lower amount space than SATA. The RAID controller on the other hand would cost more than 2000$. So the cost per terabyte is higher than acceptable.

Scalability is also a concern. If i need to add storage space, it would be much more scalable to add just computers and disks rather than buying additional raid controllers which would be more expensive.

I don't know where you are getting your information but you can intermix SAS/SATA drives. All you need is a controller with an external SAS connection and a case for the drives/power/backplane.

Obviously a real solution is going to cost more money up front, but it will save you money in the future. The purchasing cost of hardware is only part of the big picture.

How easy will your setup be to maintain? What about disaster recovery? What about scalability? If you have to buy a whole new computer every time you want to add 6TB more of space you are going to waste FAR more money in the long run. Think of the power needed to run that computer 24/7 compared to running a couple of SAS/SATA enclosures.

 

[AstroGuardian]

You are right. When i did the financial analysis of all the factors, i gave up my idea. it's better to invest more money in serious hardware now rather to have TCO being more expensive.

Thank you all guys for helping me out with this one.