Most practical way to store 1.6 terabyte of data?

[Adul]

Originally posted by: GizmoCC
Hitachi/IBM drives have a bad rep so I wanna stay away from them.

A small question regarding RAID5.... why do I need 1 extra hd? What is its purpose? Also, how will the computer behave incase one of the drives fails?

And.... no one mentioned anything about simply using 4 IDE slots?

Thanks for your assistance!

Read Ars Technica's black paper on RAID
http://arstechnica.com/paedia/r/raid-1.html

 

[Arcanedeath]

I've read though the thread again and due to your tight budget, I would suggest going w/ 1 or more of the Promise cards like CZroe suggested, but instead of using 120GB drives I would use 250GB drives as those tend to be onsale fairly often as well for around 150ish or possibly 200GB drives which I've seen as low as $80 recently (@CUSA) and it would cut down on the number of drives you would need to purchase and possibly avoid getting a new power supply. Like everyone has said getting external USB drives is going to totaly kill your performance and provide no reduncy except your CD's and this isn't really a good idea and it is not that much cheaper than the Raid 5 solution.

Another option would be to go w/ an SATA raid card as those seem to support more drives a 12 port SATA raid card can be had for around 500ish and this leaves you tons of room to upgrade and would make cable managment much easier. Also you can still use PATA drives on the SATA raid card you would just need converters which would cost in the neighborhood or 10-15ish apeice....just a thought...

 

[Zepper]

I found the software for hot-swapping with mobile racks on http://www.computergate.com they also have the mobile racks that are supposed to work well with it (beantech bt-32 with electronic switch and shock absorbers) on special w/ free shipping $20. - they have the extra inserts too $15.
.bh.

 

[CZroe]

Originally posted by: vetteguy
With all due respect, I have a few observations:

You're saying that your primary constraints are having a ton of storage available, but your budget is extremely limited. I read your last post, and the difference you calculated between the RAID setup and the USB drives is only $120. IMO, that's a small price to pay for the peace of mind RAID 5 will give you, especially if you have this much data. If you just use 8 external drives, what happens when (not if) one of them fails?

Also, you keep saying how most of this collection is Anime, movies, etc. Can you honestly say that you need INSTANT access to ALL of it, ALL THE TIME? Is there ANY of your collection that you could just burn to DVD? It's not like you COULDN'T then watch it from the DVD, and it would cost CONSIDERABLY less to store that much raw data. Or, perhaps, maybe it's time to thin out the collection? Take a look at what you have, move some of your older, less-watched stuff to DVD, freeing up room in your server for new stuff. If budget truly is your primary concern, that would solve your problem, at least for the time being. Not to mention that DVDs are (as someone else mentioned) much safer if you're looking for long-term storage.

Just my $.02

Hmm, when I try to put two and two together I get this:

_ Using terrabytes of University bandwidth
+ Has terrabytes of videos
+ Wants terrabytes of additional drive space
+ Seemingly needs "instant access to all of it all the time" as you say
= International video rip courier ala Warez group member hosting 24/7 online

JUST KIDDING! I actually take a few clues from my own usage for DVD images. My fileserver is mostly to archive all software I own as CD-images and backup hard drive images of all my PCs and laptop regularly as well as provide a central repository for all my personal files acessible from all over the home. I will soon be finishing my Media Center Edition HTPC and will be using it to store all my video content so I can access it from the laptop, desktop, HTPC, or even from work. ALL of these uses I couldn't imagine being able to do using expensive and excessive DVD-R discs or INSANE amounts of CD-Rs. Because I haven't begun to fill up my file server for it's intended purpposes (All my software discs are scattered all over and not yet archived ), I have used my fileserver in the mean-time to store a library of all my DVDs as DVD images while I work on filling it with other goodies. As far as we know he may be storing anime and movie DVD images too. Sure, they are all backed up on CD but if they are DVD images split on CDs then that's reason enough right there to roll out his new fileserver setup!

BTW, here's another suggestion for you GizmoCC:

1disk

It creates a virtual drive which you can share and access through Windows which shows the contents and attributes of all your removable discs as one huge hard drive. It's thoroughly integrated into Windows Explorer, so you could even share the directory on a filesharing service and as soon as someone attempts to download the file it will prompt you for the disc... Independantly! You could go list the contents of the drive, find the directory, open the directory, find the file, right-click it, check out the file properties/attributes, cancel out of it, then try to open/execute and only then will it ask you for the disc. I kinda with it was a little more powerful like some of the other disc cataloging programs I've used (ie, zip support where actually lets you browse and search zip files and check file attributes in those without having to insert the original disc but perhaps I'm asking too much. Some of these other programs also archive all the small text files and such, even the zip descriptions and readme.txt files in the catalog's database so you do not have to insert the disc to read and access them. Too bad these other programs do not have a drive accessible from Explorer. 1disk could really use that functionality too! Even with these shortcomings, until you can afford the drives and fileserver, I can't think of a more perfect customer for that product than you!

 

[Goosemaster]

Originally posted by: CZroe

Originally posted by: vetteguy
With all due respect, I have a few observations:

You're saying that your primary constraints are having a ton of storage available, but your budget is extremely limited. I read your last post, and the difference you calculated between the RAID setup and the USB drives is only $120. IMO, that's a small price to pay for the peace of mind RAID 5 will give you, especially if you have this much data. If you just use 8 external drives, what happens when (not if) one of them fails?

Also, you keep saying how most of this collection is Anime, movies, etc. Can you honestly say that you need INSTANT access to ALL of it, ALL THE TIME? Is there ANY of your collection that you could just burn to DVD? It's not like you COULDN'T then watch it from the DVD, and it would cost CONSIDERABLY less to store that much raw data. Or, perhaps, maybe it's time to thin out the collection? Take a look at what you have, move some of your older, less-watched stuff to DVD, freeing up room in your server for new stuff. If budget truly is your primary concern, that would solve your problem, at least for the time being. Not to mention that DVDs are (as someone else mentioned) much safer if you're looking for long-term storage.

Just my $.02

Hmm, when I try to put two and two together I get this:

_ Using terrabytes of University bandwidth
+ Has terrabytes of videos
+ Wants terrabytes of additional drive space
+ Seemingly needs "instant access to all of it all the time" as you say
= International video rip courier ala Warez group member hosting 24/7 online

JUST KIDDING! I actually take a few clues from my own usage for DVD images. My fileserver is mostly to archive all software I own as CD-images and backup hard drive images of all my PCs and laptop regularly as well as provide a central repository for all my personal files acessible from all over the home. I will soon be finishing my Media Center Edition HTPC and will be using it to store all my video content so I can access it from the laptop, desktop, HTPC, or even from work. ALL of these uses I couldn't imagine being able to do using expensive and excessive DVD-R discs or INSANE amounts of CD-Rs. Because I haven't begun to fill up my file server for it's intended purpposes (All my software discs are scattered all over and not yet archived ), I have used my fileserver in the mean-time to store a library of all my DVDs as DVD images while I work on filling it with other goodies. As far as we know he may be storing anime and movie DVD images too. Sure, they are all backed up on CD but if they are DVD images split on CDs then that's reason enough right there to roll out his new fileserver setup!

BTW, here's another suggestion for you GizmoCC:

1disk

It creates a virtual drive which you can share and access through Windows which shows the contents and attributes of all your removable discs as one huge hard drive. It's thoroughly integrated into Windows Explorer, so you could even share the directory on a filesharing service and as soon as someone attempts to download the file it will prompt you for the disc... Independantly! You could go list the contents of the drive, find the directory, open the directory, find the file, right-click it, check out the file properties/attributes, cancel out of it, then try to open/execute and only then will it ask you for the disc. I kinda with it was a little more powerful like some of the other disc cataloging programs I've used (ie, zip support where actually lets you browse and search zip files and check file attributes in those without having to insert the original disc but perhaps I'm asking too much. Some of these other programs also archive all the small text files and such, even the zip descriptions and readme.txt files in the catalog's database so you do not have to insert the disc to read and access them. Too bad these other programs do not have a drive accessible from Explorer. 1disk could really use that functionality too! Even with these shortcomings, until you can afford the drives and fileserver, I can't think of a more perfect customer for that product than you!

or you can use MS AD.

 

[dm33186]

you might want to consider the ximeta netdisk www.ximeta.com
its an external hdd with usb 2 and ethernet port on it. just get a bunch of them and a nice switch.

only downfall to the drvies are you have to install a driver for it to work

 

[tart666]

Originally posted by: Goosemaster
3+ of these....use dynamic volumes in win2k3server to combine into one spanned volume or for software raid

can 2k3 do software RAID5 on 8 drives? that would satisfy what GizmoCC wants to do, no? considering performance is not a big deal... (edit: wrong quote, bloody internet dead)

 

[Kenazo]

dbl post.

 

[Kenazo]

I think you're confusing the poor guy.

You have CD back ups of everything anyway, right?
Here's what I would do:
Buy or build a cheap arse computer (like 900mhz would be plenty) and buy a PCI Sata controller that has 4 channels.
buy 4x 200 gig SATA hard drives, 4x200gig IDE hard drives
there ya go: 1.6TB of storage. He has no need for Raid if he has back-ups anyway. Sure it might be useful, but why bother? The chance of a hard drive going isn't terribly high, and if it does, restoring off the cd's isn't that impossible.

Total cost: $1,000 or $0.63/gig
Storing the entire thing on DVD-r's would be much cheaper though, and realistically, not that much more difficult. 1.6terrabytes is 320 4.7 gig DVD's. Just catalogue them properly and it wouldn't be hard to find what you want.

 

[heartsurgeon]

get a couple of really cheap, older computers, get a couple of 250 mg ide hard drives for each computer
(500 mgs/computer), 10/100/1000 network cards (10/100 if your really cheap), a suitable swtich...

set the computers up as file servers only..bingo...you've got network storgae that is cheap, expandable, and redundant.

not need for special raid hardware, you don't need a "fast computer" with a lot of memory..

speed will be WAY better that USB, and if you get a 10/100/1000 card..it will blow firewire away as well.

just my opinion.

i would be curious what the forum member think would work as a "cheap" computer..

i am assuming a 1 mghz would be more than enough....any network storage guys out there?
what would be the lowest performance machine that would get the job done?

2 cheap used computers 1 Ghz or greater - $100.00
3 gigabit network cards - $45.00
4 250 meg harddrives - $500.00 (assuming about $.50/meg)
10/100/1000 switch - $60.00

$700.00 for 1 terabyte of online network storage..easily expandable, some degree of redundancy.

i don't know anything about linux, but with win2000 pro...you can set up a "software" raid over the network computers, or you could take a very simple approach and make the hard drives on the two computers shared folders across the network.

 

[Kenazo]

Originally posted by: heartsurgeon
get a couple of really cheap, older computers, get a couple of 250 mg ide hard drives for each computer
(500 mgs/computer), 10/100/1000 network cards (10/100 if your really cheap), a suitable swtich...

set the computers up as file servers only..bingo...you've got network storgae that is cheap, expandable, and redundant.

not need for special raid hardware, you don't need a "fast computer" with a lot of memory..

speed will be WAY better that USB, and if you get a 10/100/1000 card..it will blow firewire away as well.

just my opinion.

i would be curious what the forum member think would work as a "cheap" computer..

i am assuming a 1 mghz would be more than enough....any network storage guys out there?
what would be the lowest performance machine that would get the job done?

I think anything 1ghz and faster would be more than sufficient. Put at least 256 megs ram into it, preferably more. He won't be serving to many different computers at the same time, which would be the only reason I could see to need a faster computer.

 

[GizmoCC]

Good ideas guy.
I think the idea of having multiple computers over a network is clever but there is 1 big oversight.... physical space. My dorm room cant accomodate that kind of setup.

The more I think about this, the more I beleive that DVDs are a better option. Not because of the price, but because a "harddrive' solution would eventually get saturated no matter how many expandable slots n stuff I buy. But then again... sharing 1+tb of stuff on Direct Connect hubs has its own glory and geek status I'd love

 

[Kenazo]

Originally posted by: GizmoCC
Good ideas guy.
I think the idea of having multiple computers over a network is clever but there is 1 big oversight.... physical space. My dorm room cant accomodate that kind of setup.

The more I think about this, the more I beleive that DVDs are a better option. Not because of the price, but because a "harddrive' solution would eventually get saturated no matter how many expandable slots n stuff I buy. But then again... sharing 1+tb of stuff on Direct Connect hubs has its own glory and geek status I'd love

Get maybe half a terabyte of hard drive, but the most "Demanded" stuff on there, and then have a list where if a friend wants something bad enough, you'll copy it onto your server.

 

[GTaudiophile]

A friend of mine has a 1.4TB file server. He uses like 10 drives in a RAID config. running Gentoo/Samba.

 

[1sikbITCH]

Another option to consider.

Buy your harddrives, and then buy hard drive rack trays that allow you to plug in whichever drive you need at the time. You won't have instant access to all your stuff (have to shut down to switch drives), but on the other hand, you won't be running all your drives all the time, therefore ensuring they won't burn out from wear and tear.

Not the coolest way to do it, but most definately the cheapest. But then again, if you've already got them on cd, changing cd's can be done on the fly so having to shut down to switch out hard drives won't save you any time.

 

[minofifa]

Simply get a couple of these babies.

what's the deal with these? they look like they would work well, not much said about them though. I guess if it went down, all of your shite would be gone too... you do have CD backups though.

as far as the USB route, that sounds like frustration disaster. I don't know if this just happens to me cause i don't know what i'm doing but my laptop has a USB external hard drive and every so ofter (maybe 0-4 times a day) it will just turn off and i'll have to reboot it. Times that by 8 harddrives.... i would bash them to pieces within a month.

 

[TerryMathews]

1600/250=6.4 so 7 WD2500JBs

Your motherboard should have 2 IDE ports, so a cheap boot HD and then 3 of the WD2500JBs on the onboard IDE port (Means no CD/DVD rom)

Then buy two cheap IDE controllers like the Promise Ultra133. Connect rest of HDs to that.

Use WinXP's dynamic disks and spanning. That's the cheapest (and easiest) way to get to the 1.6TB mark.

Power the drives off of a cheap AT PSU. Just be sure to not power off the AT PSU until the computer is shut down.

 

[piasabird]

If you are on a budget you may want to price the hard drives. 4 250 - 400 Gig drives are a little on the expensive side. RAID Cards are quite expensive also. Might be better off with a motherboard that has both SATA and IDE. I think the new PCI Express chipsets from intel might come with 4 sata and also have a ATA100 port. That seems about right. You could just get 2 SATA drives and get a couple more later.