Hard drive gurus. Storage solution for a photographer.

[fuzzybabybunny]

I've already got a couple of WD 500GB SATA drives. Hoping to do something like RAID5 or 6.

I store and work with a ton of photos. Sometimes when I'm doing panoramas I can be working with multiple 700MB+ files at the same time. During these operations I'm severely hard drive limited, both in reading and writing, it seems. When working with these large files in Photoshop even simple operations take a long time because PS needs to make copies of the changing file on the scratch disk for any kind of Undo or Redo operation.

In addition, I obviously want some kind of data protection. Right now the data on one of my 500GBs is mirrored on the other one. Just simple copy and paste, no RAID 1 or anything.

Card recommendations?
Any idea of total price?
How hard is this to implement?
Is it possible to run one HDD as my boot drive with Windows and just "attach" this RAID array as extra storage? I REALLY don't feel like reinstalling Vista onto a RAID array.
I've got a Linux Clark Connect computer and a Windows Vista computer. Should I do this on the main Windows box? Not too sure how many hard drives I can fit in my case and not have them overheat.

 

[Inspector Jihad]

floppy disks ftw

 

[fuzzybabybunny]

I just looked at the price of real RAID cards.

HOLY COW they're expensive

For the price of one I can buy 4x500GB HDD

 

[vailr]

Is your memory maxed out? 4x 2 GB sticks = 8 Gb & 64-bit O.S. required.
You may also want to consider going to a Mac Pro, since Mac OSX is preferred over Windows, by many video and photo professionals.
Or else maybe: convert your Windows machine to a Hackentosh:
http://forum.insanelymac.com

 

[jackschmittusa]

Why move to a Mac? They once had a software advantage in 2d, but that has not been true for some time now.

More ram, dual core cpu, 64bit os are good ideas for apps like Photoshop.

Consider DVDs for permanent storage. One copy kept close at hand, another copy at another site (friend, relative's house, safety deposit box, etc.). A fire or natural disaster could destroy all of your photos.

 

[fuzzybabybunny]

Originally posted by: jackschmittusa
Why move to a Mac? They once had a software advantage in 2d, but that has not been true for some time now.

More ram, dual core cpu, 64bit os are good ideas for apps like Photoshop.

Consider DVDs for permanent storage. One copy kept close at hand, another copy at another site (friend, relative's house, safety deposit box, etc.). A fire or natural disaster could destroy all of your photos.

Sorry, backing up 3TB worth of photos on 4.7GB DVDs is not an option.

 

[MegaVovaN]

Data = $$$.

Here's what I recommend:
-Large case.
-Beefy PSU (you'll have lots of drives)
-dedicated RAID card, $250+ for this one. RAID 5 and you can take your array to a new system.
-3 large hard drives (1 tb, 750gb) & add more later as you use up space and prices fall. I rec. 1tb drives cuz you will only be able to add 1tb drives later on

edit: read this:
How can I connect 12HDs to my computer?

 

[spikespiegal]

I do a lot of PS work with massive panos, and know the feeling.

I've already got a couple of WD 500GB SATA drives. Hoping to do something like RAID5 or 6.

Why? You actually want a write penalty and lose everything when your controller card gets dusty? Forget RAID 5, or any other RAID array requiring parity, and don't listen to anybody telling you RAID 5 is faster than a single drive but it's no the case. I know how to milk faster write speed from a RAID 0 I've configured properly, but it's not a place you want to trust your files for long term. Entirely different topic.

During these operations I'm severely hard drive limited, both in reading and writing, it seems.

Make sure you're telling Photoshop to use 85-90% memory and have 4gig in the box if you're running Vista 64-bit and CS2/3.

When working with these large files in Photoshop even simple operations take a long time because PS needs to make copies of the changing file on the scratch disk for any kind of Undo or Redo operation.

You're talking about two different issues here. One is the time it takes to open/save files, and the other is Photoshop 'thrashing' it's swap file that it uses when it's out of available RAM. To reduce 'thrashing' you need to give Photoshop as much RAM as possible, and/or put the swap file on the fastest, emptiest drive you can get.

Photoshop CS2&3 (not sure if CS has the ability) has the ability to flush undo actions which will regain some memory back. If at a point in your session you know you won't need to go backwards due to a mistake, this might be one way to keep your sanity.

In addition, I obviously want some kind of data protection. Right now the data on one of my 500GBs is mirrored on the other one.

In other words, you do a nightly drive synch and just manually copy the files to the other drives, right? Actually, this is not a bad way to go with just data files because in the event you delete something on accident you have the old drive to pull it off. In the event a drive fails your data is never more than 24 hours out of date. However, if you want true, real time protection from a drive failure you need RAID 1, so it's your call. For budget installations I've been using the Silcon Image based cards (usually Rosewill brand) because they are not only cheap (around $20) their drivers are stable and I've yet to lose a card. However, most consumer RAID cards in the same category like Promise, etc., are pretty much grease off the same meatball and have the same general installation procedure. You can mirror and create the new RAID 1 volume without losing any data or fuss.

Is it possible to run one HDD as my boot drive with Windows and just "attach" this RAID array as extra storage?

Yep. It's what I do for most of my data servers since I don't need RAID 1 on the Windows installation since I clone it via script. So, you can migrate to RAID 1 (or RAID 0) for $20 or so. I won't help you with RAID 5 cards since it's not a solution for your problem.

BTW - I have the Tamron 17-50......superb lens at the long end.

 

[spikespiegal]

-dedicated RAID card, $250+ for this one. RAID 5 and you can take your array to a new system.

RAID 5 needs to go back to mid 90's along with 1.6 gig drives, windows 3.1, and the penny loafer wearing unemployed 50yr old IT engineers in their 50's who keep pushing it cuz they have a CNA in Netware.



I see RAID 5 arrays fail more in corporate than a Britney Spears custody hearing. There isn't that much cost between 4 drives vs 3, and with 4 drives (with a decent controller card) you can do RAID 1+0 (RAID 10).

 

[fuzzybabybunny]

Well, this is my problem: I work with very large files (not just in Photoshop). Right now I'm having Smartblend work on thirty-two 1.25GB TIFF files for a panorama. My HDD is going absolutely crazy and my CPU utilization is only at about 30%. The HDD simply can't feed data to the CPU fast enough. ETA to completion? About 5 hours. I have 4GB of RAM on Vista x64. There are many programs I use that don't seem to use hardly any of this, methinks because they're just third party freeware. For example, right now Smartblend is using just 160MB of RAM, even though it's working on all those 1.25GB files. Basically having lots of ram is worthless if the program can't even use it. In fact, right now I'm even getting some severe input lag from the keyboard as I'm typing this. I type something and the text doesn't even show up until after the next 2 seconds because the HDD is thrashing so much.

Is there any hard drive solution that can speed this up? I feel like for my work all these fast processors and stuff are worthless because my HDD is such a bottleneck pretty much 100% of the time.

I thought RAID 5/6 could recreate and recover all data should one or two hard drives fail. How do they fail so often then? 3+ drives fail all at once? A power surge kills all the drives on the array at once? One drive fails but for some reason the array doesn't get rebuilt properly? Is there even a point to RAID 5/6 then if they fail so often?

 

[Blain]

Have you thought about using a new 15k SCSI along with some type of SATA backup plan?

 

[vailr]

I've read reports that showed XP as being almost twice as fast as Vista (pre-SP 1), on disc-intensive tasks.
Might be worth trying XP-x64.
A true "hardware Raid" card from Apple costs about $1,000. But Areca SATA raid cards can be found for ~$200.
http://www.newegg.com/Product/...&name=Areca+Technology
Also: the Mac Pro is now (as of Jan. 2008) configured with dual quad-core CPU's, if multiple cores would be of any help.
"Boot Camp" would allow dual-booting with whatever flavor of Windows, on the Mac Pro. Prices start at $2,499.

 

[MegaVovaN]

I don't get it, why you are recommending him macs? As if they're faster than a PC for the same money?

"Prices start at $2,499"
For 2.5k anybody can build a quad core monster with 8gb ram and multiple hard drives

 

[vailr]

Originally posted by: MegaVovaN
I don't get it, why you are recommending him macs? As if they're faster than a PC for the same money?

"Prices start at $2,499"
For 2.5k anybody can build a quad core monster with 8gb ram and multiple hard drives

Mac Pros have dual quad-core CPU's. Where else can you find a Windows compatible desktop PC with that configuration? Depends on what your budget is. Go with a single quad-core CPU Raid 0 workstation from Dell or HP, and Windows XP or Vista x64, if that's all you can afford or really want. Dell's or HP's similar dual quad-core CPU workstations actually cost more than a similarly configured Mac Pro. Last time I checked, anyway.

 

[MegaVovaN]

Ah, I missed this bit: Mac Pros have dual quad-core CPU's.

Sorry.

 

[Rottie]

Originally posted by: fuzzybabybunny

Originally posted by: jackschmittusa
Why move to a Mac? They once had a software advantage in 2d, but that has not been true for some time now.

More ram, dual core cpu, 64bit os are good ideas for apps like Photoshop.

Consider DVDs for permanent storage. One copy kept close at hand, another copy at another site (friend, relative's house, safety deposit box, etc.). A fire or natural disaster could destroy all of your photos.

Sorry, backing up 3TB worth of photos on 4.7GB DVDs is not an option.

not if you can buy 20 DVD DL discs.

 

[MegaVovaN]

Originally posted by: Rottie

Originally posted by: fuzzybabybunny
Sorry, backing up 3TB worth of photos on 4.7GB DVDs is not an option.

not if you can buy 20 DVD DL discs.

elaborate pls. you mean dl from both sides, so about 7.5 gb on each side?

 

[fuzzybabybunny]

Originally posted by: Rottie

Originally posted by: fuzzybabybunny

Originally posted by: jackschmittusa
Why move to a Mac? They once had a software advantage in 2d, but that has not been true for some time now.

More ram, dual core cpu, 64bit os are good ideas for apps like Photoshop.

Consider DVDs for permanent storage. One copy kept close at hand, another copy at another site (friend, relative's house, safety deposit box, etc.). A fire or natural disaster could destroy all of your photos.

Sorry, backing up 3TB worth of photos on 4.7GB DVDs is not an option.

not if you can buy 20 DVD DL discs.

What? Since when does a DL DVD hold 150GB each?

3000 GB / 20 = 150GB a disc

To back up 3000GB using regular 4.7GB DVDs would require about 640 disks.

 

[MegaVovaN]

Dude lost a zero there.
If you use 200 dual layer disks from both sides then you can fit 15 gb per disk (7.5 gb on one side, 7.5 on other)...either way, phail @ math.

Not to mention 3tb != 3000 gb.

3 terabytes = 3 072 gigabytes

 

[RebateMonger]

No matter how many DVDs it would take, it's too much work, and you can't trust DVDs as an archival data backup.

There's four separate problems here:

1) Obtaining a workstation configuration that has an optimal balance of CPU speed, memory, and buffer drives.
2) Obtaining a working storage system that has adequate space and adequate speed.
3) Obtaining a separate backup storage system that is as isolated from your main storage system as practical.
4) Implementing a backup system, going from your working storage system to your backup storage system, that requires as little human effort as possible (or it won't get done).