RAID questions for a Video Editing rig

[Kaido]

I am planning out a video editing rig to build later this year. What I am looking for is basically a really fast boot drive plus a couple fast and safe scratch drives. Everything will be in hot-swap enclosures for easy replacement. My planned configuration is:

RAID 0 (Stripe) = 600GB (2 x 300gb Raptor)

+

RAID 5 (Spare) = 10TB (12 x 1TB Samsung F1) x 2 sets (2 x 10TB RAID 5 arrays)
* 10TB formatted with 11TB advertised

Then a separate 1TB drive for backup of the Raptor boot array. Basically I want the boot drive to be fast, but it doesn't have to be secure because I'll have a 1TB backup drive available. Would a 0+1 configuration be faster? Speed is the main objective for the boot array, safety doesn't matter because of the backup drive.

I want two sets of 10TB (usable) arrays for my video editing drives. The first set will be for capture and compression and the second set will be for editing. I've found that having two sets of fast drives is the best for my workflow - I can capture video to one set, then use the other set for editing, which gives me good performance because I'm reading from one set and writing to the other, instead of reading and writing to the same set. Then it can go from the editing set back to the first set for compression to the final output (disc or web). Like this:

1. Source -> 10TB #1 Capture
2. 10TB #1 Capture -> 10TB #2 Editing
3. 10TB #2 Editing -> 10TB #1 Compression
4. 10TB #1 Final -> Output

So I need capacity, speed, and safety, which is why RAID 5 looks good. I can do 12 drives per RAID set and get 10TB of usable space after formatting (10243.8GB). Plus it has a spare in case one fails. Is RAID 5 the best option to go with?

 

[Fullmetal Chocobo]

Looks like you've done your research and your configuration looks good. What controller are you planning on using? Is video data going to be stored on these arrays as well, or are they just for processing? The reason I ask is for backup reasons. Even RAID 5 or 6 arrays should have a backup--even though it is going to be a pain in the ass for a config like this.

Do you already have the other components of this system, or are you completely rebuilding? The reason I ask is if it would be possible to get one 12-port and one 16-port controller. With controllers like Areca's, you can have the controllers work together, so you could have two RAID 6 arrays, and have global hot spares. I would sleep easier having that much data on RAID 6 than on RAID 5.

 

[Kaido]

Originally posted by: Fullmetal Chocobo
Looks like you've done your research and your configuration looks good. What controller are you planning on using? Is video data going to be stored on these arrays as well, or are they just for processing? The reason I ask is for backup reasons. Even RAID 5 or 6 arrays should have a backup--even though it is going to be a pain in the ass for a config like this.

Do you already have the other components of this system, or are you completely rebuilding? The reason I ask is if it would be possible to get one 12-port and one 16-port controller. With controllers like Areca's, you can have the controllers work together, so you could have two RAID 6 arrays, and have global hot spares. I would sleep easier having that much data on RAID 6 than on RAID 5.

I will be using a 20TB RAID 6 NAS for storage; the local drives are just for processing. Here is the card I'm looking at:

http://www.newegg.com/Product/...x?Item=N82E16816115036

That plus four 5-port Port Multipliers (I may just do one card and run 20 drives - a little less space but I'm not sure if Highpoint supports more than one card per system). I've also considered RAID 6 for the video arrays and that might be a better choice to go with. I've seen 1TB drives as low as $157, so throwing on a few extra drives won't blow the budget. Even though the local drives aren't for storage, I too think RAID 6 would be a better choice. Oh, and the case will be custom and will be built into my desk using sliding trays, so it will be pretty easy to work on as well..

The system will mostly be a new build using a Tyan S2696 mainboard, dual 3.0ghz Xeon Harpertowns, and 32 gigs of FB-DIMMs. I'll only be porting over a few parts (DVD burner, internal card reader, Quadro FX 5600). I haven't decided which OS I'm going to run on it yet - I'm leaning towards Leopard, which is perfectly supported on this board, but I also like some of the Windows-based software like Neo HD (which doesn't work with Apple Color at this time), so Vista is the other version I'm looking at. However, I'm planning on using Red in the future and they've released Redcode Raw support for Final Cut Pro, so it will most likely be Leopard.

On a tangent, any suggestions for project archival? I'll have a massive NAS, but I'm curious as to what video professionals use to archive digital footage.

 

[Kaido]

Oh and by the time I build this, they'll probably have the 8-core chips out, so it may be a 16-core system. FCP's SmoothCam plugin is gonna love that :laugh:

 

[MerlinRML]

I had a nice long post written up, but my reply timed out and went bye-bye. So the short version. I apologize if it's a bit terse, I do not intend it as such.

Port multiplier will bottleneck each set of 5 drives. Theoretical max of 600MB/s (120MB/s)with 5 of the fastest 1TB drives streaming sequentially on the outer tracks, and you've got roughly 300MB/sec of bandwidth. You also have a PCIe bottleneck when you could have 1.2GB/sec from the drives but only have 1GB/sec at the card and PCIe bus. You're going to need to look at 2 cards or a different motherboard to work around that.

RAID 0+1 vs RAID 0 performance will be pretty darn close and should not be noticeably different. I'd recommend RAID 10 over RAID 0+1 for better rebuild times.

I don't know that I'd recommend RAID 6 for a high performance video workstation. RAID 6 uses a lot of CPU (whether host or dedicatiod IOP) to do calculations. I'm not familiar enough with highpoint to know, but you might want to benchmark RAID 5 vs RAID 6 to see how much performance you're losing due to the much higher parity calculations of RAID 6. I'd almost recommend RAID 5 or 50 over RAID 6 at this point if you're going to be very heavily write intensive, which I'm assuming you are. If you want additional protection for your data, maybe just keep a spare on hand as well.

 

[Kaido]

Originally posted by: MerlinRML
I had a nice long post written up, but my reply timed out and went bye-bye. So the short version. I apologize if it's a bit terse, I do not intend it as such.

Port multiplier will bottleneck each set of 5 drives. Theoretical max of 600MB/s (120MB/s)with 5 of the fastest 1TB drives streaming sequentially on the outer tracks, and you've got roughly 300MB/sec of bandwidth. You also have a PCIe bottleneck when you could have 1.2GB/sec from the drives but only have 1GB/sec at the card and PCIe bus. You're going to need to look at 2 cards or a different motherboard to work around that.

RAID 0+1 vs RAID 0 performance will be pretty darn close and should not be noticeably different. I'd recommend RAID 10 over RAID 0+1 for better rebuild times.

I don't know that I'd recommend RAID 6 for a high performance video workstation. RAID 6 uses a lot of CPU (whether host or dedicatiod IOP) to do calculations. I'm not familiar enough with highpoint to know, but you might want to benchmark RAID 5 vs RAID 6 to see how much performance you're losing due to the much higher parity calculations of RAID 6. I'd almost recommend RAID 5 or 50 over RAID 6 at this point if you're going to be very heavily write intensive, which I'm assuming you are. If you want additional protection for your data, maybe just keep a spare on hand as well.

Okay so definitely:

RAID 0 = 2 x 300gb Raptor
(2 cards) RAID 5 = 12 x 1TB

So two 12-port cards plus either a third 2-port card or use onboard RAID if it's supported.