Need to write at 250MB/sec for ~15 min

[Knavish]

I need to get a system that's capable of writing (video) at 250MB/sec for approximately 15 minutes (about 220GB). This is for a work-related effort, so buying a pre-made system might be preferable from the "don't blame it on me if it breaks" perspective. The system also needs to include a PCIe-x4 video capture board and a PCIe-x2 (or possibly old PCI) timing board. Excessive power consumption (heat) is not desirable because the computer must fit into a pretty tight space. For this reason, I'm thinking of a micro ATX system with an i7 CPU and integrated video.

The most challenging part of this build seems to be sustaining the 250MB/sec writing rate. I guess a Raid-0 of SSDs would work, but do I need a real Raid card to maximize performance vs. the integrated Intel raid?

I am actually really tempted by an OCZ RevoDrive 3 like this:
OCZ RevoDrive 3, 480GB, PCIe 2.0 x4. They're apparently capable of sequential writes well in excess of 250MB/sec.

Any experience or comments on a Revodrive vs. Raid setup?

I'll try to keep this post updated with details once I get the system figured out. (Vendor, or built from parts, and disk benchmarks...)

Thanks!

 

[mnewsham]

Do you need the drive to be the boot drive? I have heard some (not much) difficulty getting the RevoDrive to be the boot drive.

 

[Mark R]

If it's just sequential writes, then pretty much software/Intel RAID-0+ 4x any modern hard drives will do 250 MB/s writes sustained.

I've got 250 MB/s from 4x 1TB 5900 rpm drives in RAID 5 using software (linux) RAID. If you go for 2.5" drives, like WD velociraptors, and use 5.25" drive bays, you can cram them in very tight (4 or 5 hard drives in 2 5.25" bays). I'd expect 4 velociraptors to get well over 400 MB/s.

The revodrive is, however, a simpler solution - available off the shelf. It's a lot more expensive, but for a solution that'll "just work", that's about as close as it's going to get.

 

[bunnyfubbles]

I can easily hit over 300MB/sec with just a 3 drive RAID 0 on an intel chipset for sustained video writing purposes

I don't know how mission critical your work is but I'd have a hard time imagining that a simple RAID-0 setup with at least 3 current or even last gen 7200RPM HDDs wouldn't do the trick.

although with the HDD shortage flooding crisis fiasco, if you don't already have the drives the thought of buying multiple new HDDs at this juncture isn't a fun one with prices just about more than double what they were a month ago.

 

[Knavish]

Interesting -- I didn't consider that spinning drives could reliably get up to 300+ MB/sec, but I guess it makes sense. In my case I think I'll stick with SSDs since there may be some vibration that could harm the spinning HDs, and the computer case options will be very limited when I'm looking for a compact micro atx case that supports 3-4 full size disks.

mnewsham: good point about booting on the Revodrive. I'll have to consider adding a 2nd small SSD boot drive if it becomes necessary. Since it is basically a RAID chipset plus flash memory on a PCIe board, I'd hope that it would boot on any computer that allows booting from RAID cards. Maybe OCZs drivers aren't quite up to the task ...

 

[mfenn]

I wouldn't worry about the lack of bootability (new word?) on a Revo drive because you won't want to put your system on the video capture disk anyway. Since you need to guarantee a fairly high throughput, anything else that writes to that disk (like the OS) will give you trouble. It'd be better to get a smaller SSD or just a plain old HDD for the system drive.

As for the PC, you will need to of course make sure that whatever you buy has a free PCIe x4 (or longer) slot. A Dell Precision T1600 meets that requirement and is fairly small (though not tiny).

Another think to keep in mind is that many SSDs have a "write cliff" for long-term sustained writes beyond which the performance is terrible. How long "long-term" is depends on the SSD, but you don't find many reviews of consumer level products that test this case. A RAID of traditional HDDs has no such limitation.

 

[slayernine]

Another think to keep in mind is that many SSDs have a "write cliff" for long-term sustained writes beyond which the performance is terrible. How long "long-term" is depends on the SSD, but you don't find many reviews of consumer level products that test this case. A RAID of traditional HDDs has no such limitation.

TRIM is supposed to help with that write cliff effect. My understanding is that you hit a wall when you run out of blank sectors and you have to start reorganizing the data on the fly. So the first couple times you use the drive might be very different experiences after you have written and deleted files on the SSD raid array. Also there is no TRIM if you use RAID so you will see performance tank much quicker. I think there is still no TRIM with RAID, someone correct me if I'm wrong.

 

[mnewsham]

TRIM is supposed to help with that write cliff effect. My understanding is that you hit a wall when you run out of blank sectors and you have to start reorganizing the data on the fly. So the first couple times you use the drive might be very different experiences after you have written and deleted files on the SSD raid array. Also there is no TRIM if you use RAID so you will see performance tank much quicker. I think there is still no TRIM with RAID, someone correct me if I'm wrong.

http://www.bit-tech.net/news/hardware/2010/03/23/intel-releases-trim-for-raid/1

 

[slayernine]

http://www.bit-tech.net/news/hardware/2010/03/23/intel-releases-trim-for-raid/1

Kinda figured someone would have figured out a solution to it which they obviously have. TRIM being the godsend it has been for SSD technology.

To help with this problem perhaps someone who has a Revo could trying running a 15minute benchmark.

 

[Knavish]

According to Anandtech the Revodrive 3 does support TRIM, so I assume it won't have the "write cliff" problem.

I wonder how it is exposed to the OS: does it look like a single drive, or is it just like I've attached 4x 120GB drives in RAID0. Maybe this isn't a particularly good solution vs. a motherboard-based RAID array, except for the space savings.
----
edit:

The more I read about it, the OCZ card seems a bit unreliable. I should probably go with a RAID0 of 2x Intel / Corsair / Crucial drives in the >400GB size range plus a third smaller SSD for the OS. That should get me more than 300MB/sec for sequential writes, though it's making my computer more and more difficult to procure commercially. I'll have to reconsider if a desktop case (Dell Precision like mfenn suggested) will fit into the available space.

 

[mfenn]

TRIM is supposed to help with that write cliff effect. My understanding is that you hit a wall when you run out of blank sectors and you have to start reorganizing the data on the fly. So the first couple times you use the drive might be very different experiences after you have written and deleted files on the SSD raid array. Also there is no TRIM if you use RAID so you will see performance tank much quicker. I think there is still no TRIM with RAID, someone correct me if I'm wrong.

TRIM does help with the write cliff, but only under certain workloads. In a desktop workload where you're fairly constantly creating, updating, and deleting relatively small files, TRIM has lots of deletes that are spread out temporally such that it can manage the free page list fairly effectively.

The OP's case is a little different. He is going to be writing huge files flat out for long periods of time. This is very stressful on the drive because it simply does not have the time to clean up dirty blocks in the background, TRIM or no TRIM. Thus the performance is bound by the time it takes to erase a block and performance falls off the cliff. Clever algorithms can help to some extent, but raw, brute force overprovisioning is really the only way to completely mitigate it.

 

[slayernine]

TRIM does help with the write cliff, but only under certain workloads. In a desktop workload where you're fairly constantly creating, updating, and deleting relatively small files, TRIM has lots of deletes that are spread out temporally such that it can manage the free page list fairly effectively.

The OP's case is a little different. He is going to be writing huge files flat out for long periods of time. This is very stressful on the drive because it simply does not have the time to clean up dirty blocks in the background, TRIM or no TRIM. Thus the performance is bound by the time it takes to erase a block and performance falls off the cliff. Clever algorithms can help to some extent, but raw, brute force overprovisioning is really the only way to completely mitigate it.

As much as I hate to say it, advice like this points to spinning disks as a much better alternative. SSD's are still suffering from immaturity it seems.

 

[slayernine]

Does anyone here know of a common disk testing/benching utility that can be forced to run 15min continuous?

I've run some comparison benchmarks with what I have at the office. No fancy SSD setups unfortunately.

RAID 10, SAS 15K RPM 146GB Dell 2.5" drives

RAID 10 with some 7200RPM 500GB drive

Also here is an X25-M G2 160GB drive that as example doesn't make the cut.

 

[bunnyfubbles]

I wouldn't worry about the lack of bootability (new word?) on a Revo drive because you won't want to put your system on the video capture disk anyway. Since you need to guarantee a fairly high throughput, anything else that writes to that disk (like the OS) will give you trouble. It'd be better to get a smaller SSD or just a plain old HDD for the system drive.

while a huge no-no for writing to a HDD that had the OS or other apps on it, SSDs are an entirely different story

granted, I would keep things separate (ie dedicated OS/app drive) if only for the sake of simplicity and organization

 

[mfenn]

while a huge no-no for writing to a HDD that had the OS or other apps on it, SSDs are an entirely different story

granted, I would keep things separate (ie dedicated OS/app drive) if only for the sake of simplicity and organization

IOPS are IOPS, both HDD and SSDs have a limited supply of them to go around. An HDD just has a lot less of them. The OP has a situation where it sounds like he absolutely must not ever go below 250 MB/s. Having an OS running the the same drive makes it very difficult to guarantee that level of performance.

 

[mfenn]

Does anyone here know of a common disk testing/benching utility that can be forced to run 15min continuous?

Well, I know a bunch of Linux tools to do it, but that probably doesn't help you very much. You can get Iometer to do what you want though.

 

[Knavish]

Regarding spinning drives: a RAID0 of 2x or 4x high RPM SAS's would definitely get me there, but I don't think spinning disks will work for my application. In particular, I am worried about potentially high vibrations during the video recording. I'm also operating in a sealed container with limited ventilation -- temperatures can increase pretty fast when I'm running. Temps shouldn't impact me within limited record period, though. The environment is definitely closer to harsh industrial than to enterprise computer rack.

Perhaps i can leave the computer on & idling for 12+ hours before the data collection to maximize garbage collection -- that might help with the SSD write cliff, along with having plenty of free disk space. I'm pretty sure my sealed container has enough ventilation to keep temperatures steady with an idling CPU.

This is good info. Thanks for the discussion people!

 

[ViRGE]

Well, I know a bunch of Linux tools to do it, but that probably doesn't help you very much. You can get Iometer to do what you want though.

Another vote for IOMeter here.

 

[greenhawk]

just a small addition. While some SSD's do mention a good write rate, the sandforce based units do on the fly compression to get to those speeds. Video, espically compressed video, is the wrost sort of data to write to a sandforce chiped SSD in terms of performance. For some units it drops down to 100MB/s sort of speeds from their rated 300+.

the M4's I do not think are as effected, but they do not compress (as I understand it). It is an area where a spining drive can be better for massive writes as it does not care what the data is.

edit: if going the writing approach, you need to work with the minimum write speeds, and even then a little less than that to allow for seek overhead when creating new files. So even at 40MB/s, you need about 7 good drives in raid 0. the SSD on a PCI slot are looking very attracive in terms of size and robustness and speed.

still 250MB/s for 15 minutes, about 225GB. give some room for over heads and a 240GB might be the smallest you can use. though looking at the next size up (500GBish) would give some room to move and even a second session before needing to be offloaded to storage.

 

[SillyBit]

Does anyone here know of a common disk testing/benching utility that can be forced to run 15min continuous? I've run some comparison benchmarks with what I have at the office. No fancy SSD setups unfortunately.

IOMETER vote here too! Awesome tool for what you are trying to do.

This site has a nice quick start Iometer tutorial:
http://www.itechstorm.com/iometer-tutorial-introduction

 

[mfenn]

Regarding spinning drives: a RAID0 of 2x or 4x high RPM SAS's would definitely get me there, but I don't think spinning disks will work for my application. In particular, I am worried about potentially high vibrations during the video recording. I'm also operating in a sealed container with limited ventilation -- temperatures can increase pretty fast when I'm running. Temps shouldn't impact me within limited record period, though. The environment is definitely closer to harsh industrial than to enterprise computer rack.

Perhaps i can leave the computer on & idling for 12+ hours before the data collection to maximize garbage collection -- that might help with the SSD write cliff, along with having plenty of free disk space. I'm pretty sure my sealed container has enough ventilation to keep temperatures steady with an idling CPU.

This is good info. Thanks for the discussion people!

What sort of vibrations are we talking about? 2.5" 10K SAS drives are pretty good at handling moderate vibration. Also, the drives will likely draw ~12W a piece, so even 4 of them is less than what you're CPU will be drawing in the same situation. I'd probably also try to increase ventilation if at all possible.

 

[Knavish]

What sort of vibrations are we talking about? 2.5" 10K SAS drives are pretty good at handling moderate vibration. Also, the drives will likely draw ~12W a piece, so even 4 of them is less than what you're CPU will be drawing in the same situation. I'd probably also try to increase ventilation if at all possible.

I wish I knew what kind of vibration to expect. I know it is transient and intermittent, not continuous. I'm also looking into water cooling, but that could be dangerous since I'm operating in a sealed space... a leak could turn my system into a fish tank.

 

[Rebel44]

IMO, for what you want to do, Revodrive 3 might be easiest solution.

 

[slayernine]

I wish I knew what kind of vibration to expect. I know it is transient and intermittent, not continuous. I'm also looking into water cooling, but that could be dangerous since I'm operating in a sealed space... a leak could turn my system into a fish tank.

For safe water cooling I would stick to systems such as the Corsair H60 which are designed to not require maintenance such as filling a reservoir and have a significantly lower potential to leak as it is a sealed system. However depending on how sealed your system is this may not be a viable solution.

Yesterday I saw a picture of a case that had heat-pipes going from the processor to the top of the case. I think that is the only truly sealed case method to dissipate heat in a meaningful way.

 

[jlfirehawk]

I have had 2 revodrives, I currently have a RevoDrive 3 x2, if you would like me to run any benchmarks let me know if that will help your decision. In both cases I have had no issues with using the Revodrive as a boot drive, at least not in any current motherboard, I did put a series one revodrive in a dell and had some troubles with that i.e. I couldnt get windows 7 to install at all but other than that no other issues accross multiple other motherboards and installs.