Recommend me a RAID card

[i0's]

Looking for a good value RAID card that supports 4 SATA drives with hardware RAID 5, planning on putting four 1TB drives.

Currently I have a Promise FastTrak SX4100, but it tops out at 2.1TB.

Cheers

 

[FishAk]

Due to increased parity errors on high capacity disks, RAID 5 is a very bad idea. You will loose data. RAID 01 is much better.

 

[i0's]

Due to increased parity errors on high capacity disks, RAID 5 is a very bad idea. You will loose data. RAID 01 is much better.

hm... in all my research I never came across this issue. Does not RAID 5 just rebuilt itself when there is a parity error, pardon my ignorance.

Thanks

 

[bad_monkey]

hm... in all my research I never came across this issue. Does not RAID 5 just rebuilt itself when there is a parity error, pardon my ignorance.

Thanks

Yes, if there is an error the controller will try to rebuild the array but there is an chance that an uncorrected bit error could occur while rebuilding. The rate of this occurring is more or less constant so as you increase the size of the drives the probability increases to 100% (i think this happens near 2TB) so it is a statistical certainty that an uncorrected bit error will occur while rebuilding. Here is a decent albeit older article that describes RAID 5 in detail as well as giving the math and a description of an uncorrected bit error. As for a recommendation, I have heard good things about RocketRAID and it seems like a lot of the systems reviewed here at Anandtech use them. $120 for the 2300 at newegg. Hope that helps.

 

[FishAk]

Here, and here is a little more to read on the subject.

 

[sub.mesa]

Traditional RAID5 is indeed becoming obsolete with the increasing BER on drives. Or rather, the BER (Bit-Error-Rate) stays the same at 10^-14 or -15; while the capacity continuing to grow. In other words, HDDs are getting affected by amnesia as they get bigger.

BER can spoil a traditional RAID5 if you have a disk failure, then insert a new disk to rebuild, and while rebuilding you experience a BER error; failing your rebuild. That is quite realistic on many high-capacity disks and could cause a lot of headaches.

If you are concerned about BER, consider storing your precious data on ZFS as it maintains checksums and uses redundancy effectively. Even with a lot of BER, ZFS will keep your files safe without any required maintenance.

You may find this graph interesting; which shows the calculated statistical probability of failure in different RAID modes:

http://www.servethehome.com/wp-content/uploads/2010/08/Simple-MTTDL-Combined.png

Please note that this is with a rather high number of drives: 20 disks in one array. Still this graph is valuable in understanding that RAID5 is not that secure at all. And that's not counting the many firmware/driver quirks that plague RAID5.

Concluding, RAID5 may not be as safe as people expect it to be. I recommend checksumming as answer against corruption and backups to protect against dataloss. One of those backups being ZFS would greatly enhance your data security.

 

[pyr02k1]

sub, is there any chance you know of a large array benchmark of raid 5, 6, 50 and 60? im curious to see the performance numbers on the arrays before deciding on one. i was originally planning a mix of a raid 5 and a raid 6 setup split around 6 drives to the 5 and the remaining 14 to a 6. now im curious is one large raid 60 would return a better resulting security and reliability vs speed.

much like the image, im going to be running just around 20 2tb drives by the time im done want wanted to get the best speed but security of data as well. im hoping to do the base setup end of the year and then build a system that matches it exactly about 6 months to follow for a weekly backup schedule to ensure no bad data or commands screw up my day by botching the whole array (i'd rather lose a weeks data then a few years).

to the OP, i have an areca 1210 running 4 1.5tb drives in a 10 array. works perfect, not an issue yet.

 

[pjkenned]

Please note that this is with a rather high number of drives: 20 disks in one array. Still this graph is valuable in understanding that RAID5 is not that secure at all. And that's not counting the many firmware/driver quirks that plague RAID5.

Just to clarify, that graph actually looks worse than the RAID 5 v. RAID 6 graph looks with the UBE's factored in. I have already discussed this with sub.mesa a bit in PM's but that is a pretty simple model. Here is the simple raid MTTDL article if you are interested. The model got a lot more complex this week which, of course, begets more complex thinking and even more complexity.

pyr02k1: What are you trying to see benchmarks for? Those are going to be super OS/ controller dependent because if you think about it 20x drives can yield 2.4GB/s-3GB/s max sequential performance which is way too much for a dual core IOP348 single card controller to handle. Also, on some of the older Linux appliances I much preferred hardware raid to sw raid.

Here are two examples with different disks/ controllers/ stripe sizes:
Perc 5/i ATTO w/ 8x Seagate 7200.11 1.5TB drives
Adaptec 5805 ATTO w/ 8x Seagate Cheetah 15k.1 36.7GB

To the OP: Areca, LSI, Adaptec with battery backed write cache.

 

[Mark R]

Just to clarify, that graph actually looks worse than the RAID 5 v. RAID 6 graph looks with the UBE's factored in. I have already discussed this with sub.mesa a bit in PM's but that is a pretty simple model. Here is the simple raid MTTDL article if you are interested. The model got a lot more complex this week which, of course, begets more complex thinking and even more complexity.

The graphs and figures in that article aren't right. Specifically, the author seems to have got a decimal point in the wrong place when calculating RAID5/6/Z3/50/60 MTTDL- as a result, the probabilities are based on a risk 10x higher than it should be.

Some of the equations given for the MTTDL model are also wrong (although, the figures appear to have been calculated from the correct equations, save for the error above).

The BER issue is actually quite significant, especially on big arrays - and it drastically changes the figures for RAID5 and RAID10.

The author has also derated the MTBF - quite a reasonable thing to do. But, it's probably a bit unfair to derate it by 95%. My own use of consumer grade drives (both in and not in RAID) don't support such an aggressive derating.

I've made a spreadsheet which can be used to calculate the risk of data loss while allowing multiple parameters to be varied. The basic assumptions are:
1. Uncorrelated drive failures
2. Only drive failures considered
3. Complete capacity rebuild
4. Unlimited spares (even if there are no hot spares, you can still order a replacement in - in which case recovery time might be 1 week).

Download the spreadsheet: here

 

[i0's]

Oh wow, this has gotten wayyy more technical than I expected. But I am glad it did, I did learn something.

With all that said, which raid would you guys recommend then? Ranking from reliability, storage space and then performance, guess something similar to RAID 5 without the BER issues.

Thanks

 

[Voo]

With all that said, which raid would you guys recommend then? Ranking from reliability, storage space and then performance, guess something similar to RAID 5 without the BER issues.

Which would be ZFS (or more exactly Raid-Z with a ZFS filesystem)

 

[tweakboy]

Aren't these RAID cards like 1000 dollars. even more. I saw it on newegg. hmmm

 

[Emulex]

p212 or p400 card - with fan-out - or maybe a dell.

the p400 series can use the 1GB Flash Back write cache - helps immensely. no batteries that will wear out.

I have some SAAP packs cheap if you ever get one they add some cool extra features like raid 50/60/etc - i could let go cheap. p212/p400/p800 can benefit from this expansion pack.

 

[i0's]

Which would be ZFS (or more exactly Raid-Z with a ZFS filesystem)

Which cards have ZFS or Raid-Z?

Thanks

 

[Mark R]

Which cards have ZFS or Raid-Z?

Thanks

None.

ZFS is only included in 2 OSs, Solaris and FreeBSD (significantly out-dated ZFS in free BSD), or custom commercial hybrid OSs like Nexenta.

A ZFS add-on is available for Linux, but it is out-dated and said to have stability problems.

 

[pyr02k1]

pyr02k1: What are you trying to see benchmarks for? Those are going to be super OS/ controller dependent because if you think about it 20x drives can yield 2.4GB/s-3GB/s max sequential performance which is way too much for a dual core IOP348 single card controller to handle. Also, on some of the older Linux appliances I much preferred hardware raid to sw raid.

Here are two examples with different disks/ controllers/ stripe sizes:
Perc 5/i ATTO w/ 8x Seagate 7200.11 1.5TB drives
Adaptec 5805 ATTO w/ 8x Seagate Cheetah 15k.1 36.7GB

actually those were perfect. im planning to do a much larger array on a areca 1680ix-24 which ill likely run 20 drives on (w/ 4gb on it) in a raid 60 array and at least 2 hot spares all for storage purposes. my areca 1210 doing a secondary array that will store live stuff we've yet to watch and place into storage. i was always told that i would take a horrendous hit to throughput due to raid 5 and 6. i guess no one was taking into account the quantity of drives making it moot as the max i'd saturate is a gigabit ethernet network at this point. much appreciated for the info, helps a ton.

 

[sub.mesa]

None.

ZFS is only included in 2 OSs, Solaris and FreeBSD (significantly out-dated ZFS in free BSD)

That's a bit misleading, as FreeBSD tracks the stable ZFS versions in Solaris.

FreeBSD 8.0 -> ZFS pool version 13
FreeBSD 8.1 -> ZFS pool version 14
FreeBSD 9.0 -> ZFS pool version 15
Solaris -> ZFS pool version 15
FreeBSD 9.0+patchset -> ZFS pool version 28

So to say FreeBSD got outdated ZFS implementation is kind of wrong. I'd say the stable 8.1 (v14) could be compared with stable Solaris (v15). Anything higher is experimental/unstable at the moment; and only released in OpenSolaris as CDDL code.

A port of all the code Oracle/Sun released on ZFS thus far (and may ever do) is in the ZFS v28 patchset that Pawel announced recently. Aside from all the features, FreeBSD also has its own additions, such as being able to boot from RAID-Z (or even RAID-Z3) and the ability to combine ZFS with the many geom modules available in FreeBSD.

So if we view the future of FreeBSD and ZFS optimistically, you could see FreeBSD pulling ZFS development as core open source product that develops it, together with other projects like IllumOS.

A ZFS add-on is available for Linux, but it is out-dated and said to have stability problems.

These projects are nice but are much behind the stable ZFS implementation in FreeBSD and Solaris.

 

[Voo]

Yeah no problems with BSD8.1 and ZFS - works as advertised. And I don't see much interesting features in the newer releases I'd need as a home user (triple parity Raid-Z sure is nice for large raids, but for my few TB data? I can live with that little risk)

So if we view the future of FreeBSD and ZFS optimistically

Well the question is, should we? Atm ZFS is the best FS around imho, but the license limits it somehow (or at least makes it much more complex to port to a linux kernel - I'd say also one reason why the work on it there laggs behind) and Oracle isn't the most trustworthy company around when it comes to OSS :/


But I've got one question: Why HW raid at all? I can understand it for large servers that want to limit the CPU overhead, but today with all that CPU power around I couldn't care less for a homeserver. Also good controllers aren't cheap and the cheap ones perform usually worse than SW raid. That coupled with the fact that you can't get ZFS/Raid-Z but are stuck with Raid10 or something just doesn't make it look especially attractive.
Anything I overlook?

 

[Mark R]

But I've got one question: Why HW raid at all? I can understand it for large servers that want to limit the CPU overhead, but today with all that CPU power around I couldn't care less for a homeserver. Also good controllers aren't cheap and the cheap ones perform usually worse than SW raid. That coupled with the fact that you can't get ZFS/Raid-Z but are stuck with Raid10 or something just doesn't make it look especially attractive.
Anything I overlook?

The RAID 5/6 write hole. A hardware RAID engine with non-volatile cache effectively protects against the write hole problem. Data is only cleared from NV cache once the parity has been written.

With software RAID, a power failure/kernel crash, etc. could result in the parity not getting written to disk, even though the actual data has been written. The RAID will appear to work correctly, but if a drive dies, the parity will be wrong (with no way to detect this), and the recovery process will 'recover' random garbage instead of your data.

ZFS is able to work around this, by only updating file metadata once the parity has been written. In effect, the data just written to disk is inaccessible until the metadata is updated, which is only done once the parity is complete. If power is lost at an inopportune time, then all that happens is that you are left with an perfectly preserved old copy of your file (with parity intact in case you suffer a subsequent drive failure). You are never left with an unprotected file which could be corrupted at any time.

To some extent linux software RAID now offers a similar feature. Ext4 and some other up-to-date file systems do something very similar (but not quite as paranoid as ZFS), by writing new data in such a way that it is not accessible until the drive confirms that the data is safe. This has worked fine for ages in non-RAID, but software RAID hasn't supported the necessary 'write barriers' until very recently. However, since kernel 2.6.33, software RAID now offers write barriers, so with ext4, the write hole should in most cases only affect data that hasn't been made visible yet.

The write hole is only really an issue with RAID5/6 as it can take quite a while to read in a whole stripe so that parity can be calculated. However, it could theoretically occur in RAID 1 or 10 (where one drive in a mirror misses a write just before a crash because it is busy with something else). Again, NV cache deal with this problem.

Of course, there's nothing to stop you using a HW RAID card with NV cache and using software RAID. You'll get the benefit of very high performance and data integrity of the NV cache and flexibility and portability of software RAID - the only difference is that you don't use the cache as efficiently - but with big caches (1GB on modern cards) this shouldn't be much of an issue.

You're right, CPU usage for RAID on modern CPUs is negligible. In fact, with lots of fast drives, and complex RAID types (e.g. RAID 6 or 60) even a top-of-the-range HW RAID engine can be the bottleneck. There are a number of benchmarks about where people have benched 8 or 12 drives on a top-end card, like an HP P812, in HW RAID 6. The re-run the benches with SW RAID 6, and seen 10-20% increase in performance.

 

[bigi]

3ware 9650 4MPL. It is $320, which is not cheap, but I'll provide best RAID solution in industry.

 

[Voo]

@Mark R: Yep I can see how the cache helps with the write hole and corrupted data - but as you say ZFS already solves all those problems in SW, so for me personally my conclusion is the same: Stay with FreeBSD and ZFS and don't worry about it.. thanks!

 

[Jeff7]

Well this is all a bit unsettling, considering that I've got myself a RAID5 setup with 4 1TB drives here, and I was considering boosting them up to 2TB drives.

You'd almost expect the card makers to come up with something to deal with this. The way it sounds, if you lose and replace a drive due to a genuine failure, you've got a fair chance of being screwed anyway due to a random unrecoverable read error during the rebuild. That's kind of one of those things that pretty well breaks the product. "It provides some protection against data loss in the event of drive failure!...unless you have a drive failure."

 

[i0's]

Sorry for the late reply, been busy lately.

I do not think I can do FreeBSD as I have NO experience in UNIX systems and the server that houses the raid is Windows based that run other applications that require Windows. It it basically a headless Windows computer doubled as my file storage with my Promise FastTrak SX4100 raid card.

So what other options do I have? I do not have to stick with RAID 5, any other RAID is fine with me as long as the ranking goes from reliability, storage space and then performance.

3ware 9650 4MPL. It is $320, which is not cheap, but I'll provide best RAID solution in industry.

If you don't mind me asking, why?

Thanks.

 

[Mark R]

Well this is all a bit unsettling, considering that I've got myself a RAID5 setup with 4 1TB drives here, and I was considering boosting them up to 2TB drives.

You'd almost expect the card makers to come up with something to deal with this. The way it sounds, if you lose and replace a drive due to a genuine failure, you've got a fair chance of being screwed anyway due to a random unrecoverable read error during the rebuild. That's kind of one of those things that pretty well breaks the product. "It provides some protection against data loss in the event of drive failure!...unless you have a drive failure."

That's not a problem with RAID - that's a problem with the hard drives. Consumer level hard drives may not have the reliability required to work with your assumptions when building a RAID array. (Although, do note that the figures given for RAID 5 in the article posted earlier in the thread are way wrong. Use my spreadsheet (link above) for a more accurate calculation).

RAID5 is able to correct for a single failure. This is fine if you assume that failures only occur one at a time. For 'enterprise' level drives (which are often small) e.g. 147 GB, and which use more advanced data protection algorithms, and RAID arrays with fewer than 6 drives, this assumption is pretty good.

For consumer level drives which aim for maximum capacity, minimum price, and 'just good enough' data intergrity, this assumption starts to fail.

The fact is though that the card makers do put mitigation features in their cards. The cards offer battery-backed or flash-backed write cache, to protect against RAID 5's 'write hole' and boost performance. The card firmware will automatically surface scan the drives every few days to ensure that every sector is readable (to minimize the chance of unexpected bad sector) and also double check that the parity has been calculated correctly (a process called 'scrubbing').

However, the best mitigation features won't help if you've selected the wrong tool for the job (e.g. a RAID 5 for a 10 TB array of consumer drives).

The Big OEMs like Dell, HP and IBM, actually work with the HDD vendors, and buy HDDs with custom firmware specifically optimised to work with the data integrity features of their RAID cards.

 

[i0's]

Sorry for the late reply, been busy lately.

I do not think I can do FreeBSD as I have NO experience in UNIX systems and the server that houses the raid is Windows based that run other applications that require Windows. It it basically a headless Windows computer doubled as my file storage with my Promise FastTrak SX4100 raid card.

So what other options do I have? I do not have to stick with RAID 5, any other RAID is fine with me as long as the ranking goes from reliability, storage space and then performance.




If you don't mind me asking, why?

Thanks.

Anyone??

Thanks