md raid 10 performance?

[Red Squirrel]

How well would a md software raid 10 perform as opposed to the hardware solution raid 10?

In the future I'd like to setup a raid 10. Since raid 10 is basically 2 raids (0 and 1), would the performance be bad when it's a software raid?

 

[Netopia]

I thought that 10 was mirrored raid 5 arrays (5 * 2).

I thought that the others were 1+0 and 0+1, depending on whether you had striped mirrors or mirrored stripes.

Joe

 

[degibson]

SW raid's performance is usually fine, because disks are phenomenally slow compared to software. IMO, HW raid makes sense only if you have small block sizes, tons of small random accesses, or extremely high-end disks.

 

[Nothinman]

AFAIK RAID5 was never involved, RAID10 has always been when the array is a striped mirror and RAID01 when you have a mirrored stripe.

But no, the performance should be fine because neither RAID1 or RAID0 require any real amount of resources to manage.

 

[Red Squirrel]

What about raid 5 is it normal for it to eat up so many resources when it's software raid? I've been having lot of issues with mine. There always seems to be 1 drive that's very slow. I was thinking it was that one drive, but now it switched to another drive it seems, after I removed one drive to rebuild it.

I'm doing rm -rf * in a folder with 1TB worth of junk data and the load is skyrocketing. It's at 3.75 now. 😱

 

[Crusty]

Have you given the array time to rebuild itself? Or are you trying to do all of this on an unclean array?

For a 1TB Raid 5 array I'd say you're looking at around 12 hrs for a rebuild.

 

[Nothinman]

What about raid 5 is it normal for it to eat up so many resources when it's software raid?

It requires parity calculations for each write which take CPU time, but on a modern CPU it shouldn't be that much of a hit.

I'm doing rm -rf * in a folder with 1TB worth of junk data and the load is skyrocketing. It's at 3.75 now.

Load is a fairly worthless performance indicator, especially considering that in Linux processes waiting on I/O are included in the load calculation.

 

[Red Squirrel]

Oh yes for sure I waited till array rebuilt. It took about 4 hours or so. Been trying to troubleshoot why it hits the system so hard when io stuff happens. There's another thread though.

 

[degibson]

Originally posted by: Nothinman

What about raid 5 is it normal for it to eat up so many resources when it's software raid?

It requires parity calculations for each write which take CPU time, but on a modern CPU it shouldn't be that much of a hit.

I'm doing rm -rf * in a folder with 1TB worth of junk data and the load is skyrocketing. It's at 3.75 now.

Load is a fairly worthless performance indicator, especially considering that in Linux processes waiting on I/O are included in the load calculation.

A lot depends on your file system. Removing files is not going to be fast in a RAID 5, because typically that entails clobbering the root inode of the file: its a small, (fairly) random write. Small random writes on RAID 5 read data to be clobbered along with the parity, recomputes the parity, then writes the new data and the new parity. Hence, the time required is something like:
T(smallW) = Max( ReadDiskA, ReadDiskB ) + Tcpu + Max( WriteDiskA, WriteDiskB )

In the long run, you are going to be limited by your slowest disk in this configuration (which appears to be sdc, as posted above). Tcpu is going to be absurdly small compared to the disk access times, and Write time is likely to be small because of queuing, at least when you're not doing tons and tons of writes (unlike the rm -rf case).

In other words, small write performance sucks on RAID 5. Its often orders of magnitude longer than on RAID0/1, because neither of those RAIDs need to read data off disk before clobbering.

Large writes are another matter entirely. If your SW raid controller is worth its salt, you should achieve write bandwidth equivalent (or near to) the slowest disk's bandwidth multiplied by the number of data disks in the array. But again, this is for large writes only.

 

[Red Squirrel]

Oh ok, so for slow rights its fairly normal then. good to know. I would just hate to know I have a hardware problem. Later on I want to do a raid 10.

 

[degibson]

Originally posted by: RedSquirrel
Oh ok, so for slow rights its fairly normal then. good to know. I would just hate to know I have a hardware problem. Later on I want to do a raid 10.

If you have one disk a lot slower than the others, then you do have a HW problem. You might have a faulty disk, which causes it to perform badly. However, if it isn't faulty, its a bad design choice to mix vastly differing disk speeds in a RAID5. Its like getting the worst of all worlds (if thats OK, then its not a problem).

 

[Fallen Kell]

Originally posted by: RedSquirrel
What about raid 5 is it normal for it to eat up so many resources when it's software raid? I've been having lot of issues with mine. There always seems to be 1 drive that's very slow. I was thinking it was that one drive, but now it switched to another drive it seems, after I removed one drive to rebuild it.

I'm doing rm -rf * in a folder with 1TB worth of junk data and the load is skyrocketing. It's at 3.75 now. 😱

RAID 5 (or 6) are horrible performance in software unless you have a really powerful multi-core CPU. The reason you see "one drive" acting slow is probably because you are in a 3 disk RAID 5, and the "slow" disk is the disk that the parity block is being written. To get the parity block, the CPU has to calculate it first, which means your slowness is related to how long it is taking your CPU to do the calculation and then send that to the disk to write.

This is why I almost always recommend a hardware solution if you are planning on RAID 5. Having dedicated specialized hardware to perform the XOR parity calculation on a hardware controller's cache memory before the actual data is sent to the hard drives is vastly superior to writing to the disks that can be written to, and then calculate the parity and finally write to the final disk/block with the parity information. In all cases however, performance increases with the number of disks in the array, as there is a smaller parity to data block ration as you add more and more disks into the array.

 

[degibson]

Originally posted by: Fallen Kell

Originally posted by: RedSquirrel
What about raid 5 is it normal for it to eat up so many resources when it's software raid? I've been having lot of issues with mine. There always seems to be 1 drive that's very slow. I was thinking it was that one drive, but now it switched to another drive it seems, after I removed one drive to rebuild it.

I'm doing rm -rf * in a folder with 1TB worth of junk data and the load is skyrocketing. It's at 3.75 now. 😱

RAID 5 (or 6) are horrible performance in software unless you have a really powerful multi-core CPU. The reason you see "one drive" acting slow is probably because you are in a 3 disk RAID 5, and the "slow" disk is the disk that the parity block is being written. To get the parity block, the CPU has to calculate it first, which means your slowness is related to how long it is taking your CPU to do the calculation and then send that to the disk to write.

This is why I almost always recommend a hardware solution if you are planning on RAID 5. Having dedicated specialized hardware to perform the XOR parity calculation on a hardware controller's cache memory before the actual data is sent to the hard drives is vastly superior to writing to the disks that can be written to, and then calculate the parity and finally write to the final disk/block with the parity information. In all cases however, performance increases with the number of disks in the array, as there is a smaller parity to data block ration as you add more and more disks into the array.

Lets make some really conservative assumptions and figure out just how much time it takes a CPU to do XOR parity generation.

Assume:
Disk access time: 1ms (bloody fast for commodity parts)
Main memory access time: 100ns (pretty slow)
L1 cache line size: 64B
Block size: 16KB
Let us further assume that we're using in-order blocking CPUs without prefetching (no Nehalem for you!) -- aka every 64B of block size is a cache miss.

So, since the block size is 16KB, for small writes, parity generation is the XOR of two 16KB regions, for a total working set size of 32KB. Thats 512 cache blocks, and therefore by virtue of our assume processor model, 512 cache misses. Servicing these cache misses takes 512*100ns, or 51.2 us. One can even add a few us here and there to account for interrupt latency, etc.

Compare 51.2 us to disk access time. I don't see a CPU problem here.

 

[Red Squirrel]

So raid 10 would be better then raid 5 for software raid right? something I'll consider for future upgrade.