Disappoint & Strange hardware RAID 5 performance

[caveman999]

Ladies and Gentlemen:

I have a very strange situation that I hope to get your guys opinion on to see why my RAID array performance is so bad:

Common Settings:
================

* HP Z420 Workstation with E5-1620 Xeon CPU
* Win 7 x64
* LSI 9361-8i RAID Controller with 3rd Gen SAS 12 Gb/s
- Host Bus Type is x8 lane PCIe 3.0
- Internal Ports: 8
- Up to 12Gb/s per port
- 2x SFF-8643 Mini-SAS HD port
* RAID 5 with Megaraid
* Controllers' firmware upated
* MegaRAID setup parameter
- always write back
- always read ahead
- Direct IO
- Disk Cache Policy Enabled

Setup 1: Normal
================================
- 4 WD Black FZEX
- 2 Hitachi Deskstar 7k4000
- Directly link these 6 drive to LSI RAID controller using (2) 1-meter Mini-SAS HD SFF8643 to SATA fanout cables (CBLSFF8643-SATASB-10M)
- 4 drives on one of the cable (CBLSFF8643-SATASB-10M)
- 2 drives on one of the other cable (CBLSFF8643-SATASB-10M)

Setup 2: Suboptimal
==================================
- 4 WD Black FZEX
- 2 Hitachi Deskstar 7k4000
- 3 Hitachi NAS Deskstar 7k4000 (new addition, which is RAID-supported)
- Intel RES2SV240 SAS Expander Card (6Gb/s)
- Link LSI controller with Intel Expander with "SFF-8643 to SFF-8087" (Adaptec 2279700-R ACK-I-HDmSAS-mSAS-1M Cable)
- Link the above 9 hard drive to Intel Expander with three SFF-8087
- 1x 3ware CBL-SFF8087OCF-06M
- 2x areca CB-87SA-75M SFF-8087 MiniSAS to 4 SATA 0.75 Meter Breakout Cable - OEM)
- 4 drives on CBL-SFF8087OCF-06M
- 4 drives on CB-87SA-75M SFF-8087
- 1 drive on the other CB-87SA-75M SFF-8087
- Started CacheCade trial but disabled afterwards (also in the tests below)

Performance Comparision:
========================

Test with CrystalDiskMark; these statistics is the more consistent with actual user experience of SQL database query experience among different tests.

* Setup 1: Normal
---------------------------------
- Sequential Read: 815 MB/s
- Sequential Write: 814 MB/s


* Setup 2: Sub-optimal
---------------------------------
- Sequential Read: 416 MB/s
- Sequential Write: ~ 600 MB/swww.intel.com/go/RAID
- NEW UPDATE: When I initially set it up, the array was running at 1000 MB /sec, but it decrease to this 400 MB/sec later on after initialization. I'm not sure what happened. Is it possible due to hard drive bad sector?


Endnote
=========

In my rig, I also have a hardware RAID 0 setup using three Crucial M550 SSD. These drives are linked to the LSI controller with CBLSFF8643-SATASB-10M in both situations described above. Performance is as expected per specification: 1500 MB/sec.

 

[_Rick_]

There's too many variables to accurately determine where the bottleneck is.
I'd recommend going down to eight drives and cutting out the expanders first, to see if those are hampering the controllers performance.
Maybe it's also a simple RAID-5 scaling issue - the controller on the card is primarily designed around a maximum of 8 disks, so you may be outside the ideal performance envelope.

As a general note, stretching RAID-5 that thin is only very slightly better than running RAID-0. Make sure you have a relatievly high-frequency back-up solution, since the likelihood of a serious rebuild-failure (i.e. more than just a handful of single-sector errors) is starting to become non-negligible.

 

[Cerb]

Port multipliers for SAS work efficiently, and port multipliers for SATA do not. SATA is not duplex (specifically, ATA never has been duplex), so sharing drives decreases the interface's performance when multiple drives are being accessed. Speed would be much better with all SAS drives, or with a 2nd RAID controller handling the extras, to keep a 1:1 ratio of drives to controller SAS/SATA ports.

That said, use RAID 6 or RAID 10, unless these arrays will host only throw-away data (RAID 5 protects you from UREs while the array is healthy, but is risky to recover, once degraded, with big slow HDDs).

 

[caveman999]

Hi, Rick:

Thank you for your suggestions!

I totally agree with you on the RAID 5 rebuild issue. I aim for backup my data once every week. I just need RAID-5 to give me some time to backup the most essential data and then I can wipe out the entire array.

The controller is LSI 9361-8i, which is their high-end card given the price tag. So I think there should be no problem for them to handle 9 disk. I also put a fan on the radiator of the card to cool it to around 60 C.

I'll try to reduce the disk. LSI say it is due to I use consumer level drive. But I'm not quite sure that is the case, because I added NAS-optimized Hitachi drives.

There's too many variables to accurately determine where the bottleneck is.
I'd recommend going down to eight drives and cutting out the expanders first, to see if those are hampering the controllers performance.
Maybe it's also a simple RAID-5 scaling issue - the controller on the card is primarily designed around a maximum of 8 disks, so you may be outside the ideal performance envelope.

As a general note, stretching RAID-5 that thin is only very slightly better than running RAID-0. Make sure you have a relatievly high-frequency back-up solution, since the likelihood of a serious rebuild-failure (i.e. more than just a handful of single-sector errors) is starting to become non-negligible.

 

[caveman999]

Hi, Cerb!

Most of the data has compressed backup. But some of the data will take computer time to re-generate.

Now I can appreciate the difference between SAS and SATA! I read the difference before, and think SAS is good at transmitting SMART info, etc.

The thing is that I cannot mix SAS and SATA. Otherwise, I would buy SAS drive to put in my array. Maybe its time to start switching to SAS drives build SAS array? I have quite a few SATA version and don't want to throw them away just because they are SATA drive.

I also see on line people build 14-18 3 TB array using 3 TB Toshiba drives using LSI 9260 with speed around 3 thousand and RAID 6.



QUOTE=Cerb;36593770]Port multipliers for SAS work efficiently, and port multipliers for SATA do not. SATA is not duplex (specifically, ATA never has been duplex), so sharing drives decreases the interface's performance when multiple drives are being accessed. Speed would be much better with all SAS drives, or with a 2nd RAID controller handling the extras, to keep a 1:1 ratio of drives to controller SAS/SATA ports.

That said, use RAID 6 or RAID 10, unless these arrays will host only throw-away data (RAID 5 protects you from UREs while the array is healthy, but is risky to recover, once degraded, with big slow HDDs).[/QUOTE]

 

[Cerb]

If the data can be reconstructed without many human hours, then no worries on using RAID 5 w/ big SATAs. Lots of folk still want to trust that a RAID 5 has a very good chance of rebuilding after a drive failure, or being able to get all the data copied off without errors on drive failure, and so put things that are important on them.

Other configs I can think of to try, to try to even the load across ports, that might be able to improve the speed:
1. 4 on the LSI, 5 on the expander, but making sure that 3 of those 5 do not share SATA ports (does the firmware even show the connections like that?).
2. 8-in/16-out on the expander, trying only 1 shared like the above. Would need another cable to go to the expander.
3. 8-in/16-out on the expander, but in a sharing two drives per SATA port, except the last one.
4. 3/3/3 on the expander.

You aught to be able to get around 1000MBps sequential with a RAID 5 of 9 disks, somehow.

 

[caveman999]

Hi, Cerb:

So I guess you are suspecting that this problem is due to wiring on the RAID expander and bandwidth problem.

The Intel SAS expander says each port of the 6 port can handle 750 MB/sec. I will try spread the drive into different ports on the expander card and also install a second SFF8426 - SFF 8087 link from the controller to the expander. Hopefully, this will give me enough bandwidth.

The 3 Gen SAS expander from Intel (RES3FV288) will be available in this quarter according to Intel marketing books. Do you think it would be advisable to just return this 2nd Gen Intel Expander and wait for the 3rd Gen one?

I initially was able to get 1000 MB/sec for the 9-drive setup, but it dropped to 400 MB/sec. It could be that I changed the wiring.

The following is the dude I mentioned that get around 2000 MB/sec using older generation LSI controller and about the same Intel expander & all consumer drives. Mostly it is picture.

http://www.chiphell.com/thread-764993-1-1.html

If the data can be reconstructed without many human hours, then no worries on using RAID 5 w/ big SATAs. Lots of folk still want to trust that a RAID 5 has a very good chance of rebuilding after a drive failure, or being able to get all the data copied off without errors on drive failure, and so put things that are important on them.

Other configs I can think of to try, to try to even the load across ports, that might be able to improve the speed:
1. 4 on the LSI, 5 on the expander, but making sure that 3 of those 5 do not share SATA ports (does the firmware even show the connections like that?).
2. 8-in/16-out on the expander, trying only 1 shared like the above. Would need another cable to go to the expander.
3. 8-in/16-out on the expander, but in a sharing two drives per SATA port, except the last one.
4. 3/3/3 on the expander.

You aught to be able to get around 1000MBps sequential with a RAID 5 of 9 disks, somehow.

 

[imagoon]

Lots of folk still want to trust that a RAID 5 has a very good chance of rebuilding after a drive failure, or being able to get all the data copied off without errors on drive failure, and so put things that are important on them.

Rebuilt a degraded 16TB LUN just last week. Twice. We back up of course but this array has been running 4.5 years and the URE count is still zero through a couple of rebuilds and continuous low priority scrubs. I think the LUN read counter is up around 7000TB read also.

I do agree that the expander + sata is likely the issue. We have seen issues with large DAG units that are loaded with SATA on a SAS expander. The units with a certain about of brains in the DAG unit or that utilizes interposers handle it better.

 

[Cerb]

All 1TB+ SATA drives in that 16TB? If so, whose?

 

[caveman999]

Hi, imagoon

Is it possible that the 6 existing SATA drives doesn't "like" the 3 new drives?

The replacement drive (still Hitachi consumer drive) is on the road. If this doesn't resolve the performance issue, I'll return the expander along with the hard drives. From then on, all my hard drive is going to be SAS.

I never knew SAS interface can make such a large difference.

Rebuilt a degraded 16TB LUN just last week. Twice. We back up of course but this array has been running 4.5 years and the URE count is still zero through a couple of rebuilds and continuous low priority scrubs. I think the LUN read counter is up around 7000TB read also.

I do agree that the expander + sata is likely the issue. We have seen issues with large DAG units that are loaded with SATA on a SAS expander. The units with a certain about of brains in the DAG unit or that utilizes interposers handle it better.

 

[imagoon]

All 1TB+ SATA drives in that 16TB? If so, whose?

They are all 1TB Seagates with 520bytes sector firmware + controller for that type of disk.

 

[imagoon]

Hi, imagoon

Is it possible that the 6 existing SATA drives doesn't "like" the 3 new drives?

The replacement drive (still Hitachi consumer drive) is on the road. If this doesn't resolve the performance issue, I'll return the expander along with the hard drives. From then on, all my hard drive is going to be SAS.

I never knew SAS interface can make such a large difference.

Well like Cerb said, SAS is a multichannel packet based system with better node management. SAS embeds sata communications inside the packet network which adds overhead and isn't mutlipathed and full duplex.

 

[caveman999]

Thank you, Cerb and imagoon!

Other than buying SAS drive, what are the potential solutions I can try at this point? Thanks !

 

[Cerb]

They are all 1TB Seagates with 520bytes sector firmware + controller for that type of disk.

That means SAS, no? Thus rated for min 1/10th the URE rate, and realistically much better relative to the URE rate in shared enclosures. Also, they cost it .

 

[Cerb]

Thank you, Cerb and imagoon!

Other than buying SAS drive, what are the potential solutions I can try at this point? Thanks !

Running Windows? Try different drive/port configs, to see if loading fewer drives per real port makes a substantial difference, and/or if giving the controller a similar number of drives per port helps (3/3/3 should be worse than 2/2/2/1, or 1/1/1/1/1/1/1/2, FI, unless the controller prefers the same distribution per port). The more drives that share a given real port, the worse it should be. With port multipliers on Marvell SATA controllers, 3-4 drives per port really hurts IOPS. It shouldn't be as bad for sequential writing, but shouldn't be good, either.

I do not know where the SAS and SATA break down with the port multiplication--IE, does the SAS expander keep it all SAS up until the drive connection, or does it implement an internal SATA port multiplier when it sees a SATA drive?

 

[boed]

If I only needed 8 drives, the 9361 is a fantastic card. You should make sure you have the current firmware on the card, your motherboard and your hard drives (the drives can make a big difference.

I do agree the expander is probably the source of the issue.

If you need more than 8 drives you may want to consider the 81605ZQ. If they made a 9361-16i - I'd go with that but since they don't you probably want to go with the 81605ZQ which is probably what I'll do on my next system.

 

[caveman999]

Intel 3rd Gen SAS Expander RES3FV288 is going to come out this quarter. Do you think it makes a lot more sense to use 3rd gen instead of then 2nd gen?

If I only needed 8 drives, the 9361 is a fantastic card. You should make sure you have the current firmware on the card, your motherboard and your hard drives (the drives can make a big difference.

I do agree the expander is probably the source of the issue.

If you need more than 8 drives you may want to consider the 81605ZQ. If they made a 9361-16i - I'd go with that but since they don't you probably want to go with the 81605ZQ which is probably what I'll do on my next system.

 

[caveman999]

I'm going to give this approach a try. Maybe it is time to consider running both the SATA hard drives and expander.

The SAS drive costs twice as much as the SATA drives... That is a pain.

Running Windows? Try different drive/port configs, to see if loading fewer drives per real port makes a substantial difference, and/or if giving the controller a similar number of drives per port helps (3/3/3 should be worse than 2/2/2/1, or 1/1/1/1/1/1/1/2, FI, unless the controller prefers the same distribution per port). The more drives that share a given real port, the worse it should be. With port multipliers on Marvell SATA controllers, 3-4 drives per port really hurts IOPS. It shouldn't be as bad for sequential writing, but shouldn't be good, either.

I do not know where the SAS and SATA break down with the port multiplication--IE, does the SAS expander keep it all SAS up until the drive connection, or does it implement an internal SATA port multiplier when it sees a SATA drive?

 

[caveman999]

I just tried to switch around ports from:

4/3/2 to 3/3/3/ on the SAS expander. But there is no difference. Is it possible that the SATA Hitachi NAS Deskstar drives are incompatible with the six old drives? I found that the 3 new drives run much cooler than the 6 existing drives.

Running Windows? Try different drive/port configs, to see if loading fewer drives per real port makes a substantial difference, and/or if giving the controller a similar number of drives per port helps (3/3/3 should be worse than 2/2/2/1, or 1/1/1/1/1/1/1/2, FI, unless the controller prefers the same distribution per port). The more drives that share a given real port, the worse it should be. With port multipliers on Marvell SATA controllers, 3-4 drives per port really hurts IOPS. It shouldn't be as bad for sequential writing, but shouldn't be good, either.

I do not know where the SAS and SATA break down with the port multiplication--IE, does the SAS expander keep it all SAS up until the drive connection, or does it implement an internal SATA port multiplier when it sees a SATA drive?

 

[imagoon]

That means SAS, no? Thus rated for min 1/10th the URE rate, and realistically much better relative to the URE rate in shared enclosures. Also, they cost it .

They are all SATA.

 

[Cerb]

I just tried to switch around ports from:

4/3/2 to 3/3/3/ on the SAS expander. But there is no difference. Is it possible that the SATA Hitachi NAS Deskstar drives are incompatible with the six old drives? I found that the 3 new drives run much cooler than the 6 existing drives.

Clearly they aren't incompatible. I could be possible that using mismatched drives is slowing things down, but were that a substantial problem, it should have shown up before going 9 drives.

 

[boed]

I would always use matching drives. I'd eliminate any expander if possible. I'd make sure the firmware on everything is up to date.

 

[caveman999]

Thanks guys for your help!

I just formatted my 9-drive RAID 5 array, the performance jumps back to 1227 MB/sec Read and 1232 MB/sec Write even with the RAID expander in between.

So it looks like the array performance decreases as I fill up the drive capacity. How could this happen? Is this due to short-stroking? If so, why this does not happen when I'm in the previous 6-drive RAID setup.

The HDD inherent speed difference is not that big ~ around 20-30 MB/sec maybe. So I think even by taking the minimum of 150 MB/sec for each drive, the performance should be 150 * (9 - 1) = 1200 MB/sec.

I deeply puzzled and hope you guys can point me in the right direction here in terms of trouble-shooting and possible solutions.

Clearly they aren't incompatible. I could be possible that using mismatched drives is slowing things down, but were that a substantial problem, it should have shown up before going 9 drives.