Very slow write speed on my PCI SATA RAID 5

[wwroller65]

Hi.

I installed an LSI Logic "Megaraid" SATA RAID controller, and set up a 3-drive RAID 5 volume.

I am certain that there is something wrong, since the write speed to this volume is extremely slow. A single 115 MB file takes 33 seconds to transfer to the RAID volume (from an EIDE drive on the same machine). The same transfer in the other direction takes only 5 seconds.

Specs:
AMD Athlon C 1.3 GHz on an Asus A7N266 mobo.
512MB PC2700
EIDE drive is Maxtor 40 GB, 8MB cache, 7200rpm
The 3 SATA drives on the RAID card are identical: Maxtor 200GB, 8Mb cache, 7200rpm

I'm certain that the system should be able to write to the RAID volume FAR faster than 3.5 Mbytes/sec! (The separate EIDE drive is at 23 Mbytes/sec. for the same file)

Any suggestions?

--WW

 

[mwmorph]

isnt pci limited to 33mb/s so youre really getting around

33/2
=16.6
16.6mb/s for the backupdrive
8.8 mb/s bandwith for the each of the raid 0 connected drives.
so writing slows down by 2x and reading should be at full speed , cause writing has to write to 2 drives(1 virtual raid drive and 1 backup) and reading only has 1 drive to contend with(1 virtual raid or the 1 backup).

 

[Jeff7]

Originally posted by: mwmorph
isnt pci limited to 33mb/s so youre really getting around

33/2
=16.6
16.6mb/s for the backupdrive
8.8 mb/s bandwith for the each of the raid 0 connected drives.
so writing slows down by 2x and reading should be at full speed , cause writing has to write to 2 drives(1 virtual raid drive and 1 backup) and reading only has 1 drive to contend with(1 virtual raid or the 1 backup).

Nope. 33MHz. If it was 33MB/sec, don't you think that ATA/66+ would be kind of pointless?
It's 133MB/sec.

What's the model number on the LSI card? What does your CPU usage look like when writing?

 

[wwroller65]

What's the model number on the LSI card? What does your CPU usage look like when writing?

The model is SATA 150-4. I'll check on the cpu usage later tonight.

Thanks.

-WW

 

[wwroller65]

Jeff7: CPU hovers between 40 - 50% during write operation to the RAID volume.

-WW

 

[racolvin]

Originally posted by: wwroller65
Jeff7: CPU hovers between 40 - 50% during write operation to the RAID volume.

-WW

Strange. I've got one of those same cards. I'll check mine when I get home tonite and post my results as well. Frankly I hadn't noticed it was slower than it should be but then that might just be a lack of observation on my part. Since we bought a hardware RAID card, CPU utilization shouldn't be anywhere near that - that's like software RAID performance and that's not what I wanted when I coughed up for that card.

R

 

[racolvin]

I didn't get a chance to test it last nite - the wife had company over so I had to do the gracious host thing .. anyway, I'll tackle it this weekend and see how it goes. I'm definitely curious to know, since that card wasn't cheap.

R

 

[sharkeeper]

That's firmware based and frankly without a dedicated XOR processor and host based cache, your writes are gonna suck. Having that controller hitched to a rogue chipset doesn't help matters either. Search for George Breeze patch and download a utility called HD speed.

I have it and can supply a link if you need it.

 

[wwroller65]

Originally posted by: sharkeeper
That's firmware based and frankly without a dedicated XOR processor and host based cache, your writes are gonna suck. Having that controller hitched to a rogue chipset doesn't help matters either. Search for George Breeze patch and download a utility called HD speed.

I have it and can supply a link if you need it.

Please do. I'll look into it.

As for the RAID card, my understanmding is that this model has two Silicon Image Sil3112s for the Serial ATA controller, 64 MB ECC SDRAM for the cache and an Intel GC80302 XOR processor.

By the way, what does the term "rogue chipset" mean?

Thanks for the input.

-WW

 

[racolvin]

Originally posted by: sharkeeper
That's firmware based and frankly without a dedicated XOR processor and host based cache, your writes are gonna suck. Having that controller hitched to a rogue chipset doesn't help matters either. Search for George Breeze patch and download a utility called HD speed.

I have it and can supply a link if you need it.

Well that's just it .. this card has a dedicated XOR processor, so the CPU shouldn't be doing any of the parity calculations .. the operating system doesn't even know that its a RAID array as far as I understand these things - it just things is a huge HD, with the controller doing the RAID magic unbeknownst to the OS or the CPU.

R

 

[sharkeeper]

Ok so it does have a processor and memory.

Hmmm...highly suspect a rogue wave. err chipset causing your grief.

Download this and run it.

What's the bandwidth on your RAID drive?

Rogue chipset is non intel or non amd depending on who makes your processor. Older VIA chipsets had lot of issues with bandwidth sharing on the pci bus.

 

[racolvin]

Ok, here's some info ..

I have 4 identical drives in my system (see sig). One of them is the C: drive on the mobo SATA controller. The other 3 are in a RAID 5 array off the LSI logic card.

I took a 1.048Gb VOB file from my boot drive and copied it to the raid, then reversed and copied the same file back to the boot drive. All while watching the CPU utilization.

When I copied from boot to raid, CPU utilization never got over 3 percent, but frankly the write speed was slow. I know that writing to a raid 5 is gonna be slower anyway, but it was painful to watch

Going from raid to boot was very fast, say 60 seconds thereabouts. I don't have a watch handy to get an exact time until my wife gets home - I"ll use her stopwatch

More later ...

R

 

[racolvin]

Originally posted by: sharkeeper
Ok so it does have a processor and memory.

Hmmm...highly suspect a rogue wave. err chipset causing your grief.

Download this and run it.

What's the bandwidth on your RAID drive?

Rogue chipset is non intel or non amd depending on who makes your processor. Older VIA chipsets had lot of issues with bandwidth sharing on the pci bus.

Ran that utility you linked ..

Drive 0 is my boot drive, Drive 1 is my RAID

My Drive 0 Bandwidth is 75.071, Sequential is 56.033
Drive 1 Bandwidth is 52.92, Sequential is 31.62.

Obviously the RAID is slower for writes by design but man ..

R

 

[sharkeeper]

Those bandwidth numbers are pitiful!

There should be a latency timer setting for PCI in your BIOS. Try adjusting that to obtain peak readings.

Unfortunately, to get anywhere near the write speed you SHOULD be with that card is going to require a motherboard change. For AMD processor I'd suggest nForce based chipset.

 

[racolvin]

Originally posted by: sharkeeper
Those bandwidth numbers are pitiful!

There should be a latency timer setting for PCI in your BIOS. Try adjusting that to obtain peak readings.

Unfortunately, to get anywhere near the write speed you SHOULD be with that card is going to require a motherboard change. For AMD processor I'd suggest nForce based chipset.

A mobo change?? what for? The mobo I'm using came very highly recommended by AT and several other sites - heck, I just built this machine over xmas

I'll check on the latency timers and see what I can find there. Is it possible putting it in a different slot would make any difference?

R

 

[sharkeeper]

Sometimes you have incompatible hardware. I had to change out 35 MR1600's for U320-4X's because the MR1600's WB cache was not hardware compatible with the mainboard.

 

[racolvin]

Well, it turns out that some tweaking of the parameters of the card and the logical drive will dramatically improve performance.

For the OP, go to the LSI website and download the PowerController software, to allow you to make changes via windows. First thing to do on the Logical Drive configuration, change the write cache from "Thru" to "Back". The factory default is "Thru" and its WAYYY slower than "Back". Now, you do trade a data security for performance but it shouldn't be a big deal since we aren't running servers

Anyway, my transfer rates went from 4MB/sec to 15MB/sec on a write operation from that one change alone.

R

 

[sharkeeper]

For the OP, go to the LSI website and download the PowerController software, to allow you to make changes via windows. First thing to do on the Logical Drive configuration, change the write cache from "Thru" to "Back". The factory default is "Thru" and its WAYYY slower than "Back". Now, you do trade a data security for performance but it shouldn't be a big deal since we aren't running servers

Anyway, my transfer rates went from 4MB/sec to 15MB/sec on a write operation from that one change alone.

And now a warning.

Enabling write back cache is not recommended if your controller does not have a battery back up unit. (BBU). The cache contents are volatile and will be completely lost in the event of an unexpected interruption of system power. A UPS is obviously good practice, but one must remember that you can still lose data without the battery even if you have a UPS!

The WB does increase performance. It should be way faster than 15MB/S however. Those numbers point to hardware conflict which often cannot be resolved through a BIOS setting or resource shift. This is very similar to the problem we experienced between the E7505 and MR1600 series HBA.

 

[racolvin]

Originally posted by: sharkeeper
And now a warning.

Enabling write back cache is not recommended if your controller does not have a battery back up unit. (BBU). The cache contents are volatile and will be completely lost in the event of an unexpected interruption of system power. A UPS is obviously good practice, but one must remember that you can still lose data without the battery even if you have a UPS!

The WB does increase performance. It should be way faster than 15MB/S however. Those numbers point to hardware conflict which often cannot be resolved through a BIOS setting or resource shift. This is very similar to the problem we experienced between the E7505 and MR1600 series HBA.

Absolutely true on the power thing - fortunately I do use a UPS religiously.

As for the BIOS settings, changing the PCI latency didn't help. That being the case, let me reiterate a prior question: would a different PCI slot or IRQ make any difference?

R

 

[racolvin]

oh, and one thing I should add:

I dunno about the OP but in my case, I use the card in a 32bit/33mhz slot, since I don't run a server-level mobo with 64 bit slots, which has a huge effect on data transfer rates

R

 

[sharkeeper]

As for the BIOS settings, changing the PCI latency didn't help. That being the case, let me reiterate a prior question: would a different PCI slot or IRQ make any difference?

It may so it's worth a try.

 

[racolvin]

Well, moving the controller from Slot 5 to Slot 1 didn't do squat for performance changes. Guess I'll be giving LSI a call tomorrow and see what they have to say

R

 

[racolvin]

Ok, after a discussion with LSI today, some interesting things.

Note to the OP: I have now been able to get write speeds up to 45MB, so things have improved DRAMATICALLY from my original 4MB!

1: On the Logical drive, turn Write Back ON, Turn Read to NORMAL, turn Cache Policy to DIRECT

2: On your physical drives (you'll have to go into the cards bios settings for this), set Write Cache to ENABLE on all the drives in the array.

This is what I worked out with the LSI technician on the phone and as you can see, it took my I/O performance up closer to where it should be. Of course this performance can be influenced by the number of drives you have in the RAID 5 array (more drives = faster), and of course the speed and type of your drive. In my case, I'm using WD 250G drives, 7200 RPM, 8Mb cache. If you're using Raptors, your mileage may vary

In order to test this setup, I did two things: 1) real world copy of a 1GB file to/from the array, and 2) used a program called IOMETER (free download) to run a series of tests and document the results.

In test #1, the write to the array with the 1GB file originally (prior to any changes from factory default) took a few minutes. After these changes, it takes around 40 seconds (I don't have a stopwatch handy so I can't be exact - the filecopy window at the start says it'll take 30 seconds but it takes a bit longer than that by "feel")

In test #2, the IOMETER was configured to do dummy file writes of files varying from 10MB to 40MB in a succession. This was done by altering the "All In One" test that is pre-built in the IOMETER software. After running that test 3 times, my average I/O speed was 45MB

Hope that helps some .. its certainly helped my peformance, so thanx for starting this thread and making me investigate it for myself :beer:

R

 

[wwroller65]

:thumbsup:

Been away from the loop for a few days. Looks like you've got your's wrapped up pretty well! I will follow your steps and take a look for myself. Thanks for the leg work!

-WW

 

[Kamui]

one quick question,.. what are the motherboards that you guys have been using?