Troubleshooting disk bottleneck on an HP Proliant

Toggle sidebar Toggle sidebar
K

Keylimesoda

Member
May 26, 2011
43
0
0
Home user, bought a Proliant ML350 from eBay to use as a headless WMC server.

I don't need RAID, but it looks like the only way to plug a drive into the system is through the drivebay, which is attached to an E200i RAID controller.

I've got a new Samsung 840 SSD that I've attached in the bay and setup in the RAID controller as a single logical drive. But the performance is *terrible*.

I'm happy to run whatever benchmarks would be useful for troubleshooting. Here's the output from WinSAT for the Samsung 840

Code: 
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive c -seq -read'
> Run Time 00:00:28.59
> Running: Storage Assessment '-drive c -ran -read'
> Run Time 00:00:02.96
> Running: Storage Assessment '-drive c -scen 2009'
> Run Time 00:04:12.89
> Running: Storage Assessment '-drive c -seq -write'
> Run Time 00:01:36.07
> Running: Storage Assessment '-drive c -flush -seq'
> Run Time 00:00:18.94
> Running: Storage Assessment '-drive c -flush -ran'
> Run Time 00:00:19.83
> Running: Storage Assessment '-drive c -hybrid -ran -read -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.02
> Running: Storage Assessment '-drive c -hybrid -ran -read -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.02
> Disk  Sequential 64.0 Read                   167.53 MB/s          7.2
> Disk  Random 16.0 Read                       83.24 MB/s          7.0
> Responsiveness: Average IO Rate              14.62 ms/IO          2.9
> Responsiveness: Grouped IOs                  29.62 units          1.9
> Responsiveness: Long IOs                     39.01 units          1.9
> Responsiveness: Overall                      1155.41 units          1.9
> Responsiveness: PenaltyFactor                1.0
> Disk  Sequential 64.0 Write                  6.31 MB/s          2.5
> Average Read Time with Sequential Writes     7.240 ms          5.3
> Latency: 95th Percentile                     14.795 ms          5.0
> Latency: Maximum                             55.786 ms          7.8
> Average Read Time with Random Writes         7.501 ms          5.2
> Total Run Time 00:07:00.59

Just for kicks, I setup an older 160GB WD SATA HDD to see if it was just the 840. Results were equally poor. Here's the results from the WD drive:

Code: 
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive g -seq -read'
> Run Time 00:00:28.22
> Running: Storage Assessment '-drive g -ran -read'
> Run Time 00:00:18.03
> Running: Storage Assessment '-drive g -scen 2009'
> Run Time 00:03:54.92
> Running: Storage Assessment '-drive g -seq -write'
> Run Time 00:01:26.99
> Running: Storage Assessment '-drive g -flush -seq'
> Run Time 00:00:19.50
> Running: Storage Assessment '-drive g -flush -ran'
> Run Time 00:00:18.42
> Running: Storage Assessment '-drive g -hybrid -ran -read -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.01
> Running: Storage Assessment '-drive g -hybrid -ran -read -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.01
> Disk  Sequential 64.0 Read                   89.85 MB/s          6.4
> Disk  Random 16.0 Read                       0.98 MB/s          2.9
> Responsiveness: Average IO Rate              23.65 ms/IO          2.8
> Responsiveness: Grouped IOs                  25.86 units          1.9
> Responsiveness: Long IOs                     44.69 units          1.9
> Responsiveness: Overall                      1155.46 units          1.9
> Responsiveness: PenaltyFactor                1.0
> Disk  Sequential 64.0 Write                  7.02 MB/s          2.6
> Average Read Time with Sequential Writes     7.151 ms          5.3
> Latency: 95th Percentile                     10.357 ms          5.7
> Latency: Maximum                             19.232 ms          7.9
> Average Read Time with Random Writes         6.632 ms          5.5
> Total Run Time 00:06:47.46

OS is Win7 x64 Ultimate

Am I missing a software config somewhere? Or drivers? Should I just get a separate SATA card and ditch the RAID controller all together?
 
IndyColtsFan

IndyColtsFan

Lifer
Sep 22, 2007
33,655
688
126
Keylimesoda said:
Home user, bought a Proliant ML350 from eBay to use as a headless WMC server.

I don't need RAID, but it looks like the only way to plug a drive into the system is through the drivebay, which is attached to an E200i RAID controller.

I've got a new Samsung 840 SSD that I've attached in the bay and setup in the RAID controller as a single logical drive. But the performance is *terrible*.

I'm happy to run whatever benchmarks would be useful for troubleshooting. Here's the output from WinSAT for the Samsung 840

Code: 
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive c -seq -read'
> Run Time 00:00:28.59
> Running: Storage Assessment '-drive c -ran -read'
> Run Time 00:00:02.96
> Running: Storage Assessment '-drive c -scen 2009'
> Run Time 00:04:12.89
> Running: Storage Assessment '-drive c -seq -write'
> Run Time 00:01:36.07
> Running: Storage Assessment '-drive c -flush -seq'
> Run Time 00:00:18.94
> Running: Storage Assessment '-drive c -flush -ran'
> Run Time 00:00:19.83
> Running: Storage Assessment '-drive c -hybrid -ran -read -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.02
> Running: Storage Assessment '-drive c -hybrid -ran -read -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.02
> Disk  Sequential 64.0 Read                   167.53 MB/s          7.2
> Disk  Random 16.0 Read                       83.24 MB/s          7.0
> Responsiveness: Average IO Rate              14.62 ms/IO          2.9
> Responsiveness: Grouped IOs                  29.62 units          1.9
> Responsiveness: Long IOs                     39.01 units          1.9
> Responsiveness: Overall                      1155.41 units          1.9
> Responsiveness: PenaltyFactor                1.0
> Disk  Sequential 64.0 Write                  6.31 MB/s          2.5
> Average Read Time with Sequential Writes     7.240 ms          5.3
> Latency: 95th Percentile                     14.795 ms          5.0
> Latency: Maximum                             55.786 ms          7.8
> Average Read Time with Random Writes         7.501 ms          5.2
> Total Run Time 00:07:00.59

Just for kicks, I setup an older 160GB WD SATA HDD to see if it was just the 840. Results were equally poor. Here's the results from the WD drive:

Code: 
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive g -seq -read'
> Run Time 00:00:28.22
> Running: Storage Assessment '-drive g -ran -read'
> Run Time 00:00:18.03
> Running: Storage Assessment '-drive g -scen 2009'
> Run Time 00:03:54.92
> Running: Storage Assessment '-drive g -seq -write'
> Run Time 00:01:26.99
> Running: Storage Assessment '-drive g -flush -seq'
> Run Time 00:00:19.50
> Running: Storage Assessment '-drive g -flush -ran'
> Run Time 00:00:18.42
> Running: Storage Assessment '-drive g -hybrid -ran -read -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.01
> Running: Storage Assessment '-drive g -hybrid -ran -read -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.01
> Disk  Sequential 64.0 Read                   89.85 MB/s          6.4
> Disk  Random 16.0 Read                       0.98 MB/s          2.9
> Responsiveness: Average IO Rate              23.65 ms/IO          2.8
> Responsiveness: Grouped IOs                  25.86 units          1.9
> Responsiveness: Long IOs                     44.69 units          1.9
> Responsiveness: Overall                      1155.46 units          1.9
> Responsiveness: PenaltyFactor                1.0
> Disk  Sequential 64.0 Write                  7.02 MB/s          2.6
> Average Read Time with Sequential Writes     7.151 ms          5.3
> Latency: 95th Percentile                     10.357 ms          5.7
> Latency: Maximum                             19.232 ms          7.9
> Average Read Time with Random Writes         6.632 ms          5.5
> Total Run Time 00:06:47.46

OS is Win7 x64 Ultimate

Am I missing a software config somewhere? Or drivers? Should I just get a separate SATA card and ditch the RAID controller all together?
Click to expand...

That controller only supports SATA I, so that is probably part of the reason you're seeing poor performance, especially with the SSD. Have you updated the firmware to the latest version and also tried different drivers?
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
The RAID controller is running the latest firmware. By default Win7 installed the 6.12 drivers. I'm updating to the 6.14 drivers from the HP site right now.

I understood going in that a SATA I controller would limit throughput, but it's the responsiveness/write times that are bringing my system to a crawl--would those be related?
 
Last edited:
K

Keylimesoda

Member
May 26, 2011
43
0
0
I've downloaded a couple of configuration software pieces from here:

http://h20000.www2.hp.com/bizsuppor...sId=1157688&swLang=8&taskId=135&swEnvOID=2100

One problem is that my controller cache accelerator (128MB RAM) has been disabled because the battery is dead. I'm not sure how much the controller depends on that component?

Outside of that, I can't find anything in the configuration software that would seem to indicate the slowdown/bottleneck I'm seeing. I do have the full diagnostic report from the ADU tool available if someone thinks that would be helpful?
 
Last edited:
K

Keylimesoda

Member
May 26, 2011
43
0
0
Yeah, I've got the BBWC. The array won't boot if the card isn't installed, but the cache is disabled since the battery has long since lost it's charge.

It appears that the cache is used primarily in RAID scenarios (RAID 5?)--any guess if it could be slowing down normal disk operations?
 
Ketchup

Ketchup

Elite Member
Sep 1, 2002
14,559
248
106
Keylimesoda said:
Yeah, I've got the BBWC. The array won't boot if the card isn't installed, but the cache is disabled since the battery has long since lost it's charge.

It appears that the cache is used primarily in RAID scenarios (RAID 5?)--any guess if it could be slowing down normal disk operations?
Click to expand...

Doesn't sound like it. The extra memory would be great in a raid 5 setup, but the only thing that SHOULD be a tad slow is the SSD. Hard drive should run great. Couple things I can think of right now are:

1. Check device manager (any devices not installed?)
2. Check for an odd setting in the BIOS slowing you down
3. See if you can contact the seller, if it's coming from somebody who actually used the thing. They may have some ideas.

Do you know which ML350 it is? maybe a G5?
 
Last edited:
K

Keylimesoda

Member
May 26, 2011
43
0
0
Yep, it's a G5. Seller has no support here :)

The Lights-out management (ILO) devices are not installed, but those shouldn't be issues.

BIOS is reset to all defaults--I don't see much that would slow me down.
 
Ketchup

Ketchup

Elite Member
Sep 1, 2002
14,559
248
106
I am not too familiar with the configuration of these, so I would say that if none of the drivers on this page do the trick, I would just get a controller card. I got an inexpensive Rosewill model from Newegg that has served me well.

http://h20000.www2.hp.com/bizsuppor...sId=3239482&swLang=8&taskId=135&swEnvOID=1114

The only other thing I can think of is if the hard drives are going through the raid controller, even though you are not running raid, there may be some settings in that BIOS you need to adjust. Does a prompt for that controller's BIOS load when you start the machine?
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
There is a controller BIOS on boot.

That's where I had to setup both drives as individual logical, drives, each in a single RAID-0 configuration, before they'd even show up in the OS.
 
Z

zuffy

Senior member
Feb 28, 2000
684
0
71
I would install the HP Server Support Pack. Then, launch the HP Array utility. Using the utility to force cache enabled and there you can set the ratio for read/write caching.
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
I've got that option to set read/write ratio. Should I tweak that, and which direction?

It says not to force caching without a UPS, but I guess I can ignore that?
 
Last edited:
K

Keylimesoda

Member
May 26, 2011
43
0
0
Okay--enabled the "Physical Drive Write Cache" option, despite the warnings.

It looks like that fixed the problem. I'm now getting the kind of performance I would expect from both of these drives.

Thanks for all your help!
 
Ketchup

Ketchup

Elite Member
Sep 1, 2002
14,559
248
106
Very cool. Let's hope the rest of the system doesn't give you so much grief.
 
Z

zuffy

Senior member
Feb 28, 2000
684
0
71
Glad it worked. The BBWC is important for data loss if you lose power. I get the sense that the data is not as important or critical that is why I told you force the write cache enable. The ratio depends on what the server will do most, write or read. If you are not sure, just keep it at 50/50.
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
So, I found a bit more info:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01149818

Apparently, by default, the controller disables the cache on the physical drives themselves, since they can't be trusted in power loss scenarios.

So, with the BBW cache disabled because of the low battery, and the cache on the disks themselves disabled, it's no wonder I was seeing poor performance.

As I understand it, the setting I changed allowed the physical drives to utilize their own on-board cache. If I get a new battery, I should be able to see even better performance by utilizing the controller cache.

I wonder if the default physical disk cache disabling is why these controller cards get a bad rap for performance.

EDIT: Here's a technical whitepaper explaining the cache settings and architecture:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02278484/c02278484.pdf
 
Last edited:
Ketchup

Ketchup

Elite Member
Sep 1, 2002
14,559
248
106
Keylimesoda said:
So, I found a bit more info:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01149818

Apparently, by default, the controller disables the cache on the physical drives themselves, since they can't be trusted in power loss scenarios.

So, with the BBW cache disabled because of the low battery, and the cache on the disks themselves disabled, it's no wonder I was seeing poor performance.

As I understand it, the setting I changed allowed the physical drives to utilize their own on-board cache. If I get a new battery, I should be able to see even better performance by utilizing the controller cache.

I wonder if the default physical disk cache disabling is why these controller cards get a bad rap for performance.

EDIT: Here's a technical whitepaper explaining the cache settings and architecture:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02278484/c02278484.pdf
Click to expand...

Most likely. It's sad really. People who buy these things should know how to access the internet and get some answers (like you did). I guess it is easier to just start giving something a bad rap.

Did you get a good deal on this one? These things cost a pretty penny when new. What kind of hardware is in it?
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
I got the whole server (ML350 G5) for $350.

It's a dual quad-core Xeon (Core arch, L5335 proc) with 8GB of RAM (max 32GB), 6 LFF HDD bay (or 8 SFF), and the RAID controller (SATA I or SAS). It also has a few PCI-E x4 slots, a handful of PCI-X, and a hardware accelerated NIC.

Oh, and dual/redundant 1000W power supply.

Don't think I coulda build a machine like that for $350...
 
Ketchup

Ketchup

Elite Member
Sep 1, 2002
14,559
248
106
That's a sweet deal. You're right, couldn't build one for that. Great opportunity to learn how those things work as well. Nice to know there are still good deals to be had on eBay, they have just gotten so hard to find.
 
Coup27

Coup27

Platinum Member
Jul 17, 2010
2,140
3
81
Isn't what you bought completely overkill for what you are going to use it for? Or did you get it because it was dirt cheap and it was something to play around with? :p
 
K

Keylimesoda

Member
May 26, 2011
43
0
0
It'll end up serving as a headless WMC server (1 GB RAM per extender), as well as a NAS (CIFS, AFP, NFS), and it's nice to have some headroom for transcoding duties.

Still even with all that, it is overkill, and I'm a sucker for dirt cheap value hardware. And I love playing with server-class hardware. Feels so... legit.

FWIW, there's dozens of machines in this price range on eBay and craigslist (Seattle). This wasn't any particularly rare deal. Just search for things like "Xeon", "Poweredge", "Proliant" in the $200-$400 range. You can get a better deal if you're willing to put up with a rack-mount form factor.
 
Last edited:
K

Keylimesoda

Member
May 26, 2011
43
0
0
zuffy said:
Too noisy and big for home use.
Click to expand...

True that. Not to mention inefficient in power use. And exotic hardware issues (see: this thread).

Thankfully I have cheap power and an out-of-the-way basement storage room, and too much spare time on my hands.
 
E

Emulex

Diamond Member
Jan 28, 2001
9,759
1
71
dude just go grab a cheap 9240 or 9260 (ibm m5014) and it will be great. Picked up a few M5014's for $66 recently - they can handle 4 samsung 830's before they peter our (chipset) then after that its pci-e x8 (2.0) that chokes. but that thing has a ton of slots so you could probably get a few cards in there.

That thing is a beast to move huh? ;)

the e200 is the biggest POS controller ever. p400 is cheap but pci-e 1.0 and p410 is not that bad but does pci-e 2.0.

the p400/p410 or M5014 (lsi 9260-8i) are all stable - most people worry about random iops over linear with servers - not the case - reliability & stability count over performance 100x
 
You must log in or register to reply here.

TRENDING THREADS

COMPANY
RESOURCES
FOLLOW
Top Bottom