VMware Cloud Community
Tularis1
Contributor
Contributor

Disk Latency ESXi 5.1.0 799733 HP-ML350 G6

Hello,

We Deployed an HP ML350 G6 Server with ESXi 5.1.0 799733 and a P410i Raid Controller.

The media I used was the Official Media from VMware and not a "Customised" HP Image.

Since this server has been in Production we have noticed lots and lots of;

"Performance has deteriorated. I/O latency increased from average value of 13351 ms to 274421 ms."

This is also coupled with when the server is under heavy disk load (Backup etc..) it becomes un-responsive.

We didn't realised that we needed a Specific HP ESXi Installation Disk. So after some research it seems ESXi Drivers are VIM files.

So my questions are:

Will loading in a VIM file for the Storage Controller fix this issue?
Is there anything else that I can change to try and alleviate this problem...

Reply
0 Kudos
20 Replies
a_p_
Leadership
Leadership

Does the P410i Raid Controller have BBWC/FBWC to enable write-back caching?

André

Reply
0 Kudos
Tularis1
Contributor
Contributor

Good Point,

Is there anyway to check this without having Physical Access to the server?

I have a feeling that there is no Battery for the Raid-Controller....

Reply
0 Kudos
a_p_
Leadership
Leadership

I don't think you can see this in the ESXi "Health" information without the HP CIM providers (HP Offline Bundle) installed on the host. You could however - to avoid downtime - open the server's cover and check whether a battery/capacitor is attached to the controller.

André

PS: Sorry, I somehow missed "... without having Physical Access to the server"

Message was edited by: a.p.

Reply
0 Kudos
Tularis1
Contributor
Contributor

Ok, Is is possible to Load in the VIBs from the "Offline Bundle" after the ESXi has been installed?

Reply
0 Kudos
a_p_
Leadership
Leadership

Yes, it's possible to install the Offline Bundle in the default ESXi installation. However, the installation requires the host to be in Maintenance Mode (no powered on VMs) and a final reboot of the host.

André)

Tularis1
Contributor
Contributor

Reply
0 Kudos
a_p_
Leadership
Leadership

Yes, this should work. All you need after shutting down the VMs is the esxcli command to install the Offline Bundle (see HP Support site for VMware vSphere 5.1)

André

Reply
0 Kudos
Tularis1
Contributor
Contributor

Andre,

This has work and I can now see the Storage Devices in the "Configuration" screen.

But I am still getting a few "Performance has deteriorated. I/O latency increased from average value of ..."

Is there anything else we can do?

Reply
0 Kudos
BenLoveday
Enthusiast
Enthusiast

Hi there,

What disk and RAID configuration are you running? What sort of IOPS are your VM's generating? I've seen heavy workloads create quite high latencies when using the HP P400/410 controllers without optimal RAID configurations and battery/flash cache modules enabled.

One thing you can do to inspect the controller config is to use the hpacucli tool that is provided in the offline bundle from HP. Have a look here: Linux - hpacucli it will give you the various syntax options. This can also tell you if the cache module is installed and enabled, etc.

Cheers!

Ben Loveday

Reply
0 Kudos
Tularis1
Contributor
Contributor

Hey Ben,

We have 5 1TB (931GB) SATA Disks in a RAID5 on the P410i Controller.

When in SSH on the ESXi Server when i run "hpacucli" I get:

     ~ # hpacucli

     -sh: hpacucli: not found

I'm sure I installed all of the Offline Packs on the HP website.... Do I need to run it from another location?

Reply
0 Kudos
BenLoveday
Enthusiast
Enthusiast


Hey Tularis1,

Firstly, your RAID array isn't going to give you a lot of performance. With 5 SATA drives in RAID5 at best you would have about 100 write IOPS total at your disposal. With heavy backups running this is probably going to kill the performance. On average a 7200rpm SATA drive will deliver about 80-100 IOPS total, then with 5 in an array gives you around 400-500 IOPS. Apply the RAID5 write penalty and you are back at a write performance of just over a single disk.

As for the hpacucli tool, try ./opt/hp/hpacucli/bin/hpacucli

Cheers!

Ben Loveday

Reply
0 Kudos
Josh26
Virtuoso
Virtuoso

As you've now got the VIB installed and can see the storage device under "configuration", you should also be able to see "battery".

If it doesn't, you probably don't have one.

The lack of a BBWC would certainly exacerbate the problem, but with or without it, I feel you're asking for a technical solution to a problem caused by the purchase of under specced hardware.

When you say "we deployed"... I'm assuming this isn't a new server?

A G6 is two generations back and there's definitely something wrong if you feel you just bought this hardware.

Reply
0 Kudos
Tularis1
Contributor
Contributor

Hey, The server is probably 6 months old.

This whole Vmware [ESXi] is very new to us so we know there is going to be a learning curve.

Really I'm just trying to work out what the errors are, what we did wrong, what we should do next time round and if there is anything we can do to fix it.

Ben: What RAID configuration would you recommend? Should we have used 10K SAS in RAID5?

Reply
0 Kudos
regnak2012
Enthusiast
Enthusiast

Hi,

Do you have the original Bill of Materials used to purchase the server? This would list if the P410i has a Battery or not - +1 this is the culprit here....

you can check if you have a battery from the ViClient Health Status - see here:

http://serverfault.com/questions/313843/how-to-find-out-more-hardware-details-using-only-esxi4-1-and...

the official HP warning is here:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01832427&lang=en&cc=us&taskI...

Similar experience noted here:

http://bogner.sh/2012/02/vmware-esxi-low-io-performance-on-hp-proliant-ml350-p410i/

Regards,

Mike


Tularis1
Contributor
Contributor

Hi Mike,

Great I will get the BBWC Ordered.

TY!

Reply
0 Kudos
BenLoveday
Enthusiast
Enthusiast

Hey Tularis1,

That's ok, to be honest I've gone through similar issues where we have been unfamiliar with virtualising particular applications.

Certainly faster drives will help but it does depend on your application/load. In the case of your backups you would certainly find that using SAS drives in a RAID10 configuration would help. A RAID5 set containing SAS drives will do a lot better than a SATA array but you still have the RAID penalty to factor in. What sort of backup software are you using? For example if using a tool such as Commvault Simpana the dedupe store can get hammered and is best situated on high performance drives. Also as Mike suggested I have heard of varying performance on some of the HP RAID cards, certainly the newer G7/Gen8 generation are much better. The HP gear is also licensed so this can affect what additional features are available to you.

Cheers,

Ben Loveday

Reply
0 Kudos
Josh26
Virtuoso
Virtuoso

Tularis1 wrote:

Hey, The server is probably 6 months old.

?

I would be asking your supplier some questions about this.

G6's haven't been available with distributors for a number of years.

Reply
0 Kudos
tjollimore
Contributor
Contributor

Actually, that's incorrect. The ML350 G6 was available until Q3 2012, I believe. I think it pretty much jumped directly to G8 to keep pace with the other HP server lines. We bought ours Q1 2012 brand new, with 4x 600GB SAS-2 10k drives. and the FBWC option. I also recently added two more of the same spec drives and migrated the RAID5 array onto them as well. I was looking at this article because I've been seeing relatively slow file copy speeds within the VM, although they will occasionally spike from 4MBps up to 150MBps. Very strange...

Reply
0 Kudos
NTShad0w
Enthusiast
Enthusiast

MAte,

occasionally jumps in copying inside vm from 4 to 150MBps indicates that in most cases your storage is heavly loaded with smallers io and maybe bad clustered/not aligned/have no bbwc/fbwc or have bad drivers and/or firmware to hp p410 raid controller with vmware esxi v5.x...

regards

NTShad0w

Reply
0 Kudos