Keith001
Contributor
Contributor

ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

I've installed ESXi 3.5 U4 with and without the latest VMware patches on an HP DL380 G6 server with a p410i RAID controller configured with 8 SAS drives in a RAID 50 configuration. The server firmware (RAID controller and BIOS) have been updated to the most current versions.

Unfortunately, a RHEL4 VM only achieves a maximum of 5-6 MB/s in disk IO for write operations. When I run 2 different RHEL4 VM's simultaneously, the disk write performance is 2.5-3 MB/s each.

I'm fairly certain the HW is good because I've performed a native installation of RHEL5U3 on the server and have used that OS to do a local file copy at 250+ MB/s.

Does anyone have any knowledge of performance issues regarding the HP Smart Array p410i RAID controller and the VMware ESXi 3.5 U4 cciss driver?

0 Kudos
32 Replies
Jackobli
Virtuoso
Virtuoso

Hello and welcome to the VMware ESXi community forum.

Which kind of the p410 do you own? Regular setup or the extended with a battery backed write cache?

Of first, most probably the smart array adapater is not using a write cache (100% of RAM for read, 0% for write).

As ESXi itself does not cache writes, a bad write performance is obvious.

0 Kudos
Keith001
Contributor
Contributor

I believe it's a P410i without BBWC. It's integrated onto the system board with 256 MB of cache (HP Smart Array P410i/256MB Controller).

I have a hard time believeing that my RAID write IO is so slow when compared to RHEL4U3 installed natively.

0 Kudos
Jackobli
Virtuoso
Virtuoso

I have a hard time believeing that my RAID write IO is so slow when compared to RHEL4U3 installed natively.

Did you read my words? ESXi does not cache itself. So any write goes directly to the disk (except, if you install a BBWC).

Any given linux is using it's RAM quite excessive for caching IO, so you really cannot compare.

0 Kudos
fossmarkluni
Contributor
Contributor

Same here.. HP ML 370 G6 w/ 2 x E5520 and 12 gb RAM, P410i 256 mb without BBWC ESXi 3.5 U4.

2-3 MB/s write rate, which is unbelivable! Even without BBWC...

We have a HP XW4600 workstation w/ 1 x 500gb SATA for test purposes ESXi 3.5 U3, it easily writes 20 MB/s ++ with ACHI disabled (IDE mode)

BBWC battery kit ordered!

:S

0 Kudos

Do VMs show such speed all the time? By default VMDKs are created in "thick" mode, and on first access to block ESX wipes it out, causing low IOPS.

So if you run any disk test, do not take first result. Wait it to finish and run again.

Other way to prewipe all blocks is to create eagerzeroedthick VMDKs using vmkfstools command.


---

VMware vExpert '2009

http://blog.vadmin.ru

EMCCAe, MCITP: SA+VA, VCP 3/4/5, VMware vExpert http://blog.vadmin.ru
0 Kudos
fossmarkluni
Contributor
Contributor

Disk write can peak up to about 10 MB/s, but after a few secs goes down to 2-3 MB/s, have just one VM on this server ATM. (2008 server std 64 bit)

What I feel is so strange is that a normal (E8500, single SATA) workstation performs much better (disk write) than this brand new ML 370 G6, even without BBWC I expected the ML 370 to perform better!!

0 Kudos
Keith001
Contributor
Contributor

Like , we have order the HP 512MB RAID cache DIMM and battery pack. Hopefully, this will solve our disk write IO issue.

However, I would like to try prewiping all blocks to create and eagerzeroedthick VMDK. I have no experience with vmkfstools. Is it included with ESXi and accessible via an ssh session with the hypervisor? Or would additional SW/licenses be needed?

0 Kudos
depping
Leadership
Leadership

Just download the RCLI or VIMA. vmkfstools is included in those.

Duncan

VMware Communities User Moderator | VCP | VCDX

-


Blogging:

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
Keith001
Contributor
Contributor

Got the RCLI installer and see the vmksfstool.pl. Thanks.

0 Kudos
PeteLong
Contributor
Contributor

I Have the Same Problem

ML350 G6 410i RAID Card - ESX 3.5 update4

Call Logged to VMWare

Call Logged to HP - who are not being terribly helpfull - they have asked we run Array Diagnostic Utility from Smartstart.

0 Kudos
Keith001
Contributor
Contributor

We received a replacement RAID cache DIMM (1 GB) with battery pack and battery pack connector cable from HP. We replaced the 512 MB non-battery backed RAID cache DIMM in our HP DL380 G6.

After waiting a weekend for the RAID cache battery pack to be fully charged, we confirmed that our 8 disk RAID 50 volume was able to write at 70+ MB/s (at least as long as the RAID cache didn't get saturated) which is a significant improvement from a non-battery backed RAID cache (~5-7 MB/s).

Furthermore, we found that having hyperthreading enabled in the BIOS increased simultaneous, multiple VM task performance (opening notepad) on 30 Windows XP VM's by 30-40% (time for all VM's to complete the task dropped from 36 sec to 23 sec).

0 Kudos
grantdavies
Contributor
Contributor

We are running into this problem with one of our clients at the moment, can I ask - once you installed the BBWC and it was fully charged did the write performance improve without further adminisatrative intervention ordid you have to change some settings after the 2-3 days charging? Obviously we're going to upgrade them to BBWC and I'm just wondering whether we'll need to schedule a second maintenance window to configure the write caching on the controller once the battery charges.

0 Kudos
PeteLong
Contributor
Contributor

Added 512 Cache - did NOT fix the problem - HP said the Diagnostic Utility showed every drive had an error in the same sector - and that this was unlikely.

They asked that we firmware the RAID controler - it had a newer version than the current firmware CD (it was runing 1.62) Ive downgraded it to 1.58C

Just brought the server up this morning lets see how it performs..............

0 Kudos
J1mbo
Virtuoso
Virtuoso

> we confirmed that our 8 disk RAID 50 volume was able to write at 70 MB/s (at least as long as the RAID cache didn't get saturated)

Is this sequential performance? I ask since it still seems quite low, the P400's should be able to achive over 150MB/s write according to this:

I wonder if there is a partition alignment issue? ()

I read on here a while ago that automatically created VMFS partitions are not properly aligned and should be deleted and re-created using the VI Client - can't find it now predictably.

0 Kudos
Keith001
Contributor
Contributor

No further changes were needed to anything. I could not find any way to enable write caching on the RAID controller. I assume it's an automatic setting which is enabled when the RAID controller detects the battery pack attached to the RAID cache DIMM.

The only indication that the 1GB RAID cache with the battery pack was installed (other than a speed increase in writes) was the RAID controller message displayed on the server's console during power on. I seem to recall that a message was initially displayed after installing the 1GB RAID cache with the battery pack indicated that the battery pack was not fully charged and RAID performance would not optimal at that time.

0 Kudos
PeteLong
Contributor
Contributor

Downgrading the firmware on the 410i card from 1.62 to 1.58C cured the problem for me.

0 Kudos
Keith001
Contributor
Contributor

It should have been a sequential write test. I tested write performance by observing the VI Client performance chart for disk I/O while creating a 1 GB file on a RHEL4 VM. I believe the command was "dd if=/dev/zero of=/tmp/test.txt bs=1k count=1000000".

As far as partition alignment issues are concerned, I have no idea if that's a possibility. However, the datastore creation and VM creation/installation was performed via the VI Client.

0 Kudos
seocsitvmw
Contributor
Contributor

Hi, same slow disk performance problem. I've downgraded the firmware version to 1.58C but this is a rate...

time dd if=/dev/zero of=/tmp/test.txt bs=1k count=1000000

1000000+0 records in

1000000+0 records out

real 1m1.819s

user 0m0.300s

sys 0m2.380s

the I/O is less to 20MB/s... You can try the same test and past here the output?

Thanks in advance!

0 Kudos
J1mbo
Virtuoso
Virtuoso

Might be worth retesting with a bigger block and file size. On my test environment (Perc 5i raid-5, 3x sata 500GB), 1K block size 1GB file gives 15MB/s. 64K block 1GB file gives >250MB/s (effect of 512MB write cache), 64K block 2GB file gives ~60MB/s which is more realistic. Note this is dd for Windows though.

0 Kudos