VMware Communities: Message List - ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

grantdavies — Thu, 03 Sep 2009 05:25:46 GMT

Thanks very much for sharing the info, since this server is located at a remote site, I'll wait until out next on site visit and let you know the results once I've updated the firmware.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Rabie — Thu, 03 Sep 2009 05:20:45 GMT

There is a know issue with the P410 controller which has been resolved with update 2.00 of the firmware.

And although it says 2009/07/29 it was only generally availible mid Aug.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884082&swItem=MTX-1d292bb4f261465b82f27c28be&prodNameId=3884083&swEnvOID=4025&swLang=13&taskId=135&mode=5

	Version: 2.00 (27 Jul 2009)
	Fixes
	Fix to address high latency issues seen during heavy I/O conditions involving non-contiguous offset writes and RAID 5/6 write operations. Fix for a potential controller lockup condition when a drive is hot removed during a read operation. Additional fixes to address rare occurrences of controller unresponsiveness during heavy I/O stress tests in a lab environment.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

grantdavies — Wed, 02 Sep 2009 23:36:38 GMT

OK, this is perplexing me. A week ago we installed a 512mb cache DIMM with battery, and a week later (SURELY enough time for the battery to charge) and we're getting sustained write performance of about 15MB/s, up from 2-4, but nowhere near what it should be. The array is 3 x 450GB 15K SAS disks. I haven't looked closely at the burst speed, but surely when doing sustained read/write operations (eg. backups and restores) you would expect faster performance than 15MB/s!!! A saturated USB2 bus can usually manage 30-35MB/s consistantly. Any thoughts?

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Paul Mead — Wed, 29 Jul 2009 17:15:26 GMT

Thanks to all, but this was the issue; due to there being no raid
cache on the ML330 G6 P410 raid controller server as standard there
was a significant write performance hit (Performance utility and disk
queue. It was eventually found that the NOD32 ESET antivirus product
which was using XMON to check the exchange datastore; everytime the av
sig was updated, the entire exchange datastore would be rechecked!
Normally this would not matter as most of the time our servers and our
clients servers are performing optimally and the effect of the scan is
masked.

Due to the cache issue, the scan effect dragged down the entire
server performance. Disk queues could be as high as 40-50 for tens of
minutes at a time.

The XMON background scanning can be turned off (in fact some
recommend that it is turned off, despite it being on by default); We
have resolved the overall issue by adding 512Mb battery backed cache to
the controller and performance is much much much better and I believe
that if we had started with the cache, we would never have noticed any
issues at all.

Please note that the ESET NOD32 XMON issue is not at fault - it simply highlighted the disk performance issue.

My main gripe is that for some reason HP sell the current range of ML330 G6 generation of servers with NO=Zero cache memory as standard on the P410 controller! Silly me thought that they would not possibly sell something so handicapped.Not only that, when you decide to upgrade you can either purchase a 256Mb module or 512Mb with battery. Why not 256Mb with battery?

FYI we now get disk queues of <2 most of the time and occasional blips of higher queue levels - but it is like comparing the norfolk broads with Wales! IOMeter tests show throughputs of 160MBs reads and 230MBs writes (not bad for 7200rpm mirrored pair) - which compares to early non-bbwc results of approx 66MBps read and 52MBps writes. These non-bbwc results are worse than the headline values indicate due to the type of iometer test and also the fact that the disk queue length was so high too during the test.

Still a learning exercise for me - (1) ESXi NEEDS a properly cached controller for decent write performance - more so that running a physcial box it would seem and (2) not to take HP specs at face value.

I would love to get some feedback from HP on this matter as I was intending to standardise on their kit as we seek to use ESXi for all our clients i.e. why sell a raid controller with zero cache memory and not insist on cache upgrade or at least make it very clear tha you are buying something which for many purposes will be too slow? I sign off somewhat bemused.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

patparks1 — Mon, 20 Jul 2009 15:21:20 GMT

grantdavies:

We recently added the BBWC to a DL360 Gen 5 that was performing slowly in our data centre. I took the server down to physically install the battery. Then booted the server with Smart Start to ensure that the cache ratios were setup to the defaults (75% write and 25% read). They were there automatically. I then booted the server back up into VMWare ESXi 3.5.

Approx 24 hours later, i confirmed that speeds were approx 60-70MB/s...which was a huge improvement over previous speeds of 2-4MB/s.

Thus, I did nothing else other than shut down, plug in battery and turn back on. It's worth noting too that I had a P400i with 256MB of cache and I left the 256MB of cache, just added the BBWC and our problems are resolved.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

J1mbo — Mon, 20 Jul 2009 07:48:53 GMT

The performance stats gathered confirm that write caching (or lack thereof) is the problem. However the current configuration is dangerous.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Paul Mead — Mon, 20 Jul 2009 01:52:48 GMT

Well - I am back; mixed feelings - especially with some of the feedback - thanks to all for your input so far.

Additional info: The server was first prepped at our site and then taken to client site for final config and physical to virtual conversion. When the server first came online after the move, but before conversion, one of the 2 mirrored disks had complained and appeared to be performing a rebuild. I let this run overnight before the 1/2 day semi live run.

Tonight I have been over to the client out of hours and run a few tests. Running IOMeter with 2 workers with 2 access specifications 32kblocks with 100%read and 0%random and the 2nd with 32k blocks with 100%write and 0%random for 1 minute for each of the access specifications (2 minute total run); (In simple terms a seq read test and then a seq write test); I then proceeded to delberately switch OFF the ondisk caching (which I had in fact switched on during initial build - just thought I had not) ran the tests and then reran the tests with ondisk caching switched back on. FULL physical server and virtual server power down and restart for completeness before each test.

Summary Results:

OnDisk caching OFF: avg 47MBps read and avg 5.6MBps writes

OnDisk caching ON: avg 66MBps read and avg 52MBps writes

So that is looking quite good; unfortunately I do not have exactly the same results for comparison from the previous 24 hours of poor performance; although the closest comparison would seem to be:

Bad/Weird State with partial caching?: avg 37MBps read and avg 29MBps writes

Interim conclusion: I suspect that perhaps the disk/array problem after the server physical move may have left the ondisk caching in an odd state?

What I would say is that although things seem to be better - at times the server avg disk queue length is still higher than I would normally be happy with - I suspect that we may need to invest in some BBWC kit.

Just for completeness: the disks used in this server are 3x 250Gb "Midline" disks pn: 458926-B21

If anyone has the same setup but with 256 or 512Mb cache with or without battery backup - I would be very interested to see your results; I can send the iometer settings file across if desired.

My own virtualised server running running in a similar fashion using an LSI Megaraid SATA 300-8XLP SATA II raid adapter with 128Mb embedded cache and 2x1.5TB pn: WD15EADS mirrored disks gets figures of 92Mbps read and 110Mbps write! seems odd, but overall appears nippy enough in practice and no long disk queue during any of the normal usage or iometer workloads tested so far (obviously not trying hard enough).

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Jackobli — Sun, 19 Jul 2009 21:59:16 GMT

Paul Mead wrote:
Of course - I am wondering if people are running their server with decent levels of battery backed cache, but that may still have the default ondisk cache off - I guess they might never realise that the ondisk cache is OFF and could be enjoying even better performance - what a waster that would be.

Oh boy, this has been discussed and disputed for eons.
THOU SHALL NOT USE WRITE CACHE ON THE DISK ITSELF.
Anything could happen, if your disk controller and your OS are thinking, they wrote the data and it hasn't happen. There is more than the main power that could fail.
Go buy a BBWC for your controller, if you are depending on your VMs and your data.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

J1mbo — Sun, 19 Jul 2009 21:58:06 GMT

Unfortunately that servers disk spec is very poor for ESXi. I read recently that 'near-line' is derived from the drives being somewhere between on-line and off-line (tape) storage - generally little more than desktop SATA disks. In any case the controller based BBWC is critical. The underlying disk write cache I'm not too sure will make any measurable difference when a controller BBWC is installed (and particularly with SAS disks).

As a point of interested, I performed a clean shut-down on my test ESXi box (which has 512MB BBWC on a Dell Perc 5i), turned it off and disconnected the battery for a few moments. The result was that the volume was corrupted and I had to restore all my VMs from a backup - I would conclude that enabling write-caching anywhere without battery backup is pretty brave.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Paul Mead — Sun, 19 Jul 2009 21:25:57 GMT

I have recently rolled out an HP ML330 G6 server with P410 zero cache controller and 3x250Gb SATA near-line disks; am running a virtualised version of the old client server in the main VM (only VM at present) and this was achieved by using the VMware converter - all went reasonably well. The disk arrangement is two of the disks are mirrored with the 3rd as a hot spare.

First half morning of semi-live tests indicate that there is a performance issue - which I relatively quickly had down as a write cache issue. Used Windows Performance monitor (disk queue length>50 for much of the time) and IOMeter showing disk throughput lower than I would expect.

I have just discovered that HP ship all their disks with the onboard DISK write cache switched off (I am guessing that this is just in case you are using zero Mb controller with no battery backed cache and maybe you do not run a server with a UPS either); I guess I must have missed the great big NEON sign that said - "for customer safety we have ensured that your write performance will be abysmal - just in case you are silly enough to run without safety nets").

I have not been able to go back to check that switching the ondisk caching back on helps (4am site visit tomorrow am will give us the results - argh). I hope the above diatribe and debrief helps anyone else.

Of course - I am wondering if people are running their server with decent levels of battery backed cache, but that may still have the default ondisk cache off - I guess they might never realise that the ondisk cache is OFF and could be enjoying even better performance - what a waster that would be.

Makes you wonder why anyone works in IT - when the default settings make an expensive SAS raid controller running mirrored disks slower than a very slow thing indeed.

I will report back late late tomorrow or perhaps even Tuesday when my brain returns from its hiding place. Any feedback welcomed.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Rabie — Thu, 16 Jul 2009 19:35:26 GMT

Just a note, we have an open call with HP on the P410 SmartArray. Under certain specific conditions on a raid 5 array with 512mb cache we have managed to reliably hang up the raid controller in both Windows 2003 and RHEL 5.

On Windows using IO meter we have seen brilliant performance when just read or just write IO's but when using small IO with combinded read/write the performance VERY quickly degrades where after about 30minutes the disks stop responding.

I would suggest you log a call with HP.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

patparks1 — Thu, 16 Jul 2009 18:46:45 GMT

We got our BBWC in this week and got it installed. After about 24 hours, we tested write speeds and are now seeing write speeds of approx 60MB/s on our RAID5 array...which we can live with. Sure beats the pants off 4MB/s

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

patparks1 — Mon, 06 Jul 2009 18:05:06 GMT

Thanks for posting that adding the BBWC helped improve your performance. I have 3 ESXi 3.5 boxes which are HP DL360Gen5's, all with similar slowness. I've requested to my manager that we need to purchase these BBWC to improve the situation. It's just good seeing another confirmed instance that this helped resolve the issue.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

J1mbo — Fri, 03 Jul 2009 14:34:23 GMT

Might be worth retesting with a bigger block and file size. On my test environment (Perc 5i raid-5, 3x sata 500GB), 1K block size 1GB file gives 15MB/s. 64K block 1GB file gives >250MB/s (effect of 512MB write cache), 64K block 2GB file gives ~60MB/s which is more realistic. Note this is dd for Windows though.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

seocsitvmw — Fri, 03 Jul 2009 13:26:43 GMT

Hi, same slow disk performance problem. I've downgraded the firmware version to 1.58C but this is a rate...

time dd if=/dev/zero of=/tmp/test.txt bs=1k count=1000000
1000000+0 records in
1000000+0 records out

real 1m1.819s
user 0m0.300s
sys 0m2.380s

the I/O is less to 20MB/s... You can try the same test and past here the output?

Thanks in advance!

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Thu, 02 Jul 2009 12:46:03 GMT

It should have been a sequential write test. I tested write performance by observing the VI Client performance chart for disk I/O while creating a 1 GB file on a RHEL4 VM. I believe the command was "dd if=/dev/zero of=/tmp/test.txt bs=1k count=1000000".

As far as partition alignment issues are concerned, I have no idea if that's a possibility. However, the datastore creation and VM creation/installation was performed via the VI Client.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

PeteLong — Thu, 02 Jul 2009 12:28:31 GMT

Downgrading the firmware on the 410i card from 1.62 to 1.58C cured the problem for me.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Thu, 02 Jul 2009 12:26:04 GMT

No further changes were needed to anything. I could not find any way to enable write caching on the RAID controller. I assume it's an automatic setting which is enabled when the RAID controller detects the battery pack attached to the RAID cache DIMM.

The only indication that the 1GB RAID cache with the battery pack was installed (other than a speed increase in writes) was the RAID controller message displayed on the server's console during power on. I seem to recall that a message was initially displayed after installing the 1GB RAID cache with the battery pack indicated that the battery pack was not fully charged and RAID performance would not optimal at that time.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

J1mbo — Wed, 01 Jul 2009 09:00:38 GMT

> we confirmed that our 8 disk RAID 50 volume was able to write at 70 MB/s (at least as long as the RAID cache didn't get saturated)

Is this sequential performance? I ask since it still seems quite low, the P400's should be able to achive over 150MB/s write according to this: http://cnscenter.future.co.kr/resource/rsc-center2/Tolly/Tolly206162Intel-IOP333SASRAID.pdf

I wonder if there is a partition alignment issue? (http://www.vmware.com/pdf/esx3_partition_align.pdf)

I read on here a while ago that automatically created VMFS partitions are not properly aligned and should be deleted and re-created using the VI Client - can't find it now predictably.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

PeteLong — Wed, 01 Jul 2009 08:44:27 GMT

Added 512 Cache - did NOT fix the problem - HP said the Diagnostic Utility showed every drive had an error in the same sector - and that this was unlikely.

They asked that we firmware the RAID controler - it had a newer version than the current firmware CD (it was runing 1.62) Ive downgraded it to 1.58C

Just brought the server up this morning lets see how it performs..............

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

grantdavies — Wed, 01 Jul 2009 06:29:15 GMT

We are running into this problem with one of our clients at the moment, can I ask - once you installed the BBWC and it was fully charged did the write performance improve without further adminisatrative intervention ordid you have to change some settings after the 2-3 days charging? Obviously we're going to upgrade them to BBWC and I'm just wondering whether we'll need to schedule a second maintenance window to configure the write caching on the controller once the battery charges.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Tue, 16 Jun 2009 12:45:18 GMT

We received a replacement RAID cache DIMM (1 GB) with battery pack and battery pack connector cable from HP. We replaced the 512 MB non-battery backed RAID cache DIMM in our HP DL380 G6.

After waiting a weekend for the RAID cache battery pack to be fully charged, we confirmed that our 8 disk RAID 50 volume was able to write at 70+ MB/s (at least as long as the RAID cache didn't get saturated) which is a significant improvement from a non-battery backed RAID cache (~5-7 MB/s).

Furthermore, we found that having hyperthreading enabled in the BIOS increased simultaneous, multiple VM task performance (opening notepad) on 30 Windows XP VM's by 30-40% (time for all VM's to complete the task dropped from 36 sec to 23 sec).

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

PeteLong — Tue, 16 Jun 2009 11:41:04 GMT

I Have the Same Problem

ML350 G6 410i RAID Card - ESX 3.5 update4

Call Logged to VMWare

Call Logged to HP - who are not being terribly helpfull - they have asked we run Array Diagnostic Utility from Smartstart.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Fri, 15 May 2009 18:40:01 GMT

Got the RCLI installer and see the vmksfstool.pl. Thanks.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

depping — Fri, 15 May 2009 12:31:32 GMT

Just download the RCLI or VIMA. vmkfstools is included in those.

Duncan
VMware Communities User Moderator | VCP | VCDX

Blogging: http://www.yellow-bricks.com
Twitter: http://www.twitter.com/depping

If you find this information useful, please award points for "correct" or "helpful".

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Fri, 15 May 2009 11:14:25 GMT

Like fossmarkluni, we have order the HP 512MB RAID cache DIMM and battery pack. Hopefully, this will solve our disk write IO issue.

However, I would like to try prewiping all blocks to create and eagerzeroedthick VMDK. I have no experience with vmkfstools. Is it included with ESXi and accessible via an ssh session with the hypervisor? Or would additional SW/licenses be needed?

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

fossmarkluni — Fri, 15 May 2009 09:52:37 GMT

Disk write can peak up to about 10 MB/s, but after a few secs goes down to 2-3 MB/s, have just one VM on this server ATM. (2008 server std 64 bit)

What I feel is so strange is that a normal (E8500, single SATA) workstation performs much better (disk write) than this brand new ML 370 G6, even without BBWC I expected the ML 370 to perform better!!

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Anton V Zhbankov — Wed, 13 May 2009 19:15:33 GMT

Do VMs show such speed all the time? By default VMDKs are created in "thick" mode, and on first access to block ESX wipes it out, causing low IOPS.
So if you run any disk test, do not take first result. Wait it to finish and run again.

Other way to prewipe all blocks is to create eagerzeroedthick VMDKs using vmkfstools command.

---
VMware vExpert '2009
http://blog.vadmin.ru

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

fossmarkluni — Wed, 13 May 2009 16:21:27 GMT

Same here.. HP ML 370 G6 w/ 2 x E5520 and 12 gb RAM, P410i 256 mb without BBWC ESXi 3.5 U4.

2-3 MB/s write rate, which is unbelivable! Even without BBWC...

We have a HP XW4600 workstation w/ 1 x 500gb SATA for test purposes ESXi 3.5 U3, it easily writes 20 MB/s ++ with ACHI disabled (IDE mode)

BBWC battery kit ordered!

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Jackobli — Thu, 07 May 2009 10:59:35 GMT

Keith001 schrieb:
I have a hard time believeing that my RAID write IO is so slow when compared to RHEL4U3 installed natively.

Did you read my words? ESXi does not cache itself. So any write goes directly to the disk (except, if you install a BBWC).
Any given linux is using it's RAM quite excessive for caching IO, so you really cannot compare.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Thu, 07 May 2009 10:30:21 GMT

I believe it's a P410i without BBWC. It's integrated onto the system board with 256 MB of cache (HP Smart Array P410i/256MB Controller).

I have a hard time believeing that my RAID write IO is so slow when compared to RHEL4U3 installed natively.

Re: ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Jackobli — Wed, 06 May 2009 21:52:11 GMT

Hello and welcome to the VMware ESXi community forum.

Which kind of the p410 do you own? Regular setup or the extended with a battery backed write cache?
Of first, most probably the smart array adapater is not using a write cache (100% of RAM for read, 0% for write).
As ESXi itself does not cache writes, a bad write performance is obvious.

ESXi 3.5 U4 on HP DL380 G6 has slow disk performance

Keith001 — Wed, 06 May 2009 21:45:39 GMT

I've installed ESXi 3.5 U4 with and without the latest VMware patches on an HP DL380 G6 server with a p410i RAID controller configured with 8 SAS drives in a RAID 50 configuration. The server firmware (RAID controller and BIOS) have been updated to the most current versions.

Unfortunately, a RHEL4 VM only achieves a maximum of 5-6 MB/s in disk IO for write operations. When I run 2 different RHEL4 VM's simultaneously, the disk write performance is 2.5-3 MB/s each.

I'm fairly certain the HW is good because I've performed a native installation of RHEL5U3 on the server and have used that OS to do a local file copy at 250+ MB/s.

Does anyone have any knowledge of performance issues regarding the HP Smart Array p410i RAID controller and the VMware ESXi 3.5 U4 cciss driver?