VMware Cloud Community
ArnisR
Enthusiast
Enthusiast
Jump to solution

Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

I performed a clean install of ESXi 6 with provided HP image (ESXi-6.0.0-2494585-HP-600.9.1.39-Mar2015).
Access to and from datastores over network through vSphere Client (6.0.0-2502222) is very slow - ~10 Mbps (  there are several hundreds under ESXi 5.5)
I'm suspecting that ESXi 6 limits CPU resources used by host system to 230 MHz.
I can't edit "System Resource Reservation" parameters (contrary to ESXi 5.5, where I can edit them and switch to Simple or Advanced view).

Tags (2)
1 Solution

Accepted Solutions
nthorst
Contributor
Contributor
Jump to solution

Hey ArnisR,

I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.

The problem is caused by the last (HP) ISO's (from march);

- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso

- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso

I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared Smiley Happy.

The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!

When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.

I hope this helps!

View solution in original post

29 Replies
kubasedeep
Contributor
Contributor
Jump to solution

Hello, I also my Microserver G8 slow down with HP's customized imaged. I have installed HP's 5.5U2 image and after ugraded with ESXi 6 standard image with better results, but it cannot detect my DVD ROM device. Hope that helps.

David

Reply
0 Kudos
ArnisR
Enthusiast
Enthusiast
Jump to solution

Thanks for the answer. I'll also plan to try 5.5->6 upgrade scenario.
I  haven't DVD drive. I have installed SSD drive in that place and configured through RAID controller as RAID0 (single drive).
ESXi 5.5 and 6.0 see them fine.

Arnis

Reply
0 Kudos
Alistar
Expert
Expert
Jump to solution

Hi there, I can update the system CPU reservation in "thick" client that came with vSphere 6.0 by connecting to the standalone host with it:

Stop by my blog if you'd like 🙂 I dabble in vSphere troubleshooting, PowerCLI scripting and NetApp storage - and I share my journeys at http://vmxp.wordpress.com/
ArnisR
Enthusiast
Enthusiast
Jump to solution

"thick" client is free or paid option?
I've lost in vmw vSphere vCenter vWeb vClient vVirtual vJungles 😕

Reply
0 Kudos
nthorst
Contributor
Contributor
Jump to solution

Hey ArnisR,

I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.

The problem is caused by the last (HP) ISO's (from march);

- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso

- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso

I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared Smiley Happy.

The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!

When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.

I hope this helps!

ArnisR
Enthusiast
Enthusiast
Jump to solution

Thanks, nthorst

I made some investigation now and found that hpsva ...-90 driver is marked as hotfix and urgent. I didn't find yet what the bug was in -088, but at this moment I don't want to risk with older one.

Arnis

P.S. There is also newer HP iso for ESXi 6 (...HP-600.9.2.38-Mar2015) from 30 March, but it didn't help me also

Reply
0 Kudos
nthorst
Contributor
Contributor
Jump to solution

Hi Arnis,

ESXi 6.0 iso also uses the wrong storage drivers. As I indicated before Smiley Wink.

These are the fixes in the new releases:

5.5.0-90.0(6 Oct 2014)

  • Fixes an issue where HDD logical volumes were not being detected at installation.
  • Fixes an issue where an updated driver was not persistent after a reboot. Previously, after updating the driver and rebooting, the system would revert to the originally installed driver.

5.5.0.92-1(30 Mar 2015)

  • Spin lock update to fix a potential boot hang.

possible, the older driver is stable enough to use Smiley Happy

ArnisR
Enthusiast
Enthusiast
Jump to solution

Ok, thanks nthorst for changelog. (BTW, where they are?)

Installed 088 driver and indeed in Ubuntu 14.04 LTS speeds goes up tenfold.
vSphere client however still doesn't work fast - ~100-150Mbps from SSD datastore to desktop SSD.

Also problem is still when I copy from one VM to another (Ubuntu<->Win10) on the same SSD. Speed often fall below 1 MB/s and even freezes for some time.

I decided to try Hyper-V. I also have large HDDs and could stuck in 2 TB limit with esxi.

Thanks again

A

Reply
0 Kudos
flippyg
Contributor
Contributor
Jump to solution

Hey nthorst,

I have the same problem I think...

I upgraded my ESXI 5.5.0 to ESXI 6.0 today and my top speed is 20 MB/s sometimes lower... Got RAID 10 with 4 hard disks...

How can I downgrade the storage driver?

Can you help me maybe?

Thank you very much!

Edit:

Found my old installation ISO: VMware-ESXi-5.5.0-1331820-HP-5.71.3-Sep2013.iso

This one worked fine. But I dont know how to install only that driver package Smiley Sad

Edit 2:

Installed now "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" from here.

ESXI 6.0 running better now! Speed collapsing to 120 MB/s (I think it should be faster Smiley Sad, maybe some of you guys have any idea) but it runs smooth again.

username2015abc
Contributor
Contributor
Jump to solution

How to install scsi-hpvsa-5.5.0-88 on ESXi 6 on HP microserver gen8

(if an other  scsi-hpvsa-driver is installed)

1. Stop all VMs

2. Enable ssh-conection

3. Copy "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" to /tmp  (using WinSCP)

4. Start ssh-conection (using putty)

5. chage directory to /tmp

cd /tmp

6. Copy the vib file to /var/log/vmware

cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/

7. Start maintenanceMode

esxcli system maintenanceMode set --enable true

8. Deinstall the running  scsi-hpvsa driver

esxcli software vib remove -n scsi-hpvsa -f

9. Install scsi-hpvsa-5.5.0-88

esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode

10. Restart ESXi, stop maintenanceMode, disabel ssh and start VMs again

yuanlinios
Contributor
Contributor
Jump to solution

Hi nthorst

Thanks a lot for the post! Do you know if this driver issue also applies to AHCI mode?

I am using AHCI mode for 2 local SATA disks, and got about 80MB/s to 120MB/s during VM copy between these 2 local disks in the same ESXi. Is this performance normal?

VCIX6-DCV/NV, RHCA, CCIE www.linkedin.com/in/yuanlinios
Reply
0 Kudos
speedunlock
Contributor
Contributor
Jump to solution

ls how to install this driver

!i used mac os

Reply
0 Kudos
ijdod
Contributor
Contributor
Jump to solution

Just for reference: driver version 100 in the June 2015 HP ESXi 6 image hasn't fixed the problems, version 88 as described above works like a charm.

Reply
0 Kudos
DaeronBox
Contributor
Contributor
Jump to solution

I was a little skeptic about this hp faulty driver, so i made some test in my lab made of two identical MicroServer Gen8, same G2020T cpu, 16 MB memory and same raid-zero lun as datastore built on two WD black 1TB drives.

As testing VMs I used two clones of CentOS 7 minimal with fio 2.1.10 installed, testing to /dev/sdb1 (thick eager zeroed vmdk).

No other vms were running during test.

fio command lines used

fio --filename=/dev/sdb1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=8k7030test

fio --filename=/dev/sdb1 --direct=1 --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=4kseqread

host1 : scsi-hpvsa 5.5.0.100-1OEM

host2 : scsi-hpvsa 5.5.0-88OEM

8k7030test

read : io=358192KB, bw=2971.9 KB/s, iops=371, runt=120529msec
write: io=153376KB, bw=1272.6 KB/s,
iops=159, runt=120529msec

read : io=341192KB, bw=2832.1 KB/s, iops=354, runt=120438msec
write: io=145792KB, bw=1210.6 KB/s, iops=151, runt=120438msec

4kseqread

read : io=5732.3MB, bw= 48'906 KB/s, iops=12226, runt=120024msec

read : io=43512MB, bw= 371'288 KB/s, iops=92821, runt=120005msec

The randow rw test are equivalent and i even got some more iops with driver version 100,

but the sequential read test was very bad.

After downgrading hpvsa to version 88 and rebooting, 4kseqread test now gives me : read io=39374MB, bw=335981KB/s, iops=83995, runt=120005msec.

I think I'll stick with driver version 88.

Reply
0 Kudos
Baltasar
Contributor
Contributor
Jump to solution

I was trying to report this problem to HP. I made a lot of effort to give all the information that HP needs to take a look.

They closed my case after I send my system config, because the harddisks and USB Stick are "not certified by HP".


<HP>

How i promised i forward the information to my colleague of our second technical Level.  Unfortunately the Microserver is in not support configuration and that’s why it is not possible for us to reproduce the issue in the lab.

From the Report is to be seen that the HDD’s are not HP branded:

Slot 0,1I,0,1       WDC WD60EFRX-68M  SATA  WD-WX31D45RFL3U         0A82      Unknown

Slot 0,2I,0,2       WDC WD60EFRX-68M  SATA  WD-WX31D45RF8DC         0A82      Unknown

The firmware on the hard drives didn’t correspondent to the firmware on the controller. On the other side HP recommends to use also HP branded USB in order to use it with VMware.

HP Flash Media Kits for USB Drives

NOTE: HP qualified blank USB key for use with HP ProLiant servers that support the VMware virtualization environment and other HP value-add software. HP recommends this industry standard USB flash device for use with VMware ESXi. USB device must be installed in the internal slot of the ProLiant server for use with VMware ESXi. Refer to HP VMware Getting Started Guide for installation instructions.

HP 8GB USB Enterprise Mainstream Flash Media Drive Key Kit 737953-B21.

I am sending you the quickspecs of this Microserver with all supported components:

http://www8.hp.com/h20195/v2/GetPDF.aspx/c04128132.pdf

If you can test the server with supported configuration and you still see the slow performance please provide us new AHS report with the more information about the test so I can forward the case to Level 2 / 3.

Please do not hesitate to contact me if you have more questions.

Thank you.

Mit freundlichen Gruessen / Regards

Andrey Andreev

Response Center Engineer

Industry Standard Server Support

Customer Solution Center Sofia

</HP>

HP can of course do that, but it is plain stupid, because i am really sure the problem is not caused by WD harddisk.

Does anyone have HP certified hardware and can open a case with HP ?

Reply
0 Kudos
ijdod
Contributor
Contributor
Jump to solution

Just for information: The 28 Oct 2015 ESXi 6.0 U1 image by HP still uses the v100 drivers.

Reply
0 Kudos
Twister5800
Contributor
Contributor
Jump to solution

Thanks a lot for this thread - I was almost going to throw out the server, but know....vrrroooommm 🙂

Br. Martin

Best regards Martin 🙂
Reply
0 Kudos
pauliem
Contributor
Contributor
Jump to solution

Just thought I would add my results to this. I was having performance issues with Microserver, although apparently not as severe as some of you were having. I decided to run a basic benchmark before and after changing the drivers. Results were quite dramatic and I could tell as soon as I booted the VM that the performance had changed.

I ran Crystal Disk Mark (all default settings) and the output is as follows:

ESXi 5.5.0 U3a

Before (v100 drivers):

Before.PNG


After (v88 drivers):

After.PNG

Don't really understand the significance of the numbers that Crystal Disk Produces, but the numbers seem a lot higher!

Thanks for the help all.

Reply
0 Kudos
YourAndroid
Contributor
Contributor
Jump to solution

Just wanted to share my results with you guys.

I've startet with this setup.

v88 with ESXi 6.0 Build 4510822

v88.PNG

Then i have updated my ESXi to the latest build and installed the latest hpvsa driver (v102)

v102 with ESXi 6.0 Build 4600944

v102.PNG

And because of the bad results, i've switched back to the old driver (v88) with the new ESXi Build

v88 with ESXi 6.0 Build 4600944

v88 Updated.PNG

Keep at v88 if you want to have good performance.