topic Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client. in ESXi Discussions

Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ArnisR — Mon, 16 Mar 2015 21:11:11 GMT

I performed a clean install of ESXi 6 with provided HP image (ESXi-6.0.0-2494585-HP-600.9.1.39-Mar2015).
Access to and from datastores over network through vSphere Client (6.0.0-2502222) is very slow - ~10 Mbps ( there are several hundreds under ESXi 5.5)
I'm suspecting that ESXi 6 limits CPU resources used by host system to 230 MHz.
I can't edit "System Resource Reservation" parameters (contrary to ESXi 5.5, where I can edit them and switch to Simple or Advanced view).

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

kubasedeep — Wed, 18 Mar 2015 08:24:54 GMT

Hello, I also my Microserver G8 slow down with HP's customized imaged. I have installed HP's 5.5U2 image and after ugraded with ESXi 6 standard image with better results, but it cannot detect my DVD ROM device. Hope that helps.

David

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ArnisR — Wed, 18 Mar 2015 08:41:52 GMT

Thanks for the answer. I'll also plan to try 5.5->6 upgrade scenario.
I haven't DVD drive. I have installed SSD drive in that place and configured through RAID controller as RAID0 (single drive).
ESXi 5.5 and 6.0 see them fine.

Arnis

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

Alistar — Wed, 18 Mar 2015 13:21:45 GMT

Hi there, I can update the system CPU reservation in "thick" client that came with vSphere 6.0 by connecting to the standalone host with it:

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ArnisR — Sat, 21 Mar 2015 21:11:55 GMT

"thick" client is free or paid option?
I've lost in vmw vSphere vCenter vWeb vClient vVirtual vJungles

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

nthorst — Sun, 12 Apr 2015 21:10:28 GMT

Hey ArnisR,

I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.

The problem is caused by the last (HP) ISO's (from march);

- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso

- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso

I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared .

The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!

When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.

I hope this helps!

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ArnisR — Sun, 12 Apr 2015 22:04:39 GMT

Thanks, nthorst

I made some investigation now and found that hpsva ...-90 driver is marked as hotfix and urgent. I didn't find yet what the bug was in -088, but at this moment I don't want to risk with older one.

Arnis

P.S. There is also newer HP iso for ESXi 6 (...HP-600.9.2.38-Mar2015) from 30 March, but it didn't help me also

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

nthorst — Mon, 13 Apr 2015 08:10:43 GMT

Hi Arnis,

ESXi 6.0 iso also uses the wrong storage drivers. As I indicated before .

These are the fixes in the new releases:

5.5.0-90.0(6 Oct 2014)

Fixes an issue where HDD logical volumes were not being detected at installation.
Fixes an issue where an updated driver was not persistent after a reboot. Previously, after updating the driver and rebooting, the system would revert to the originally installed driver.

5.5.0.92-1(30 Mar 2015)

Spin lock update to fix a potential boot hang.

possible, the older driver is stable enough to use

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ArnisR — Mon, 13 Apr 2015 16:15:08 GMT

Ok, thanks nthorst for changelog. (BTW, where they are?)

Installed 088 driver and indeed in Ubuntu 14.04 LTS speeds goes up tenfold.
vSphere client however still doesn't work fast - ~100-150Mbps from SSD datastore to desktop SSD.

Also problem is still when I copy from one VM to another (Ubuntu<->Win10) on the same SSD. Speed often fall below 1 MB/s and even freezes for some time.

I decided to try Hyper-V. I also have large HDDs and could stuck in 2 TB limit with esxi.

Thanks again

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

flippyg — Fri, 29 May 2015 03:08:21 GMT

Hey nthorst,

I have the same problem I think...

I upgraded my ESXI 5.5.0 to ESXI 6.0 today and my top speed is 20 MB/s sometimes lower... Got RAID 10 with 4 hard disks...

How can I downgrade the storage driver?

Can you help me maybe?

Thank you very much!

Edit:

Found my old installation ISO: VMware-ESXi-5.5.0-1331820-HP-5.71.3-Sep2013.iso

This one worked fine. But I dont know how to install only that driver package

Edit 2:

Installed now "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" from here.

ESXI 6.0 running better now! Speed collapsing to 120 MB/s (I think it should be faster , maybe some of you guys have any idea) but it runs smooth again.

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

username2015abc — Wed, 03 Jun 2015 08:13:29 GMT

How to install scsi-hpvsa-5.5.0-88 on ESXi 6 on HP microserver gen8

(if an other scsi-hpvsa-driver is installed)

1. Stop all VMs

2. Enable ssh-conection

3. Copy "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" to /tmp (using WinSCP)

4. Start ssh-conection (using putty)

5. chage directory to /tmp

cd /tmp

6. Copy the vib file to /var/log/vmware

cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/

7. Start maintenanceMode

esxcli system maintenanceMode set --enable true

8. Deinstall the running scsi-hpvsa driver

esxcli software vib remove -n scsi-hpvsa -f

9. Install scsi-hpvsa-5.5.0-88

esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode

10. Restart ESXi, stop maintenanceMode, disabel ssh and start VMs again

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

yuanlinios — Mon, 13 Jul 2015 02:58:48 GMT

Hi nthorst

Thanks a lot for the post! Do you know if this driver issue also applies to AHCI mode?

I am using AHCI mode for 2 local SATA disks, and got about 80MB/s to 120MB/s during VM copy between these 2 local disks in the same ESXi. Is this performance normal?

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

speedunlock — Tue, 21 Jul 2015 18:37:15 GMT

ls how to install this driver

!i used mac os

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ijdod — Mon, 12 Oct 2015 20:54:16 GMT

Just for reference: driver version 100 in the June 2015 HP ESXi 6 image hasn't fixed the problems, version 88 as described above works like a charm.

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

DaeronBox — Sun, 18 Oct 2015 12:34:56 GMT

I was a little skeptic about this hp faulty driver, so i made some test in my lab made of two identical MicroServer Gen8, same G2020T cpu, 16 MB memory and same raid-zero lun as datastore built on two WD black 1TB drives.

As testing VMs I used two clones of CentOS 7 minimal with fio 2.1.10 installed, testing to /dev/sdb1 (thick eager zeroed vmdk).

No other vms were running during test.

fio command lines used

fio --filename=/dev/sdb1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=8k7030test

fio --filename=/dev/sdb1 --direct=1 --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=4kseqread

	host1 : scsi-hpvsa 5.5.0.100-1OEM	host2 : scsi-hpvsa 5.5.0-88OEM
8k7030test	read : io=358192KB, bw=2971.9 KB/s, iops=371, runt=120529msec write: io=153376KB, bw=1272.6 KB/s, iops=159, runt=120529msec	read : io=341192KB, bw=2832.1 KB/s, iops=354, runt=120438msec write: io=145792KB, bw=1210.6 KB/s, iops=151, runt=120438msec
4kseqread	read : io=5732.3MB, bw= 48'906 KB/s, iops=12226, runt=120024msec	read : io=43512MB, bw= 371'288 KB/s, iops=92821, runt=120005msec

The randow rw test are equivalent and i even got some more iops with driver version 100,

but the sequential read test was very bad.

After downgrading hpvsa to version 88 and rebooting, 4kseqread test now gives me : read io=39374MB, bw=335981KB/s, iops=83995, runt=120005msec.

I think I'll stick with driver version 88.

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

Baltasar — Sun, 18 Oct 2015 13:50:43 GMT

I was trying to report this problem to HP. I made a lot of effort to give all the information that HP needs to take a look.

They closed my case after I send my system config, because the harddisks and USB Stick are "not certified by HP".

<HP>

How i promised i forward the information to my colleague of our second technical Level. Unfortunately the Microserver is in not support configuration and that’s why it is not possible for us to reproduce the issue in the lab.

From the Report is to be seen that the HDD’s are not HP branded:

Slot 0,1I,0,1 WDC WD60EFRX-68M SATA WD-WX31D45RFL3U 0A82 Unknown

Slot 0,2I,0,2 WDC WD60EFRX-68M SATA WD-WX31D45RF8DC 0A82 Unknown

The firmware on the hard drives didn’t correspondent to the firmware on the controller. On the other side HP recommends to use also HP branded USB in order to use it with VMware.

HP Flash Media Kits for USB Drives

NOTE: HP qualified blank USB key for use with HP ProLiant servers that support the VMware virtualization environment and other HP value-add software. HP recommends this industry standard USB flash device for use with VMware ESXi. USB device must be installed in the internal slot of the ProLiant server for use with VMware ESXi. Refer to HP VMware Getting Started Guide for installation instructions.

HP 8GB USB Enterprise Mainstream Flash Media Drive Key Kit 737953-B21.

I am sending you the quickspecs of this Microserver with all supported components:

http://www8.hp.com/h20195/v2/GetPDF.aspx/c04128132.pdf

If you can test the server with supported configuration and you still see the slow performance please provide us new AHS report with the more information about the test so I can forward the case to Level 2 / 3.

Please do not hesitate to contact me if you have more questions.

Thank you.

Mit freundlichen Gruessen / Regards

Andrey Andreev

Response Center Engineer

Industry Standard Server Support

Customer Solution Center Sofia

</HP>

HP can of course do that, but it is plain stupid, because i am really sure the problem is not caused by WD harddisk.

Does anyone have HP certified hardware and can open a case with HP ?

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

ijdod — Sat, 07 Nov 2015 15:05:07 GMT

Just for information: The 28 Oct 2015 ESXi 6.0 U1 image by HP still uses the v100 drivers.

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

Twister5800 — Thu, 25 Feb 2016 12:27:16 GMT

Thanks a lot for this thread - I was almost going to throw out the server, but know....vrrroooommm

Br. Martin

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

pauliem — Tue, 15 Mar 2016 23:00:08 GMT

Just thought I would add my results to this. I was having performance issues with Microserver, although apparently not as severe as some of you were having. I decided to run a basic benchmark before and after changing the drivers. Results were quite dramatic and I could tell as soon as I booted the VM that the performance had changed.

I ran Crystal Disk Mark (all default settings) and the output is as follows:

ESXi 5.5.0 U3a

Before (v100 drivers):

After (v88 drivers):

Don't really understand the significance of the numbers that Crystal Disk Produces, but the numbers seem a lot higher!

Thanks for the help all.

Re: Very slow acces to datastores on HP MIcroserver Gen8. Can't edit System Resource Reservation with vSphere client.

YourAndroid — Mon, 05 Dec 2016 12:45:27 GMT

Just wanted to share my results with you guys.

I've startet with this setup.

v88 with ESXi 6.0 Build 4510822

Then i have updated my ESXi to the latest build and installed the latest hpvsa driver (v102)

v102 with ESXi 6.0 Build 4600944

And because of the bad results, i've switched back to the old driver (v88) with the new ESXi Build

v88 with ESXi 6.0 Build 4600944

Keep at v88 if you want to have good performance.