I performed a clean install of ESXi 6 with provided HP image (ESXi-6.0.0-2494585-HP-600.9.1.39-Mar2015).
Access to and from datastores over network through vSphere Client (6.0.0-2502222) is very slow - ~10 Mbps ( there are several hundreds under ESXi 5.5)
I'm suspecting that ESXi 6 limits CPU resources used by host system to 230 MHz.
I can't edit "System Resource Reservation" parameters (contrary to ESXi 5.5, where I can edit them and switch to Simple or Advanced view).
Hey ArnisR,
I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.
The problem is caused by the last (HP) ISO's (from march);
- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso
- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso
I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared .
The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!
When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.
I hope this helps!
Hello, I also my Microserver G8 slow down with HP's customized imaged. I have installed HP's 5.5U2 image and after ugraded with ESXi 6 standard image with better results, but it cannot detect my DVD ROM device. Hope that helps.
David
Thanks for the answer. I'll also plan to try 5.5->6 upgrade scenario.
I haven't DVD drive. I have installed SSD drive in that place and configured through RAID controller as RAID0 (single drive).
ESXi 5.5 and 6.0 see them fine.
Arnis
Hi there, I can update the system CPU reservation in "thick" client that came with vSphere 6.0 by connecting to the standalone host with it:
"thick" client is free or paid option?
I've lost in vmw vSphere vCenter vWeb vClient vVirtual vJungles 😕
Hey ArnisR,
I finally found the problem. In the last ESXi 5.5 and the 6.0 release they have replaced the hpvsa driver (HP vSphere ISO !!). With these drivers, I have experienced major problems with the data stores and thus the B120i (RAID) controller of the HP MicroServer Gen8.
The problem is caused by the last (HP) ISO's (from march);
- VMware ESXi 5.5.0-Update2-2403361-HP-550.9.2.27 Mar2015.iso
- VMware ESXi 6.0.0-2494585-HP-600.9.1.39 Mar2015.iso
I have found out (because I had an older ISO), the ISO VMware ESXi 5.5.0-Update2-2068190-HP-5.77.3-Nov2014.iso works well! With this ISO, all performance problems disappeared .
The difference between the ISO's is the following. The older ISO contains the storage driver scsi-hpvsa 5.5.0-88OEM. ISO(s) of March come with newer storage driver scsi-hpvsa 5.5.0-92OEM or scsi hpvsa-5.5.0-90. These drivers are the cause of the problems in the B120i controller!
When installing ESXi 5.5 or 6 from the last ISOs, you must manually install the older storage driver (SCSI hpvsa-5.5.0-88). I've tested this with the vib file scsi HPSA-5.5.0-88OEM.550.0.0.1331820.x86_64.vib. I then adjusted the HP ISO and equipped with the older driver. Now I can install ESXi 5.5 or 6 from the (customized) HP ISO, which includes the older driver.
I hope this helps!
Thanks, nthorst
I made some investigation now and found that hpsva ...-90 driver is marked as hotfix and urgent. I didn't find yet what the bug was in -088, but at this moment I don't want to risk with older one.
Arnis
P.S. There is also newer HP iso for ESXi 6 (...HP-600.9.2.38-Mar2015) from 30 March, but it didn't help me also
Hi Arnis,
ESXi 6.0 iso also uses the wrong storage drivers. As I indicated before .
These are the fixes in the new releases:
5.5.0-90.0(6 Oct 2014)
5.5.0.92-1(30 Mar 2015)
possible, the older driver is stable enough to use
Ok, thanks nthorst for changelog. (BTW, where they are?)
Installed 088 driver and indeed in Ubuntu 14.04 LTS speeds goes up tenfold.
vSphere client however still doesn't work fast - ~100-150Mbps from SSD datastore to desktop SSD.
Also problem is still when I copy from one VM to another (Ubuntu<->Win10) on the same SSD. Speed often fall below 1 MB/s and even freezes for some time.
I decided to try Hyper-V. I also have large HDDs and could stuck in 2 TB limit with esxi.
Thanks again
A
Hey nthorst,
I have the same problem I think...
I upgraded my ESXI 5.5.0 to ESXI 6.0 today and my top speed is 20 MB/s sometimes lower... Got RAID 10 with 4 hard disks...
How can I downgrade the storage driver?
Can you help me maybe?
Thank you very much!
Edit:
Found my old installation ISO: VMware-ESXi-5.5.0-1331820-HP-5.71.3-Sep2013.iso
This one worked fine. But I dont know how to install only that driver package
Edit 2:
Installed now "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" from here.
ESXI 6.0 running better now! Speed collapsing to 120 MB/s (I think it should be faster , maybe some of you guys have any idea) but it runs smooth again.
How to install scsi-hpvsa-5.5.0-88 on ESXi 6 on HP microserver gen8
(if an other scsi-hpvsa-driver is installed)
1. Stop all VMs
2. Enable ssh-conection
3. Copy "scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib" to /tmp (using WinSCP)
4. Start ssh-conection (using putty)
5. chage directory to /tmp
cd /tmp
6. Copy the vib file to /var/log/vmware
cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/
7. Start maintenanceMode
esxcli system maintenanceMode set --enable true
8. Deinstall the running scsi-hpvsa driver
esxcli software vib remove -n scsi-hpvsa -f
9. Install scsi-hpvsa-5.5.0-88
esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode
10. Restart ESXi, stop maintenanceMode, disabel ssh and start VMs again
Hi nthorst
Thanks a lot for the post! Do you know if this driver issue also applies to AHCI mode?
I am using AHCI mode for 2 local SATA disks, and got about 80MB/s to 120MB/s during VM copy between these 2 local disks in the same ESXi. Is this performance normal?
ls how to install this driver
!i used mac os
Just for reference: driver version 100 in the June 2015 HP ESXi 6 image hasn't fixed the problems, version 88 as described above works like a charm.
I was a little skeptic about this hp faulty driver, so i made some test in my lab made of two identical MicroServer Gen8, same G2020T cpu, 16 MB memory and same raid-zero lun as datastore built on two WD black 1TB drives.
As testing VMs I used two clones of CentOS 7 minimal with fio 2.1.10 installed, testing to /dev/sdb1 (thick eager zeroed vmdk).
No other vms were running during test.
fio command lines used
fio --filename=/dev/sdb1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=8k7030test
fio --filename=/dev/sdb1 --direct=1 --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=16 --numjobs=16 --runtime=120 --group_reporting --name=4kseqread
host1 : scsi-hpvsa 5.5.0.100-1OEM | host2 : scsi-hpvsa 5.5.0-88OEM | |
8k7030test | read : io=358192KB, bw=2971.9 KB/s, iops=371, runt=120529msec | read : io=341192KB, bw=2832.1 KB/s, iops=354, runt=120438msec |
4kseqread | read : io=5732.3MB, bw= 48'906 KB/s, iops=12226, runt=120024msec | read : io=43512MB, bw= 371'288 KB/s, iops=92821, runt=120005msec |
The randow rw test are equivalent and i even got some more iops with driver version 100,
but the sequential read test was very bad.
After downgrading hpvsa to version 88 and rebooting, 4kseqread test now gives me : read io=39374MB, bw=335981KB/s, iops=83995, runt=120005msec.
I think I'll stick with driver version 88.
I was trying to report this problem to HP. I made a lot of effort to give all the information that HP needs to take a look.
They closed my case after I send my system config, because the harddisks and USB Stick are "not certified by HP".
<HP>
How i promised i forward the information to my colleague of our second technical Level. Unfortunately the Microserver is in not support configuration and that’s why it is not possible for us to reproduce the issue in the lab.
From the Report is to be seen that the HDD’s are not HP branded:
Slot 0,1I,0,1 WDC WD60EFRX-68M SATA WD-WX31D45RFL3U 0A82 Unknown
Slot 0,2I,0,2 WDC WD60EFRX-68M SATA WD-WX31D45RF8DC 0A82 Unknown
The firmware on the hard drives didn’t correspondent to the firmware on the controller. On the other side HP recommends to use also HP branded USB in order to use it with VMware.
HP Flash Media Kits for USB Drives
NOTE: HP qualified blank USB key for use with HP ProLiant servers that support the VMware virtualization environment and other HP value-add software. HP recommends this industry standard USB flash device for use with VMware ESXi. USB device must be installed in the internal slot of the ProLiant server for use with VMware ESXi. Refer to HP VMware Getting Started Guide for installation instructions.
HP 8GB USB Enterprise Mainstream Flash Media Drive Key Kit 737953-B21.
I am sending you the quickspecs of this Microserver with all supported components:
http://www8.hp.com/h20195/v2/GetPDF.aspx/c04128132.pdf
If you can test the server with supported configuration and you still see the slow performance please provide us new AHS report with the more information about the test so I can forward the case to Level 2 / 3.
Please do not hesitate to contact me if you have more questions.
Thank you.
Mit freundlichen Gruessen / Regards
Andrey Andreev
Response Center Engineer
Industry Standard Server Support
Customer Solution Center Sofia
</HP>
HP can of course do that, but it is plain stupid, because i am really sure the problem is not caused by WD harddisk.
Does anyone have HP certified hardware and can open a case with HP ?
Just for information: The 28 Oct 2015 ESXi 6.0 U1 image by HP still uses the v100 drivers.
Thanks a lot for this thread - I was almost going to throw out the server, but know....vrrroooommm 🙂
Br. Martin
Just thought I would add my results to this. I was having performance issues with Microserver, although apparently not as severe as some of you were having. I decided to run a basic benchmark before and after changing the drivers. Results were quite dramatic and I could tell as soon as I booted the VM that the performance had changed.
I ran Crystal Disk Mark (all default settings) and the output is as follows:
ESXi 5.5.0 U3a
Before (v100 drivers):
After (v88 drivers):
Don't really understand the significance of the numbers that Crystal Disk Produces, but the numbers seem a lot higher!
Thanks for the help all.
Just wanted to share my results with you guys.
I've startet with this setup.
v88 with ESXi 6.0 Build 4510822
Then i have updated my ESXi to the latest build and installed the latest hpvsa driver (v102)
v102 with ESXi 6.0 Build 4600944
And because of the bad results, i've switched back to the old driver (v88) with the new ESXi Build
v88 with ESXi 6.0 Build 4600944
Keep at v88 if you want to have good performance.