ESXi 7.0.2 Locks Up

[bearhntr]

I am hoping you can help me. I have built a home VMServer with ESXi 7.0 Update 2 and have 2 NICs and 2 VMs setup (one is Windows Server 2019 and one is FreeBSD installation of pfSense 2.5.2). Both fully installed from .ISO and licensed accordingly. Over the last 3 days - the VMHost just locks up. Going to the system where it is installed, and touching keyboard - nothing happens. No change of screen - nothing works F2 or F12 - no response. I am seeing no HDD activity at all. I press and hold the POWER button until it turns off. I then turn it back on. There are no logs in the BIOS indicating a hardware failure - so I am assuming something in the Software.

I have pulled the logs off the box, and while the hardware is not a full-blown server - it meets all the hardware specs for running VMWare ESXi 7.0.2. Again this is a basic home installation.

VMware ESXi™
Client version:
1.34.8

Client build number:
17417756

ESXi version:
7.0.2

ESXi build number:
17867351

[Srijithk]

is the host booting from usb/sd card?

Thanks,

Srijith

[bearhntr]

Thanks for responding.

Yes.  The host machine is booting from a 32GB USB 3.0 drive - that I installed ESXi to it from the CD I created.  

There are 2 HDDs installed - a SAMSUNG EVO 850 SSD (where my VMs are built) and a 500GB SATA drive (as a 2nd datastore - where my ISOs are stored).  

[Srijithk]

check the vmkwarning and vmkernel logs inside (var/run/log)

see if you are saying bootbank not found errors, you may grep for it

if you see these errors patch to 7.0 u2 c

https://kb.vmware.com/s/article/83963

Thanks,

Srijith

[bearhntr]

Srijith,

  I am already running that version.  Note the subject states 7.0.2.  

  I pulled the logs off right after I rebooted it - not sure if there is anything in them; and I would not know how to read them, anyway. 

  Not sure I can attach them here.

Curtis

[NilsS]

The ESXi build number 17867351 you mention is 7.0U2a, you need to patch with 7.0U2c; https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u2c-release-notes.html 

We were seeing the same issue and patched all hosts. In addition we moved vmtools to RAMdisk beforehand.

v7.0U2 is using the new Filesystem on bootdevice and allows much more intensive use of disk, causing issues with USB/SD

[Srijithk · Accepted solution]

bearhntr,

please refer --> https://kb.vmware.com/s/article/2143832

Reading logs can be tricky and that is the reason I asked you to grep for a specific keyword 🙂

Nevertheless, patch to 7.0 U2 C and see if it resolves your problem and do revert back to this thread if it does 

Thanks,

Srijith

[WuGeDe]

https://communities.vmware.com/t5/ESXi-Discussions/SD-Boot-issue-Solution-in-7-x/td-p/2852027

[bearhntr]

Srijith,

  I followed the steps and downloaded the patch - I am guessing that it installed correctly - as the CLI shell when I installed it told me the patch was installed correctly.  I rebooted and took HOST out of MAINT MODE, now I am seeing this.

Client version:  1.34.8

Client build number:  17417756

ESXi version:  7.0.2

ESXi build number:  18426014

  Does look like it was applied?    Do I need to do anything else?  

  From al my reading, I have to boot from USB or SD - as placing the installation on one of the HDDs in the box renders that HDD useless for placing VMs.  

Curtis

[bearhntr]

WuGeDe,

  Thanks for the link.  From reading through that - it seems that they are now recommending that ESXi 7.x no longer be launched or run form a USB flash drive or SD card?  I am a home user and have run my old VMware boxes off a durable USB drive for years.  

  I have been having lock-ups about every 4-8 hours and have to reboot the HOST.  

  I did install the patch 7.0.2c  this AM, and waiting to see if I crash again.  

  From all my previous reading - it is not recommended to install the ESXI software to the HDD or SSD where you are also going to host VMs.  Is this true?   As I am not an enterprise user - basically a home user with some VMs I use for testing and such.  The box is an older DELL OptiPlex 7010 with Core i5 CPU and 16GB of RAM with a 250GB SSD (where my VMs are located) and a 500GB HDD that I use for storage and such (like my ISOs).  

  Any suggestions on changes?  as ESXi boots off a 32GB USB Flash Drive (and not a cheapie one).  Got good marks for speed and longevity.  

Curtis

[Srijithk]

As per the build no. mentioned it looks like the patch is installed the right way, you may now monitor for a while and let us know 

you may ssh to the host and validate from CLI as well (in case you want to :))

the command would be ---> vmware -vl          (will give you the build no. and ESXi version)

[Srijithk]

All these issues arose only after the partition changes by VMware to improve I/O performance 

I assume we won't have any issues with USB/sd from 7.0u2 c and later

I would also suggest the workaround in the below KB for anyone having USB as a boot media (although I can't comment on the quality of SD cards or the endurance of boot device but still a recommended practice)

https://kb.vmware.com/s/article/83376

Thanks,

Srijith

[NilsS]

We also implemented above before the patch came out. It definitely won't do any harm.  Scratch partition and core dumps should also be configured to point to somewhere else than the bootdevice.

VMware has previously reintroduced bugs which were fixed so I wouldn't assume anything 😉

That being said, the new VMFS-L does impose more load on boot devices and therefore for new installs I would not recommend USB or SD bootdevices. Whether the SD or USB bootdevices stay healthy or corrupt earlier than before VMFS-L was introduced for bootdevices I can't judge (yet)

[bearhntr]

😁  Thank You All  👍

   I just wanted to post a thank you to everyone here who contributed.  I was able to patch the system to 7.0.2c - and move my SCRATCH and Other stuff off the USB drive I use for booting to one of my Datastores (the 500GB HDD) --  well I think it is there now.  LOL

  The system was up all day yesterday with no issues.  I am now re-evaluating doing a complete reload of this system (if I can find a 7.0.2c file to make make a CD or USB) to do this - as I wanted the system installed using UEFI - but could never getting it working.  I found this:

Host Fails to Boot After You Install ESXi in UEFI Mode 

and I am thinking I may give it a try.  Seeing that this is a test environment until it is stable - and since my VMs are on a separate drive - I can simply re-import them without losing them.  😁

Again - my thanks and appreciation for helping me fix this. 

[WuGeDe]

• Since yesterday 7xx U2d should be out.

[bearhntr]

I guess I will have to look for this one.  Would be nice if they just put an UPDATE option into the WebGUI - that worked and did these.  😁

Right now my system is OFF - as I am adding another NIC and installing a RAID controller.  Then going to see about re-installing the software using the UEFI method. 

[bearhntr]

Well.....dang -  I had no problems installing the 7.0.2c  update.....but for the life of me I cannot get the 7.0.2d to install using 

esxcli software vib update //vmfs/volumes/612cdff6-89d701f2-f8f0-90b11c806bfa/VMware-ESXi-7.0U2d-18538813-depot.zip

VM host is in MAINT MODE and SSH and CONSOLE SHELL is enabled.

[a_p_]

Unless it's a typo in your post, the command syntax isn't correct. Try

esxcli software vib update -d /vmfs/volumes/612cdff6-89d701f2-f8f0-90b11c806bfa/VMware-ESXi-7.0U2d-18538813-depot.zip

André

[bearhntr]

By Golly --  I think I got it.   I forgot to add the VIBs  folder in the path.   🙄

Client version:

1.34.8

Client build number:

17417756

ESXi version:

7.0.2

ESXi build number:

18538813