Error when attempting to Update certain hosts with Update Manager. Appears to be a host problem ESXi 5.1

[MrBigFeathas]

I have been on phone calls with VMware for well over 8 hours for this issue with no resolution. I am hoping someone may have some insight. I have had vCenter 5.1 U1b installed since release and have been using update manager on all the hosts with no issue up until now. Even the hosts that are having the issue previously worked fine.

When I attempt to update certain hosts I get error 15 which says to check the logs. The weird thing is these hosts let me apply certain patches to them but I believe I am being stopped at the same point when I get further in. In the esxupdate log this is where it looks like the error is:

2014-07-18T21:02:08Z esxupdate: HostImage: DEBUG: Payload prep.py from vib VMware_bootbank_esx-base_5.1.0-2.23.1483097 skipped by BootBankInstaller

2014-07-18T21:02:28Z esxupdate: BootBankInstaller.pyc: INFO: /altbootbank/boot.cfg: bootstate changed from 3 to 3

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 216, in main

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:     cmd.Run()

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esximage/HostImage.py", line 677, in Stage

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-1472666/bora/build/esx/release/vmvisor/sys/lib/python2.6/site-packages/vmware/esximage/HostImage.py", line 463, in _download_and_stage

2014-07-18T21:02:30Z esxupdate: esxupdate: ERROR: InstallationError: ('VMware_bootbank_esx-base_5.1.0-2.23.1483097', '[Errno 4] Socket Error: [Errno 104] Connection reset by peer')

So here are the things I have done...

ran the /alt/bootbank repair (KB2033564)

verified there wasn't a state.xxxx folder in the /bootbank

renamed the locker folder (KB2030665)

reinitialized the Update Manager database (KB2043170)

recreated a brand new update manager server, nothing brought over - same error

I can still patch some of the hosts which is strange. The only commonalities for the hosts that aren't patching are they have 2GB flash drives for the ESXi install, and they are running IBM customizations. Other hosts with the same IBM customizations are fine. I just recently patched an identical host 2 days ago.

I am completely lost. Any help would be much appreciated.

[Matt23220111014]

It may not be the issue, but the 2GB flash drive seems small. I've occasionally had issues with tmp files (which in your case may be in a ramdisk) during ESXi upgrades/updates/patching and their partitions being too small. I've had success setting up a persistent scratch storage to counter that or at least take a look at the free space on the different partitions.

Investigating disk space on an ESX or ESXi host (1003564) Note the section on unnecessary files perhaps its as easy as cleaning up some space after a support bundle was created

Creating a persistent scratch location for ESXi 4.x and 5.x (1033696)

If VUM isn't doing the job you could try one of the other methods to update also. But probably wise to have a few fall back options if things go wrong if you don't already.

Methods for upgrading to ESXi 5.1 (2032757)

Backing up and restoring ESXi configuration using the vSphere Command-Line Interface and vSphere Pow...

Reverting to a previous version of ESXi after a failed upgrade (1033604)

Also based on the build number the the esx-base VIB (1483097) in the above logs you are going for ESXi 5.1u2 with a vCenter 5.1u1b, while it should work perhaps the VUM version that comes with vCenter 5.1u2 may have better success. Since you are moving from patching to doing an update in this case it may be why this is starting now....

I'm taking a few wild stabs in the dark here so hopefully this is actually constructive, sometimes you just need a another set of eyes to think about the problem differently...good luck!

Edit: looking at the error again its failing at the "_download_and_stage" line.....which adds a little more weight to the issue being capacity of the partition/ramdisk which has the tmp/scratch setup.

[Matt23220111014]

maybe also check out the bootbank partition sizes....

ESXi and Lopsided Bootbanks

[MrBigFeathas]

The only thing that is strange is I just built two new ESXi servers to test. One with the IBM customization and one with the regular ISO. I can patch both of them fine right up to the latest 5.1 build. The only difference was that those installs were on hard drives not the standard 2GB IBM flash drive. I am at a loss. I also, just patched an IBM server running on the same 2GB flash storage a couple weeks ago all the way up to the latest version. The hosts having the issue let me apply one patch at a time until I got up to build 5.1.0 1472666. As soon as I try to apply this patch VMware KB: VMware ESXi 5.1, Patch ESXi510-201401202-UG: Updates ESXi 5.1 tools-light vib or the whole update 2 rollup it dies. Also, on one of the other hosts that I manually updated to 5.1 Update 2, it will fail on any patches left as well.

[Matt23220111014]

Read the KB article on scratch storage, a USB stick is consider "remote storage" and as a result the scratch/temp partition isn't created on it but rather on a RAMdisk. With A HDD if it is created locally on the disk.

My experience of the RAMdisk scratch storage is incredibly varied, I've a seen the at 120% of capacity and clearing some space is enough, other times they are virtually empty and there are all kinds of problems.

Did you try some of the things I suggested? Any results?

Keep in mind you may have more than one problem that is causing similar symptoms.

[Matt23220111014]

That patch is for the tools-lights vib, its quite large, if you were going to have storage capacity issues that is probably the one to cause the problem.

[MrBigFeathas]

I thought it was a scratch space issue as well. I didn't have any scratch space configured so I put that on one of my LUNs and the first host patched fine after that. I thought I had found the fix but the other 5 hosts that were having the issue would not patch even with the scratch configured to use the datastore. It's very weird. I ended up patching them all via the command line and that works fine but I'm sure I will have issues again when there is another new patch or when it comes time to patch one of the other servers that aren't part of this cluster. Thank you for all of your help.

[warring]

I have seen this before, after creating /tmp/scratch on the host, the problem was fixed.

[MrBigFeathas]

That's what I thought it was too but 5 of the 6 hosts still didn't work after creating the scratch partition on a datastore. Very strange.

[Matt23220111014]

No worries, did you try looking at the bootbanks and maybe updating vCenter and VUM?

[warring]

Create the scratch on the hypervisor USB partition.

[RMSPhil]

Sorry to post on an old thread but I was wondering if anyone got to the root of this issue. We have a 5.5U3 host running from mirrored SDs that is having this exact same issue. 2 identical hosts, one is fine the other has issues when we patch and try to update VMware tools.

2015-12-21T19:40:31Z esxupdate: esxupdate: ERROR: InstallationError: ('VMware_locker_tools-light_5.5.0-3.68.3029944', '[Errno 4] Socket Error: [Errno 104] Connection reset by peer')

I have a case open with VMWare and as of yet they don't know what the issue is. We also see connections die via manual SCP of a large tools ISO (Linux.iso or windows.iso). I fee like the SD cards are bad or causing a problem but they verify as find when checking the file system for bad clusters etc.

Any thoughts or ideas?

Thanks

[Matt23220111014]

Had an issue with update manager and an ESXi 5.5u3b update (from 5.5u1) over the weekend. One host was fine the other failed the update.

Was intending to use the CLI to do the update on another host so switched to that after uploading the patch bundle to the local datastore. That worked without a problem.

While playing around I noticed an "esxcli software profile validate" command. Which highlighted that I still had a few 5.1 VIBs on the host. Changing from and "esxcli software profile update" to "install" rather than an update let me remove the older VIBs.

This might be a red herring but hopefully it might give you a bit more info to go on. Listing the image profile on the host and downloading it from the patch downloads might give you a bit more to go on if there are some stray older VIBs.

VMware KB: Installing patches on an ESXi 5.x/6.x host from the command line