Failing to update from 7.0u2 to 7.0u3

[jmbraben2]

I am trying to update my HPE ML30 gen 10 with

esxcli software profile update --dry-run --depot=/vmfs/volumes/esxi_backups/iso/VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022-depot.zip --profile=HPE-Custom-AddOn_703.0.0.10.8.1-3

I am now in a situation where system boots to 7.0u2, but I cannot update it (apparently because altbootbank in bad state).  Trying to .iso update yields this (and no good suggestions there)

Here is what happened (see attached):

• esxcli software profile update --dry-run --depot=/vmfs/volumes/esxi_backups/iso/VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022-depot.zip --profile=HPE-Custom-AddOn_703.0.0.10.8.1-3
  • Complained about a community vib
• esxcli software acceptance set --level=CommunitySupported
• esxcli software profile update --dry-run --depot=/vmfs/volumes/esxi_backups/iso/VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022-depot.zip --profile=HPE-Custom-AddOn_703.0.0.10.8.1-3
  
  • No complaint about community vib
• esxcli software profile update --depot=/vmfs/volumes/esxi_backups/iso/VMware-ESXi-7.0.3-19193900-HPE-703.0.0.10.8.1.3-Jan2022-depot.zip --profile=HPE-Custom-AddOn_703.0.0.10.8.1-3
  
  • VIB virtuallyGhetto_bootbank_ghettoVCB_1.0.0-0.0.0's acceptance level is community, which is not compliant with the ImageProfile acceptance level partner
• esxcli software vib remove --vibname=ghettoVCB
  [InstallationError]
  Failed to copy files from /tmp/stagebootbank to /altbootbank: [Errno 116] Stale file handle: '/altbootbank/preupgrade-state.tgz'
  cause = [Errno 116] Stale file handle: '/altbootbank/preupgrade-state.tgz'
  • appears altbootbank in bad state and now stuck no matter what i do.
  • In retrospect I should have removed the community VIB prior...but I should not be sitting with a non-upgradeable system now.

I would think there should be some way to abandon the upgrade in process (reset the /altbootbank). But I cannot find it.

TIA

[baijup]

@jmbraben2 As per the log snippets you have shared, looks like you are hitting the SD card issue described in KB https://kb.vmware.com/s/article/83376 and that could be the reason for "Stale file handle" error message while removing the VIB.

Pasting some snippets from the uploaded logs "HPML30_esxi_fail.txt"

2022-02-06T19:00:22.999Z cpu0:524394)WARNING: FS3: 608: VMFS volume LOCKER-61ce33ef-4f05bfe8-2e94-90e2ba35c3a0/61ce33ef-4f05bfe8-2e94-90e2ba35c3a0 on mpx.vmhba32:C0:T0:L0:7 has been detected corrupted
2022-02-06T19:00:34.698Z cpu1:524394)WARNING: FS3: 608: VMFS volume LOCKER-61ce33ef-4f05bfe8-2e94-90e2ba35c3a0/61ce33ef-4f05bfe8-2e94-90e2ba35c3a0 on mpx.vmhba32:C0:T0:L0:7 has been detected corrupted
2022-02-06T19:01:08.205Z cpu0:524394)WARNING: FS3: 608: VMFS volume LOCKER-61ce33ef-4f05bfe8-2e94-90e2ba35c3a0/61ce33ef-4f05bfe8-2e94-90e2ba35c3a0 on mpx.vmhba32:C0:T0:L0:7 has been detected corrupted

[jmbraben2]

Thanks...

I read the link, how do I know if I am currently using 7.0u2 (a/b/c/x/y/z?) does the web ui show it?

I had just recently updated with VMware-ESXi-7.0.2-17867351-HPE-702.0.0.10.7.0.52-May2021.iso (latest HPE image one month ago).
I had to re-image to new device has 7.0.1 destroyed my prior boot USB stick...I thought this above version mitigated these USB problems.

In any case...is there a way to recover from this state I am in (not seeing any recovery in the link)? Or do I need to re-image again?

[baijup]

To check the build number, you may use the command 'vmware -vl' from the SSH session or check it from the Web UI and compare the build number with KB https://kb.vmware.com/s/article/2143832. Build number 17867351 is 7.0 U2a and the fix for SD Card is in 7.0 U2c (build 18426014) onwards.

In this state, you may try the workaround steps mentioned in KB 83376 (change ToolsRamdisk configuration and reboot the host). If that does not help, can plan for re-image.

[jmbraben2]

Thanks again...is an SSD via usb->sata bridge supported as valid boot option? Is that a better option?

[baijup]

Can you share a sample device name or hyperlink for the SSD via USB->SATA bridge, so that I can do some research to answer your questions. Following are two VMware KBs available on the recommendation:

https://kb.vmware.com/s/article/2145210 - vSphere SSD and Flash Device Support

https://kb.vmware.com/s/article/85685 - Removal of SD card/USB as a standalone boot device option

[jmbraben2]

Looking at the TBW of that link...I don't believe I have ever seen those number for any SSD (even 1000 TBW for a 1Tb seems optimistic)

Regular Datastore

3650 TBW or better1

I understand the boot device requirements not as high.

Anyhow...I'd be looking at random bridge such as Startech USB3S2SAT3CB and Samsung 250Gb 870 evo.

[sramanuja]

In short, we don't support USB devices even if they can provide the endurance because

• They don't advertise SMART health measurements so we can't detect issues
• USB controllers and bus are not built for high resiliency and always on connectivity, we've seen too many cases of device detaches and the drivers need to perform reconnects. 

[jmbraben2]

Finally got around to trying KB 83376

Tried twice, the setting is not sticking...guess I get to reimage 😞

# esxcli system settings advanced set -o /UserVars/ToolsRamdisk -i 1

# esxcli system settings advanced list --option /UserVars/ToolsRamdisk
Path: /UserVars/ToolsRamdisk
Type: integer
Int Value: 1
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Use VMware Tools repository from /tools ramdisk.

each time after the reboot  /UserVars/ToolsRamdisk=0 

esxcli system settings advanced list -o /UserVars/ToolsRamdisk
Path: /UserVars/ToolsRamdisk
Type: integer
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Use VMware Tools repository from /tools ramdisk.