Error on one ESXi server when scanning for updates - Page 2

eddilimp · ‎06-26-2017

Hi all,

Im getting thesere error message in the esxupdate.log

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: An unexpected exception was caught:

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: Traceback (most recent call last):

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 238, in main

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: cmd.Run()

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py", line 113, in Run

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 246, in Scan

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 106, in _generateOperationData

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: File "/build/mts/release/bora-5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/MetadataScanner.py", line 88, in _getInstallProfile

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: ERROR: AttributeError: 'NoneType' object has no attribute 'Copy'

2017-06-26T12:45:15Z esxupdate: 789827: esxupdate: DEBUG: <<<

In vCenter the error message is:

The host returns esxupdate error code:99. An

unhandled exception was encountered. Check

the Update Manager log files and esxupdate

log files for more details.

Where should I start for the error? All other ESXi Server were scanned without a Problem.

Thanks.

eddilimp · ‎06-28-2017

I get this error when executing this command:

Error in copying from /bootbank/scsi_lpf.v00 to /tmp/stagebootbank/scsi_lpf.v00: [Errno 2] No such file or directory: '/bootbank/scsi_lpf.v00'

vijayrana968 · ‎06-28-2017

On both hosts, execute cat /bootbank/boot.cfg and post results.

eddilimp · ‎06-28-2017

Problem host:

bootstate=0

kernel=tboot.b00

title=Loading VMware ESXi

kernelopt=no-auto-partition

modules=b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- net_tg3.v00 --- elxnet.v00 --- ima_be2i.v00 --- lpfc.v00 --- scsi_be2.v00 --- amshelpe.v00 --- conrep.v00 --- hpbootcf.v00 --- hpe_buil.v00 --- hpe_cru.v00 --- hpe_esxi.v00 --- hpe_ilo.v00 --- hpe_nmi.v00 --- hpe_smx_.v00 --- hponcfg.v00 --- hptestev.v00 --- scsi_hps.v00 --- ssacli.v00 --- hpssacli.v00 --- scsi_hpd.v00 --- scsi_hpv.v00 --- intelcim.v00 --- net_i40e.v00 --- net_igb.v00 --- net_ixgb.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- net_mst.v00 --- qedentv.v00 --- ima_qla4.v00 --- misc_cni.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_nx_n.v00 --- net_qlcn.v00 --- scsi_bfa.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_qla.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- misc_dri.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_vmxn.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_fni.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- uhci_usb.v00 --- vmware_f.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- nmst.v00 --- xorg.v00 --- imgdb.tgz --- state.tgz

build=6.0.0-5050593

updated=35

Working Host:

bootstate=0

kernel=tboot.b00

title=Loading VMware ESXi

kernelopt=no-auto-partition

modules=b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- net_tg3.v00 --- elxnet.v00 --- ima_be2i.v00 --- lpfc.v00 --- scsi_be2.v00 --- amshelpe.v00 --- conrep.v00 --- hpbootcf.v00 --- hpe_buil.v00 --- hpe_cru.v00 --- hpe_esxi.v00 --- hpe_ilo.v00 --- hpe_nmi.v00 --- hpe_smx_.v00 --- hponcfg.v00 --- hptestev.v00 --- scsi_hps.v00 --- ssacli.v00 --- hpssacli.v00 --- scsi_hpd.v00 --- scsi_hpv.v00 --- intelcim.v00 --- net_i40e.v00 --- net_igb.v00 --- net_ixgb.v00 --- scsi_mpt.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- net_mst.v00 --- qedentv.v00 --- ima_qla4.v00 --- misc_cni.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_nx_n.v00 --- net_qlcn.v00 --- scsi_bfa.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_qla.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- misc_dri.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_vmxn.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_fni.v00 --- scsi_ips.v00 --- scsi_lpf.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v01 --- scsi_mpt.v02 --- uhci_usb.v00 --- vmware_f.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- nmst.v00 --- xorg.v00 --- imgdb.tgz --- state.tgz

build=6.0.0-5050593

updated=32

vijayrana968 · ‎06-28-2017

I guess the easier way it to rebuilt this from scratch, but if we want to dig up for RCA then this is good

As you see, scsi_lpf.v00 is LSI SCSI mpt2sas driver which is installed on another host but not on this host. That's why we get 'No such file error'. Please execute below command on both host and paste results. Need to check the version of scsi controller because if both are same hosts with same hardware, why scsi driver is missing on this one.

esxcfg-scsidevs -l | egrep -i 'display name|vendor'

eddilimp · ‎06-28-2017

Good host:

Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

Vendor: HP iLO Model: Internal SD-CARD Revis: 2.10

Display Name: Local hp CD-ROM (mpx.vmhba33:C0:T0:L0)

Vendor: hp Model: DVD D DS8D3SH Revis: HHE8

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011010)

Vendor: PURE Model: FlashArray Revis: 8888

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011011)

Vendor: PURE Model: FlashArray Revis: 8888

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011012)

Vendor: PURE Model: FlashArray Revis: 8888

Bad host:

Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

Vendor: HP iLO Model: Internal SD-CARD Revis: 2.10

Display Name: Local hp CD-ROM (mpx.vmhba33:C0:T0:L0)

Vendor: hp Model: DVD D DS8D3SH Revis: HHE8

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011010)

Vendor: PURE Model: FlashArray Revis: 8888

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011011)

Vendor: PURE Model: FlashArray Revis: 8888

Display Name: PURE Fibre Channel Disk (naa.624a937037e06d355f6043e300011012)

Vendor: PURE Model: FlashArray Revis: 8888

vijayrana968 · ‎06-28-2017

Sorry, that was for devices. We need to check controller

esxcli storage core adapter list

eddilimp · ‎06-28-2017

Bad host:

HBA Name Driver Link State UID Capabilities Description

-------- ----------- ---------- ------------------------------------ ------------------- -----------------------------------------------------------------------------

vmhba0 hpsa link-n/a sas.50014380280180a0 (0000:02:00.0) Hewlett Packard Enterprise Smart Array P420i

vmhba1 ata_piix link-n/a sata.vmhba1 (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

vmhba2 qlnativefc link-up fc.5001438033179731:5001438033179730 Second Level Lun ID (0000:04:00.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba3 qlnativefc link-up fc.5001438033179733:5001438033179732 Second Level Lun ID (0000:04:00.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba32 usb-storage link-n/a usb.vmhba32 () USB

vmhba33 ata_piix link-n/a sata.vmhba33 (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

Good host:

HBA Name Driver Link State UID Capabilities Description

-------- ----------- ---------- ------------------------------------ ------------------- -----------------------------------------------------------------------------

vmhba0 hpsa link-n/a sas.5001438028b3d310 (0000:02:00.0) Hewlett Packard Enterprise Smart Array P420i

vmhba1 ata_piix link-n/a sata.vmhba1 (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

vmhba2 qlnativefc link-up fc.5001438033168a6d:5001438033168a6c Second Level Lun ID (0000:04:00.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba3 qlnativefc link-up fc.5001438033168a6f:5001438033168a6e Second Level Lun ID (0000:04:00.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba32 usb-storage link-n/a usb.vmhba32 () USB

vmhba33 ata_piix link-n/a sata.vmhba33 (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

vijayrana968 · ‎06-28-2017

Download this mpt2sas vib, upload to datastore and install on problematic host. This is supported up to esxi 6.5

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-LSI-MPT2SAS-151006001VMW&productI...

esxcli software vib install -d "/vmfs/volumes/Datastore/DirectoryName/PatchName.zip"

eddilimp · ‎06-28-2017

I get this error when I want to install the zip file:

[MetadataDownloadError]

Could not download from depot at zip:/vmfs/volumes/PURE_ISO/mpt2sas-15.10.06.00.1vmw-1403633.zip?index.xml, skipping (('zip:/vmfs/volumes/PURE_ISO/mpt2sas-15.10.06.00.1vmw-1403633.zip?index.xml', '', 'Error extracting index.xml from /vmfs/volumes/PURE_ISO/mpt2sas-15.10.06.00.1vmw-1403633.zip: "There is no item named \'index.xml\' in the archive"'))

url = zip:/vmfs/volumes/PURE_ISO/mpt2sas-15.10.06.00.1vmw-1403633.zip?index.xml

Please refer to the log file for more details.

eddilimp · ‎06-28-2017

Sorry, was my mistake.

I could run the command without error now and im getting this result:

Installation Result

Message: Host is not changed.

Reboot Required: false

VIBs Installed:

VIBs Removed:

VIBs Skipped: LSI_bootbank_scsi-mpt2sas_15.10.06.00.1vmw-1OEM.550.0.0.1198610

vijayrana968 · ‎06-28-2017

This is strange, how this VIB got installed on other hosts. Can you check if the HPE firmware version is same on both hosts. Any HP SPP you installed recently on other hosts.

eddilimp · ‎06-28-2017

HP SPP 2017.04 is installed on all hosts.

vijayrana968 · ‎06-28-2017

Last thing we can try >

Download ESXI 6.0 Patch 5 (5572656) manually (which you have on another working hosts but noton this single host) from below link.

https://my.vmware.com/group/vmware/patch#search

Then upload > Create baseline > and scan this particular host and see if this is showing compliant.

eddilimp · ‎06-28-2017

I imported the update und did a rescan. Update Manager say 5 patches are missing, but when I want to stage the patches im getting this error:

The host returns esxupdate error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details.

Here is the update log:

2017-06-28T13:33:41Z esxupdate: 77461: imageprofile: DEBUG: VIB VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.600.0.0.2494585 is being removed from ImageProfile (Updated) HPE-ESXi-6.0.0-Update3-iso-600.9.7.0.17

2017-06-28T13:33:41Z esxupdate: 77461: imageprofile: DEBUG: VIB VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.600.2.43.4192238 is being removed from ImageProfile (Updated) HPE-ESXi-6.0.0-Update3-iso-600.9.7.0.17

2017-06-28T13:33:41Z esxupdate: 77461: imageprofile: DEBUG: VIB VMware_bootbank_scsi-aic79xx_3.1-5vmw.600.0.0.2494585 is being removed from ImageProfile (Updated) HPE-ESXi-6.0.0-Update3-iso-600.9.7.0.17

2017-06-28T13:33:41Z esxupdate: 77461: imageprofile: DEBUG: VIB VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.600.2.52.4600944 is being removed from ImageProfile (Updated) HPE-ESXi-6.0.0-Update3-iso-600.9.7.0.17

2017-06-28T13:33:41Z esxupdate: 77461: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-06-28T13:33:41Z esxupdate: 77461: BootBankInstaller.pyc: INFO: Unrecognized value "title=Loading VMware ESXi" in boot.cfg

2017-06-28T13:33:42Z esxupdate: 77461: vmware.runcommand: INFO: runcommand called with: args = '['/usr/sbin/vsish', '-e', '-p', 'cat', '/hardware/bios/dmiInfo']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-06-28T13:33:42Z esxupdate: 77461: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/smbiosDump']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-06-28T13:33:42Z esxupdate: 77461: esxupdate: INFO: All done!

2017-06-28T13:33:42Z esxupdate: 77461: esxupdate: DEBUG: <<<

vijayrana968 · ‎06-28-2017

What about the build number, any changes ?

eddilimp · ‎06-28-2017

Yes, 5224934

vijayrana968 · ‎06-28-2017

What happen if you scan any other host against this new baseline, does it shows any pending patches !

eddilimp · ‎06-28-2017

For one host with the same build number it says 6 patches and for that broken one it says 5 patches.

vijayrana968 · ‎06-29-2017

I am not sure what we have left to troubleshoot. You should check further with VMware support I guess. 5224934 is one patch behind to the latest one, not sue how other hosts got was patched up to date.

Are you seeing the same patch VMware ESXi 6.0, Patch ESXi-6.0.0-20170601001 Update manager downloaded in its repository ?

eddilimp · ‎06-29-2017

No, I dont see that patch in the repository.