We had a vSphere 5.5 cluster configured for us by a 3rd-party through Dell as part of our purchase agreement. We are running vCenter Server 5.5 U2e on Windows 2012 R2 as a physical host. It does run Update Manager. In my most recent check today, Update Manager will scan a host and report patches that are needed. In this case we're at 5.5.0 Build 2456374 on our ESXi hosts. Update Manager detects 5 missing patches. After I put the host in maintenance mode, I remediate, select the patch group, and proceed through the screens. It starts its process and goes from 0% to 16% to 25% then sits there for about 1–3 minutes, then goes to 90%, 91%, and errors out saying:
"Cannot download software packages from patch source. Check the events and the Update Manager log for download details."
Multiple strange things:
- This server was updating just fine a few months back when some previous patches were released.
- It can see the internet just fine. No firewall issues.
- Our failover site on the same network, but in a different location in the city is able to update just fine. Same hardware, network, and software versions.
Any ideas on where to look or what to try? I had gone ahead and update from vCenter Server 5.5 U2b to U2e (along with a VUM reinstall) hoping that would fix it, but it did not. Any help would be greatly appreciated, because we're not VMware experts and thought everything was set up correctly by the vendor. We were given a small amount of knowledge transfer, but nothing that would allow us to troubleshoot this.
The Update Manager config listed in vCenter Server points to just three sources, that look to be the default from hostupdate.vmware.com and vapp-updates.vmware.com. None of the settings have been changed from the initial install and configuration that were working at one point in the last few months (when I last patched the hosts to 2456374).
Looking through the vmware-vum-server-log4cpp.log I see these failures:
[2015-05-06 16:17:33:039 'JobDispatcher' 6068 DEBUG] [JobDispatcher, 405] The number of tasks: 11
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 545] Url: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es..., Download destination: C:/ProgramData/VMware/VMware Update Manager/Data/hostupdate/vmw/vib20/esx-base/VMware_bootbank_esx-base_5.5.0-2.62.2702864.vib
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 396] Inserted download job {45798736} to queue
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 416] Current download count: 0
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 432] Current download count: 1
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 438] Adding download job {45798736} to scheduler
[2015-05-06 16:17:38:642 'DownloadMgr' 2448 DEBUG] [downloadMgr, 458] Waiting for download job {45798736} to finish...
[2015-05-06 16:17:38:643 'DownloadMgr' 6364 INFO] [downloadMgr, 593] Executing download job {45798736}, url=https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...
[2015-05-06 16:17:38:644 'httpDownload' 6364 INFO] [httpDownload, 556] Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...
[2015-05-06 16:17:39:501 'httpDownload' 6364 INFO] [httpDownload, 791] Status code: 404
[2015-05-06 16:17:39:501 'httpDownload' 6364 ERROR] [httpDownload, 827] Error retrieving document 'https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...'. Status code: 404
[2015-05-06 16:17:39:502 'httpDownload' 6364 WARN] [httpDownload, 420] Download https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es... failed: Error retrieving document 'https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...'. Status code: 404
[2015-05-06 16:17:39:502 'httpDownload' 6364 ERROR] [httpDownload, 435] Reached retry download limit
[2015-05-06 16:17:39:503 'DownloadMgr' 6364 ERROR] [downloadMgr, 618] Executing download job {45798736} throws error: Reached retry download limit
[2015-05-06 16:17:39:503 'DownloadMgr' 6364 ERROR] [downloadMgr, 705] Download failed for url: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...
[2015-05-06 16:17:39:503 'DownloadMgr' 6364 INFO] [downloadMgr, 567] Download job {45798736} finished, bytes downloaded = 0
[2015-05-06 16:17:39:503 'DownloadMgr' 2448 DEBUG] [downloadMgr, 464] Download job {45798736} finished
[2015-05-06 16:17:39:503 'DownloadMgr' 2448 DEBUG] [downloadMgr, 477] Removing download job {45798736} in queue
[2015-05-06 16:17:39:503 'DownloadMgr' 2448 DEBUG] [downloadMgr, 496] Current download count: 0
[2015-05-06 16:17:39:504 'HostUpdateDepotManager' 2448 ERROR] [vibDownloaderImpl, 402] Error downloading vib file from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es... Error:downloading file: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es... failed, 0 bytes downloaded.
[2015-05-06 16:17:39:504 'HostUpdateDepotManager' 2448 ERROR] [vibDownloaderImpl, 418] Downloading file failed for: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...
[2015-05-06 16:17:39:504 'HostUpdateDepotManager' 2448 WARN] [vibDownloaderImpl, 440] Failed to download 1 VIBs:
https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...,
[2015-05-06 16:17:39:504 'HostUpdateDepotManager' 2448 ERROR] [patchDepotManager, 466] Failed to download the vibs: https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es...
[2015-05-06 16:17:39:504 'HostUpdateDepotManager' 2448 ERROR] [installController, 349] Install on host: <HOST FQDN REMOVED>(Entity: host-16) failed. Error: Failed to download Vibs
[2015-05-06 16:17:39:548 'SingleHostRemediateTask.SingleHostRemediateTask{29}' 2448 ERROR] [singleHostRemediateTask, 492] Remediation error: Failed to download Vibs
[2015-05-06 16:17:39:571 'VciTaskBase.VciClusterJobDispatcherTask{25}' 2448 INFO] [vciClusterJobSchedulerTask, 613] Remediation failed due to non mmode failure
