Re: VAMI Update 6.7 - Infinite Loading Screen

nrinaldi · ‎09-04-2019

Hi! My name is Nicholas.

I'm having a problem with my VCSA 6.7.0.32000. I've recently updated my vCenter Appliance version from 6.5 to 6.7. I've had this problem in both versions. When I login into my VAMI and go to the Update tab, the loading screen keeps loading infinitely.

I searched for info, but I couldn't find anything. Also I checked all my services are up, but they seem okay.

Does anyone knows how to solve this?

Thanks!

Spaceballs1234 · ‎09-04-2019

Does VUM work as expected or does it also get blocked from downloading patches? is it possible a firewall or proxy server is stopping the appliance from reaching out to the internet?

nrinaldi · ‎09-04-2019

Hi. Thanks for your reply. I thought about that, but when I try to update via SSH it downloads the patches, so it isn't blocked. I also gave full permissions to my VCSA in my firewall.

msgcuisdept · ‎09-04-2019

try "tail -f /var/log/vmware/applmgmt/applemgmt.log" in cli then run the check for updates and see what errors, hopefully, come up. it will do a wget for the external repository url. the url can also be found in update in the settings button.

nrinaldi · ‎09-05-2019

Hi. Thanks for your answer.

This is the log that the tail prompts.

2019-09-05T12:06:08.788 [28455]DEBUG:vmware.appliance.update.update_functions:Running /usr/lib/applmgmt/support/scripts/manifest-verification /storage/core/software-update/tmp/latest/manifest-latest.xml 256

2019-09-05T12:06:08.859 [28455]DEBUG:vmware.appliance.update.update_b2b:Found a product version in xml manifest: VMware vCenter Server Appliance 6.7.0.40000

2019-09-05T12:06:08.860 [28455]DEBUG:vmware.appliance.update.update_functions:Error reading fss override file [Errno 2] No such file or directory: '/etc/vmware/vsphereFeatures/b2b_fss_override.json'

2019-09-05T12:06:08.860 [28455]DEBUG:vmware.appliance.update.update_functions:Returning override switch productNoVersionCheck = None

2019-09-05T12:06:08.860 [28455]DEBUG:vmware.appliance.update.update_functions:WGET: https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200...

2019-09-05T12:06:08.861 [28455]DEBUG:vmware.appliance.update.update_functions:Error reading fss override file [Errno 2] No such file or directory: '/etc/vmware/vsphereFeatures/b2b_fss_override.json'

2019-09-05T12:06:08.861 [28455]DEBUG:vmware.appliance.update.update_functions:Returning override switch updateAllowUnsecureUrl = None

2019-09-05T12:06:08.866 [28455]DEBUG:vmware.appliance.networking.proxy.proxy_impl:method=get, protocol=https

2019-09-05T12:06:08.867 [28455]DEBUG:vmware.appliance.update.update_functions:Policy not set.

2019-09-05T12:06:08.868 [28455]DEBUG:vmware.appliance.update.update_functions:Running /usr/bin/wget --no-check-certificate --connect-timeout 10 -P /storage/core/software-update/tmp/latest https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200...

2019-09-05T12:06:11.864 [28455]INFO:vmware.vherd.transport.ssh_access_collector:Feature switch: VCSA_SSH_ACCESS_EVENT disabled.

2019-09-05T12:06:13.816 [28455]DEBUG:vmware.appliance.update.update_b2b:Changing the base appliance version: True

2019-09-05T12:06:13.818 [28455]DEBUG:vmware.appliance.update.update_b2b:This patch supports versions: None

2019-09-05T12:06:13.818 [28455]DEBUG:vmware.appliance.update.update_functions:WGET: https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200...

2019-09-05T12:06:13.819 [28455]DEBUG:vmware.appliance.update.update_functions:Error reading fss override file [Errno 2] No such file or directory: '/etc/vmware/vsphereFeatures/b2b_fss_override.json'

2019-09-05T12:06:13.819 [28455]DEBUG:vmware.appliance.update.update_functions:Returning override switch updateAllowUnsecureUrl = None

2019-09-05T12:06:13.824 [28455]DEBUG:vmware.appliance.networking.proxy.proxy_impl:method=get, protocol=https

2019-09-05T12:06:13.826 [28455]DEBUG:vmware.appliance.update.update_functions:Policy not set.

2019-09-05T12:06:13.826 [28455]DEBUG:vmware.appliance.update.update_functions:Running /usr/bin/wget --no-check-certificate --connect-timeout 10 -P /storage/core/software-update/tmp/latest https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200...

I'm not sure what does "Error reading fss override file [Errno 2] No such file or directory: '/etc/vmware/vsphereFeatures/b2b_fss_override.json'" means.

Spaceballs1234 · ‎09-05-2019

I think the issue is here:

2019-09-05T12:06:08.868 [28455]DEBUG:vmware.appliance.update.update_functions:Running /usr/bin/wget --no-check-certificate --connect-timeout 10 -P /storage/core/software-update/tmp/latest https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200...

2019-09-05T12:06:13.818 [28455]DEBUG:vmware.appliance.update.update_b2b:This patch supports versions: None

It can pull the manifest fine, which I'm sure you saw, but in my environment I see:

Running /usr/bin/wget --no-check-certificate --connect-timeout 10 -P /storage/core/software-update/tmp/6.7.0.40000 https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.4000...

2019-09-05T12:56:52.733 [3535]INFO:vmware.appliance.update.update_b2b:Our version 6.7.0.40000 is not supported by this patch 6.7.0.40000

Maybe try running the command "/usr/bin/wget --no-check-certificate --connect-timeout 10 -P /storage/core/software-update/tmp/latest https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.7.0.3200..." manually and seeing what the result is.

I'm also curious what happens when it tries to wget the patch-metadata-scripts.zip at the end of the log, is that where it starts looping or does it loop?

nrinaldi · ‎09-05-2019

This is what the console prompts when I execute that command:

Saving to: ‘/storage/core/software-update/tmp/latest/rpm-manifest.json.1’

rpm-manifest.json.1 100%[=======================================================================================================>] 149.23K 69.8KB/s in 2.1s

2019-09-05 10:38:42 (69.8 KB/s) - ‘/storage/core/software-update/tmp/latest/rpm-manifest.json.1’ saved [152813/152813]

The thing is that the /etc/vmware/vsphereFeatures directory is missing and also the b2b_fss_override.json file.

And yes, after opening the Update Manager from the VAMI, that error starts looping.

I think I need to get the b2b_fss_override.json file from a working vCenter. Could you check if it does exists in your vcsa and send it to me?

Thanks!

Spaceballs1234 · ‎09-05-2019

I also do not have b2b_fss_override.json or /etc/vmware/vsphereFeatures

but when you ran the command you got a json.1? if you go /storage/core/software-update/tmp/ what files are in there? if tmp exists is there files with version numbers? also which files do you have in

/storage/core/software-update?

nrinaldi · ‎09-05-2019

This is all I've got:

root@APU [ ~ ]# ls -l /storage/core/software-update/

total 4

drwxr-xr-x 3 root root 4096 Sep 5 11:01 updates

root@APU [ ~ ]# ls -l /storage/core/software-update/updates/

total 4

drwxr-xr-x 6 root root 4096 Sep 5 11:01 6.7.0.40000

root@APU [ ~ ]# ls -l /storage/core/software-update/updates/6.7.0.40000/

total 240

drwxr-xr-x 2 root root 4096 Aug 9 23:17 eulas

drwxr-xr-x 2 root root 4096 Aug 9 23:17 headers

drwxr-xr-x 2 root root 4096 Aug 9 23:17 ignorelists

-rw-r--r-- 1 root root 146880 Sep 5 11:01 manifest-latest.xml

-rw-r--r-- 1 root root 1917 Sep 5 11:01 manifest-latest.xml.sha256

-rw-r--r-- 1 root root 74247 Sep 5 11:01 rpm-manifest.json

drwxr-xr-x 3 root root 4096 Aug 9 23:17 scripts

It's weird there isn't a temp folder...

Spaceballs1234 · ‎09-05-2019

I agree, It's odd it showed multiple jsons and the folder doesn't exist, but I don't see the folder either. it looks like vcenter can pull the updates and store them but can't validate it's own version to start the upgrade.

try looking in /var/log/vmware/applmgmt/update_microservice.log, maybe do a cat update_microservice.log |grep *last date shown for attempted upgrade* | more

I did a tail first to see the last time vcenter did an upgrade. maybe something gets blocked or something happens in the pre-check. additionally you could take a snapshot, rm * the /storage/core/software-update/updates folder, and re-attempt looking for upgrades to get a fresh look?

nrinaldi · ‎09-05-2019

root@APU [ ~ ]# tail /var/log/vmware/applmgmt/

applmgmt.log applmgmt.log.6.gz backupSchedulerCron.log statsmoitor-alarms.log StatsMonitorStartup.log.stderr

applmgmt.log.10.gz applmgmt.log.7.gz dcui.log StatsMonitor-1.log.gz StatsMonitorStartup.log.stdout

applmgmt.log.1.gz applmgmt.log.8.gz detwist.log StatsMonitor-2.log.gz vami.log

applmgmt.log.2.gz applmgmt.log.9.gz firewall-reload.log StatsMonitor-3.log.gz vami.log.1.gz

applmgmt.log.3.gz applmgmt_vmonsvc.stderr progress.log StatsMonitor-4.log.gz

applmgmt.log.4.gz applmgmt_vmonsvc.stdout size.log StatsMonitor-5.log

applmgmt.log.5.gz backup.log software-packages.log StatsMonitor.log

root@APU [ ~ ]# tail /var/log/vmware/applmgmt/^C

root@APU [ ~ ]# cat update_microservice.log |grep *last date shown for attempted upgrade* | more

cat: update_microservice.log: No such file or directory

grep: date: No such file or directory

grep: shown: No such file or directory

grep: for: No such file or directory

grep: attempted: No such file or directory

grep: upgrade*: No such file or directory

It seems I don't have the update_microservice.log either...

I did a fresh install of my vCenter two weeks ago, and copied my configs from my 6.5 to my 6.7 server.

Spaceballs1234 · ‎09-05-2019

hmm... that's odd

do you have ls /usr/lib/applmgmt/update/py/vmware/appliance/update?

__init__.py patchFilter.py task_manager.py update_b2b.py update_functions.py update_pending.py update_staged.py

ms_connector.py __pycache__ tests update_constants.py update_microservice.py update_policy.py update_state.py

nrinaldi · ‎09-05-2019

Yes, I do:

ls /usr/lib/applmgmt/update/py/vmware/appliance/update

__init__.py patchFilter.py task_manager.py update_b2b.py update_functions.py update_pending.py update_staged.py

ms_connector.py __pycache__ tests update_constants.py update_microservice.py update_policy.py update_state.py