VCenter Appliance Update 8.0.0-21216066 to 8.0.1-21495797 failed

[GuenterKrembsle]

Hi,

after updating from R7 to R8 i now wanted to bring my vcenter to the latest version.

Trying first in the vcenter interface if found out i need to do this from vcsa management.

Everything looked good - i got a "success message" after the update.

But - neither vcenter nor vcsa interface is usable.

VCenter is giving me this error:

[500] Beim Abrufen von Identitätsanbietern ist ein Fehler aufgetreten. Versuchen Sie es erneut. Falls das Problem weiterhin besteht, wenden Sie sich an Ihren Administrator.

VCSA management is showing me:

503 - Service Not Available

I only have access via ssh to the appliance.

Now i am stuck - how can i go on in troubleshooting and find the cause of the error?

I can see some error on boot but

I found some error in vpxd.log - for me it looks like the "transfer" of the database failed.

2023-05-27T07:04:03.022+02:00 error -[08305] [Originator@6876 sub=VMOMI-FSS] Unknown feature 'VMODL_LIFECYCLE'. ASSUMING DISABLED
2023-05-27T07:04:11.267+02:00 error vpxd[08305] [Originator@6876 sub=vpxdVdb] Database version will not be set in vpxd.cfg
2023-05-27T07:04:11.997+02:00 error vpxd[08305] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(Could not find file  : /etc/motd)
2023-05-27T07:04:12.183+02:00 error vpxd[08305] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: ads.checkInterval
2023-05-27T07:04:12.183+02:00 error vpxd[08305] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: ads.checkIntervalEnabled
2023-05-27T07:04:12.202+02:00 error vpxd[08305] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.DBPassword
2023-05-27T07:04:12.240+02:00 error vpxd[08305] [Originator@6876 sub=MoOptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: Fault cause: vmodl.fault.InvalidArgu
ment
2023-05-27T07:04:38.971+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.cluster.VsanClusterHclInfo already exists.
2023-05-27T07:04:38.971+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Managed type vim.cluster.VsanClusterHealthSystem already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VsanHclControllerInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Managed type vim.host.VsanHealthSystem already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VsanHostHclInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Managed type vim.host.VsanDiskManagementSystem already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.vsan.host.DiskMapInfoEx already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.vsan.host.VsanDiskManagementSystemCapability already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsFaultDomainInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsNodeInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsPreferredFaultDomainInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostCapability already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostInfo already exists.
2023-05-27T07:04:38.981+02:00 error vpxd[08305] [Originator@6876 sub=Vsan] Managed type vim.host.VsanStretchedClusterSystem already exists.
2023-05-27T07:04:48.423+02:00 error vpxd[08305] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: Could not find file  : /var/lib/vmware/hpMetadataCache.xml
2023-05-27T07:05:00.171+02:00 error vpxd[08305] [Originator@6876 sub=EventTypes] Failed to load event type <EventType>
2023-05-27T07:05:00.509+02:00 error vpxd[08305] [Originator@6876 sub=EventTypes] Failed to load event type <EventType>
2023-05-27T07:05:04.443+02:00 error vpxd[08715] [Originator@6876 sub=Main opID=CheckCertificateExpiry-6420f902] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0, error info:
 other failure
2023-05-27T07:05:04.444+02:00 error vpxd[08715] [Originator@6876 sub=Main opID=CheckCertificateExpiry-6420f902] Unable to get certificates from the store APPLMGMT_PASSWORD
2023-05-27T07:05:18.491+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.cloud.provider.services
.plugin/public.key
2023-05-27T07:05:18.492+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.492+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.492+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
2023-05-27T07:05:18.523+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.vsphere.client/public.k
ey
2023-05-27T07:05:18.523+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.523+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.523+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
2023-05-27T07:05:18.523+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.vsan.client/public.key
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.lcm.client/public.key
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.vcIntegrity/public.key
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.524+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
2023-05-27T07:05:18.525+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] init BIO error for file /etc/vmware-vpx/extensions/com.vmware.rbd/public.key
2023-05-27T07:05:18.525+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] openssl error: [VpxPublicKey::VpxPublicKey] error in BIO_read_filename()
2023-05-27T07:05:18.525+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:02001002:system library:fopen:No such file or directory
2023-05-27T07:05:18.525+02:00 error vpxd[08305] [Originator@6876 sub=vpxCrypt] [VpxPublicKey::VpxPublicKey(const string&)] ERR error:20074002:BIO routines:FILE_CTRL:system lib
​

any idea i the appliance can be repaired or better setup from scratch?

[a_p_]

Some things you may want to verify:

1. DNS forward AND reverse resolution works for the vCSA's FQDN and IP address
2. The cofigured DNS servers are available
3. none of the VCSA's SSL certificates have expired

André

[GuenterKrembsle]

Hello Andre,

as i had a working vcsa before doing the update - why should DNS config change or certificate "suddenly" expire?

The update from 7 -> 8 went fine and with 8.0.0-x the appliance was runing for some days?

But - i will give it a try.

Thx.