VMware Cloud Community
Kev_Johnson
Enthusiast
Enthusiast

Failed upgrade to vCenter 8u2

Running on VMUG licensing, so no support entitlement here, but I figured it was worth asking if anyone else has hit on this.

The update process (have tried both the online update as well as the full patch ISO) seems to run through pretty smoothly, and gets to the "converting data" stage, before bombing out with

"Exception occured in postInstallHook for sps:Patch"

Checking PatchRunner.log I see the below, which seems to be the breaking point

"2023-09-25T11:38:55.465Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting spbmVpUrlToIdMapProvider provider
2023-09-25T11:38:55.465Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'spbmVpUrlToIdMapProvider'.
2023-09-25T11:38:55.473Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'spbmVpUrlToIdMapProvider'.
2023-09-25T11:38:55.474Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting spbmRollupComplianceResultProvider provider
2023-09-25T11:38:55.474Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'spbmRollupComplianceResultProvider'.
2023-09-25T11:38:55.474Z [pool-3-thread-15] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 9 millis to execute.
2023-09-25T11:38:55.483Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'spbmRollupComplianceResultProvider'.
2023-09-25T11:38:55.483Z [pool-3-thread-16] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 7 millis to execute.
2023-09-25T11:38:55.483Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - Deleting fcdProfileAssociationProvider provider
2023-09-25T11:38:55.483Z [main] INFO opId= com.vmware.vim.storage.common.kv.KvManager - Deleting the provider 'fcdProfileAssociationProvider'.
2023-09-25T11:38:55.491Z [main] DEBUG opId= com.vmware.vim.storage.common.kv.KvManager - Deleted the provider 'fcdProfileAssociationProvider'.
2023-09-25T11:38:55.491Z [pool-3-thread-17] DEBUG opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 7 millis to execute.
2023-09-25T11:38:55.491Z [main] INFO opId= com.vmware.sps.upgrade.b2b.patch.action.RemoveUnusedSpbmKvProviders - [printPatchInfo] Patch SPBM unused provider cleanup patch support executed successfully to patch to VERSION11
2023-09-25T11:38:55.491Z [main] INFO opId= com.vmware.vim.storage.common.serviceclient.ls.LsServiceEndpointUpdater - Fetching the SPS service info from the lookup service.
2023-09-25T11:38:55.496Z [main] DEBUG opId= com.vmware.vim.storage.common.serviceclient.vmomi.AbstractVmomiClient - Creating stub for com.vmware.vim.binding.lookup.ServiceRegistration
2023-09-25T11:38:55.503Z [main] INFO opId= com.vmware.vim.storage.common.util.OperationIdUtil - New operationID: 27ba4ad7-f703-4a41-aa22-0e7a01cfc4b1 set in request context
2023-09-25T11:38:55.527Z [main] ERROR opId= com.vmware.vim.storage.common.serviceclient.ls.impl.ExternalServiceRegistryImpl - LS returned null or empty result, for filter={product = com.vmware.vim.sms, type = sms, nodeId = d66e54e0-c05c-4981-a931-46d2567a9cb6}
2023-09-25T11:38:55.528Z [main] ERROR opId= com.vmware.vim.storage.common.serviceclient.ls.LsServiceEndpointUpdater - Could not find the SPS service info.
com.vmware.vim.storage.common.serviceclient.ls.LsException: Could not find SPS info"

 

This led me to https://kb.vmware.com/s/article/78900, but unfortunately none of the VASA providers are showing a statusFault = null, so no dice there.


Any thoughts before I just redeploy?

Reply
0 Kudos
7 Replies
lamw
Community Manager
Community Manager

Hey Kev - Can you provide support bundle from the appliance?

Reply
0 Kudos
Ajay1988
Expert
Expert

By any chance sps service was down before upgrade ? I feel like you have sps registration missing. But logs could give more insight. 
From what version are you upgrading ?

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk > /var/core/services.txt
and share the services.txt file 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
Kev_Johnson
Enthusiast
Enthusiast

I don't believe the service was down, and there are references further up the PatchRunner.log file (I'll attach here so you can see what I mean) that imply that it was checked and fine, and then the update deleted some references. 

The VCSA has been reverted to snapshot before the upgrade, but I've attached the services.txt from the pre-upgrade appliance in case this is helpful. I can happily rerun the upgrade and capture this after the upgrade too, just let me know.

 

Reply
0 Kudos
Kev_Johnson
Enthusiast
Enthusiast

I *could* share a support bundle @lamw , but I've reverted it back to the pre-upgrade snapshot. I can happily rerun the upgrade and snag the upgrade bundle afterwards if that would help, else I have a copy of everything in /var/log/vmware/ after the upgrade. If any of that would be helpful, happy to share, just let me know what you need.

Reply
0 Kudos
Kev_Johnson
Enthusiast
Enthusiast

(As an aside, I opted to deploy a fresh vCenter in the meantime, so there's no all fired rush on this - I posted this more as a curiosity and in case anyone else got hit by it - please don't feel like you need to spend any time on this!)

Reply
0 Kudos
nirmalgnair
VMware Employee
VMware Employee

Hi Kev,

Going by the service registration text file, Service Type: sms is missing.

Since you have reverted the setup, double check again if the above one is missing. Also make sure vmdird is NORMAL. 

You can use lsdoc tool to try and recreate the missing SMS service registration. https://kb.vmware.com/s/article/80469

Regards,

Nirmal Nair

Ajay1988
Expert
Expert

Perfect @nirmalgnair . @Kev_Johnson Please follow the same to check and create the missing registrations before upgrade. 

Below from a 8.x lab. 
[ /var/core ]# grep "Service Type:" services.txt |sort | uniq -c
1 Service Type: activedirectory
1 Service Type: applmgmt
1 Service Type: certificateauthority
1 Service Type: certificatemanagement
1 Service Type: cis.cls
1 Service Type: cis.vlcm
1 Service Type: cis.vmonapi
1 Service Type: client
1 Service Type: com.vmware.vsphere.client
1 Service Type: cs.authorization
1 Service Type: cs.componentmanager
1 Service Type: cs.ds
1 Service Type: cs.eam
1 Service Type: cs.identity
1 Service Type: cs.inventory
1 Service Type: cs.keyvalue
1 Service Type: cs.license
1 Service Type: cs.perfcharts
1 Service Type: cs.registry
1 Service Type: cs.vapi
1 Service Type: cs.vsm
1 Service Type: cs.vstats
1 Service Type: hvc
1 Service Type: imagebuilder
1 Service Type: infraprofile
1 Service Type: observability
1 Service Type: phservice
1 Service Type: rbd
1 Service Type: sca
1 Service Type: sms
1 Service Type: sso:admin
1 Service Type: sso:groupcheck
1 Service Type: sso:sts
1 Service Type: svcaccountmgmt
1 Service Type: tokenservice
1 Service Type: topologysvc
1 Service Type: trustmanagement
1 Service Type: vcenter.apigw
1 Service Type: vcenter.computepolicy
1 Service Type: vcenterserver
1 Service Type: vcha
1 Service Type: vcIntegrity
1 Service Type: vsan-dps
1 Service Type: vsan-health
1 Service Type: vsphereui
1 Service Type: vsphereui-plugins
1 Service Type: wcp

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos