This is a VCSA 6.0 with external PSC and linked mode. The Web Client doesn't work, but that is nothing new. I logged into the C# client and I am getting an error message on all the hosts and all VMs are showing as orphaned.
Error on host: Cannot synchronize host "hostname". License not available to perform the operation. Quick stats on "hostname" is not up-to-date.
To try and resolve that error, I logged into one of the ESXi hosts and restarted all services. Made no difference. I also disconnected one of the hosts and attempted to reconnect it. I just end up getting the following error when trying to reconnect the host. "Can not assign a license to Host "hostname". Make sure the License Service is available."
I've restarted the VCSA and PSC several times. Restarted services on the VCSA. Checked the vpxd.log and this is what I am seeing.
2017-07-05T10:16:43.060-04:00 info vpxd[7FF50CB94700] [Originator@6876 sub=vpxLro opID=74736762] [VpxLRO] -- BEGIN task-internal-1135 -- PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 523af613-d48b-7539-96e7-6b8978308d21(52cde336-c1de-cb5e-c432-3e5c4a098937)
2017-07-05T10:16:43.060-04:00 info vpxd[7FF50CB94700] [Originator@6876 sub=vpxLro opID=74736762] [VpxLRO] -- FINISH task-internal-1135
2017-07-05T10:16:43.067-04:00 info vpxd[7FF50CF9C700] [Originator@6876 sub=vpxLro opID=3e5421da] [VpxLRO] -- BEGIN task-internal-1136 -- PerfMgr -- vim.PerformanceManager.queryStats -- 523af613-d48b-7539-96e7-6b8978308d21(52cde336-c1de-cb5e-c432-3e5c4a098937)
2017-07-05T10:16:43.084-04:00 info vpxd[7FF50CF9C700] [Originator@6876 sub=vpxLro opID=3e5421da] [VpxLRO] -- FINISH task-internal-1136
2017-07-05T10:16:44.117-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-24@0-3d44e32e] [VpxLRO] -- BEGIN task-internal-1228 -- host-24 -- SpecSyncLRO.Synchronize --
2017-07-05T10:16:44.118-04:00 error vpxd[7FF50D2A2700] [Originator@6876 sub=licenseClientFaultTolerance opID=HB-SpecSync-host-24@0-3d44e32e] SetLicenseSourceSpec threw N7Vmacore9ExceptionE(License client start has failed.)
2017-07-05T10:16:44.188-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-24@0-3d44e32e] [VpxLRO] -- FINISH task-internal-1228
2017-07-05T10:16:44.566-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-663@0-26019a5d] [VpxLRO] -- BEGIN task-internal-1232 -- host-663 -- SpecSyncLRO.Synchronize --
2017-07-05T10:16:44.567-04:00 error vpxd[7FF50D2A2700] [Originator@6876 sub=licenseClientFaultTolerance opID=HB-SpecSync-host-663@0-26019a5d] SetLicenseSourceSpec threw N7Vmacore9ExceptionE(License client start has failed.)
2017-07-05T10:16:44.632-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-663@0-26019a5d] [VpxLRO] -- FINISH task-internal-1232
It shows that the license client start has failed, which seems obvious given the error I am receiving. Now I just need to know how to fix it.
vCenter is now working again. I don't know the exact root cause, but the problem was on the PSC. The service "vmware-cm" would start, but then stop. Since we were unable to fix that service, it was decided to repoint the impacted vCenter to a PSC in another site that was working. After going through that process, vCenter is now working including the web client.
I still need to unregister and remove the broken PSC, rebuild a new one, and then go through the repoint process again, but at least it is in a working state.
Did you tried readding esxi license and restarting service.
I did. Although the license on the ESXi host appears to be fine. This seems directly related to vCenter. Especially since it affects all ESXi hosts.
Check ls.log under C:\ProgramData\VMware\VMware VirtualCenter\Logs\ on vCenter if you find some clue.
This is the appliance. I don't see a ls.log file anywhere.
I am seeing this in the inv-svc.log.
2017-07-05T11:07:51.496-04:00 [pool-30-thread-1 WARN com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector opId=] Could not inject STS certificates into adapter servercom.vmware.vim.query.server.ssoauthen
tication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup
2017-07-05T11:07:51.496-04:00 [pool-30-thread-1 INFO com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector opId=] Failed to fetch trusted certs - Next trusted certs retrieval attempt to happen in 10s
2017-07-05T11:07:51.581-04:00 [pool-12-thread-1 ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : Administrator@vsphere.local
com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException:
Hit ExecutionException during SSO-Lookup
Maybe a certificate issue?
Have you seen this KB already
Virtual machines appear as invalid or orphaned in vCenter Server (1003742) | VMware KB
If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.
Best Regards,
Deepak Koshal
CNE|CLA|CWMA|VCP4|VCP5|CCAH
Yes, seems to be certificate issue, dig up more under /var/log/vmware/vmcad/certificate-manager.log
I did. Unfortunately nothing in that KB was helpful.
No errors in the certificate-manager.log file. Just shows previous attempts to regenerate the certificate.
With the windows based Vcenter at same level I'm seeing this a bug. Any recent changes you made ?
At the time I was trying to resolve the Web Client issue I have, and posted in https://communities.vmware.com/thread/567329.
Besides rebooting, the only change I made was changing the root password on the VCSA and PSC, and changing both to never expire. I wasn't logged into vSphere at the time, so I am not sure exactly when the issue started. I just noticed when Veeam backups started failing that there must be a communication problem with vCenter.
Whatever part of vCenter controls the license is definitely the issue. Just can't figure out what that is.
This appears to be the issue based on the error I am getting when trying to access the licenses. Only problem is the time is identical on both the VCSA and PSC. So their resolution doesn't work.
any snapshot you made before making change, you can check with reverting these ones. Or can check if any patch is available for Vcente.
So, I think the issue is with the STS signing certificates.
2017-07-05T12:39:30.647-04:00 pool-3-thread-1 WARN common.vmomi.authn.impl.SsoAuthenticatorImpl STS signing certificates are missing or empty
2017-07-05T12:39:30.647-04:00 pool-3-thread-1 WARN common.vmomi.authn.impl.SsoAuthenticatorImpl authenticate: Session auth data not set in request: sessionNonce: '533598c8-68b0-4e51-93dd-50e38bcf1148' sessionUser: '' requestUri: '/ls/sdk' requestContext: '{operationID=E29E226C-00000927-8b, realUser=VSPHERE.LOCAL\Administrator}'
2017-07-05T12:39:30.647-04:00 pool-3-thread-1 INFO vim.vmomi.server.impl.ValidatorFutureImpl Validation failed for 48: Authentication result: Missing session auth data
2017-07-05T12:39:30.659-04:00 pool-3-thread-1 INFO server.common.lookup.impl.LookupServiceInfoProviderImpl Searching for local service: com.vmware.cis:cs.identity
2017-07-05T12:39:30.663-04:00 pool-3-thread-1 INFO server.common.lookup.impl.AfdLookupClientPoolableObjectFactoryImpl Existing Lookup Service connection valid: true
2017-07-05T12:39:30.672-04:00 pool-3-thread-1 INFO server.common.lookup.impl.LookupServiceInfoProviderImpl com.vmware.cis:cs.identity found!
2017-07-05T12:39:30.672-04:00 pool-3-thread-1 ERROR server.common.sso.impl.SsoAdminProviderImpl Refetch STS certificates failed
Only problem is my Web Client doesn't work and therefore I am not able refresh the STS certificate...
No snapshot, and I am on the latest patches.
Yeah, I see they provided the way only via web client. Not sure if any commands are available for this.
Haven't found any. Also haven't heard back from VMware Support. Just over 4 hours since I submitted a case.
Keep calling them, I know how much time they spend on a single case :smileygrin:, have gone through this.