Highlighted
Enthusiast
Enthusiast

vCenter Issue - All VMs Showing as Orphaned

Jump to solution

This is a VCSA 6.0 with external PSC and linked mode. The Web Client doesn't work, but that is nothing new. I logged into the C# client and I am getting an error message on all the hosts and all VMs are showing as orphaned.

Error on host: Cannot synchronize host "hostname". License not available to perform the operation. Quick stats on "hostname" is not up-to-date.

vCenter.png

To try and resolve that error, I logged into one of the ESXi hosts and restarted all services. Made no difference. I also disconnected one of the hosts and attempted to reconnect it. I just end up getting the following error when trying to reconnect the host. "Can not assign a license to Host "hostname". Make sure the License Service is available."

I've restarted the VCSA and PSC several times. Restarted services on the VCSA. Checked the vpxd.log and this is what I am seeing.

2017-07-05T10:16:43.060-04:00 info vpxd[7FF50CB94700] [Originator@6876 sub=vpxLro opID=74736762] [VpxLRO] -- BEGIN task-internal-1135 -- PerfMgr -- vim.PerformanceManager.queryProviderSummary -- 523af613-d48b-7539-96e7-6b8978308d21(52cde336-c1de-cb5e-c432-3e5c4a098937)

2017-07-05T10:16:43.060-04:00 info vpxd[7FF50CB94700] [Originator@6876 sub=vpxLro opID=74736762] [VpxLRO] -- FINISH task-internal-1135

2017-07-05T10:16:43.067-04:00 info vpxd[7FF50CF9C700] [Originator@6876 sub=vpxLro opID=3e5421da] [VpxLRO] -- BEGIN task-internal-1136 -- PerfMgr -- vim.PerformanceManager.queryStats -- 523af613-d48b-7539-96e7-6b8978308d21(52cde336-c1de-cb5e-c432-3e5c4a098937)

2017-07-05T10:16:43.084-04:00 info vpxd[7FF50CF9C700] [Originator@6876 sub=vpxLro opID=3e5421da] [VpxLRO] -- FINISH task-internal-1136

2017-07-05T10:16:44.117-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-24@0-3d44e32e] [VpxLRO] -- BEGIN task-internal-1228 -- host-24 -- SpecSyncLRO.Synchronize --

2017-07-05T10:16:44.118-04:00 error vpxd[7FF50D2A2700] [Originator@6876 sub=licenseClientFaultTolerance opID=HB-SpecSync-host-24@0-3d44e32e] SetLicenseSourceSpec threw N7Vmacore9ExceptionE(License client start has failed.)

2017-07-05T10:16:44.188-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-24@0-3d44e32e] [VpxLRO] -- FINISH task-internal-1228

2017-07-05T10:16:44.566-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-663@0-26019a5d] [VpxLRO] -- BEGIN task-internal-1232 -- host-663 -- SpecSyncLRO.Synchronize --

2017-07-05T10:16:44.567-04:00 error vpxd[7FF50D2A2700] [Originator@6876 sub=licenseClientFaultTolerance opID=HB-SpecSync-host-663@0-26019a5d] SetLicenseSourceSpec threw N7Vmacore9ExceptionE(License client start has failed.)

2017-07-05T10:16:44.632-04:00 info vpxd[7FF50D2A2700] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-663@0-26019a5d] [VpxLRO] -- FINISH task-internal-1232

It shows that the license client start has failed, which seems obvious given the error I am receiving. Now I just need to know how to fix it.

https://tactsol.com https://vmware.solutions
1 Solution

Accepted Solutions
Highlighted
Enthusiast
Enthusiast

vCenter is now working again. I don't know the exact root cause, but the problem was on the PSC. The service "vmware-cm" would start, but then stop. Since we were unable to fix that service, it was decided to repoint the impacted vCenter to a PSC in another site that was working. After going through that process, vCenter is now working including the web client.

I still need to unregister and remove the broken PSC, rebuild a new one, and then go through the repoint process again, but at least it is in a working state.

https://tactsol.com https://vmware.solutions

View solution in original post

22 Replies
Highlighted
Virtuoso
Virtuoso

Did you tried readding esxi license and restarting service.

0 Kudos
Highlighted
Enthusiast
Enthusiast

I did. Although the license on the ESXi host appears to be fine. This seems directly related to vCenter. Especially since it affects all ESXi hosts.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Virtuoso
Virtuoso

Check ls.log under C:\ProgramData\VMware\VMware VirtualCenter\Logs\ on vCenter if you find some clue.

0 Kudos
Highlighted
Enthusiast
Enthusiast

This is the appliance. I don't see a ls.log file anywhere.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Enthusiast
Enthusiast

I am seeing this in the inv-svc.log.

2017-07-05T11:07:51.496-04:00 [pool-30-thread-1  WARN  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Could not inject STS certificates into adapter servercom.vmware.vim.query.server.ssoauthen

tication.exception.ServiceNotFoundException: Hit ExecutionException during SSO-Lookup

2017-07-05T11:07:51.496-04:00 [pool-30-thread-1  INFO  com.vmware.vim.query.server.ssoauthentication.impl.AdapterServerCertificateInjector  opId=] Failed to fetch trusted certs - Next trusted certs retrieval attempt to happen in 10s

2017-07-05T11:07:51.581-04:00 [pool-12-thread-1  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user : Administrator@vsphere.local

com.vmware.vim.query.server.ssoauthentication.exception.ServiceFaultException: com.vmware.vim.query.server.authentication.exception.TokenProviderException: com.vmware.vim.query.server.ssoauthentication.exception.ServiceNotFoundException:

Hit ExecutionException during SSO-Lookup

Maybe a certificate issue?

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Hot Shot
Hot Shot

Have you seen this KB already

Virtual machines appear as invalid or orphaned in vCenter Server (1003742) | VMware KB

If you found this or any other answer helpful, please consider the use of the Correct or Helpful to award points.

Best Regards,

Deepak Koshal

CNE|CLA|CWMA|VCP4|VCP5|CCAH

Highlighted
Virtuoso
Virtuoso

Yes, seems to be certificate issue, dig up more under /var/log/vmware/vmcad/certificate-manager.log

Highlighted
Enthusiast
Enthusiast

I did. Unfortunately nothing in that KB was helpful.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Enthusiast
Enthusiast

No errors in the certificate-manager.log file. Just shows previous attempts to regenerate the certificate.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Virtuoso
Virtuoso

With the windows based Vcenter at same level I'm seeing this a bug. Any recent changes you made ?

0 Kudos
Highlighted
Enthusiast
Enthusiast

At the time I was trying to resolve the Web Client issue I have, and posted in https://communities.vmware.com/thread/567329.

Besides rebooting, the only change I made was changing the root password on the VCSA and PSC, and changing both to never expire. I wasn't logged into vSphere at the time, so I am not sure exactly when the issue started. I just noticed when Veeam backups started failing that there must be a communication problem with vCenter.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Enthusiast
Enthusiast

Whatever part of vCenter controls the license is definitely the issue. Just can't figure out what that is.

License.png

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Enthusiast
Enthusiast

This appears to be the issue based on the error I am getting when trying to access the licenses. Only problem is the time is identical on both the VCSA and PSC. So their resolution doesn't work.

Error "Received SOAP response fault from TCP:<vCenter FQDN>:443 GetLicenses Authorization resu...

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Virtuoso
Virtuoso

any snapshot you made before making change, you can check with reverting these ones. Or can check if any patch is available  for Vcente.

0 Kudos
Highlighted
Enthusiast
Enthusiast

So, I think the issue is with the STS signing certificates.

2017-07-05T12:39:30.647-04:00 pool-3-thread-1  WARN  common.vmomi.authn.impl.SsoAuthenticatorImpl STS signing certificates are missing or empty

2017-07-05T12:39:30.647-04:00 pool-3-thread-1  WARN  common.vmomi.authn.impl.SsoAuthenticatorImpl authenticate: Session auth data not set in request: sessionNonce: '533598c8-68b0-4e51-93dd-50e38bcf1148' sessionUser: '' requestUri: '/ls/sdk' requestContext: '{operationID=E29E226C-00000927-8b, realUser=VSPHERE.LOCAL\Administrator}'

2017-07-05T12:39:30.647-04:00 pool-3-thread-1  INFO  vim.vmomi.server.impl.ValidatorFutureImpl Validation failed for 48: Authentication result: Missing session auth data

2017-07-05T12:39:30.659-04:00 pool-3-thread-1  INFO  server.common.lookup.impl.LookupServiceInfoProviderImpl Searching for local service: com.vmware.cis:cs.identity

2017-07-05T12:39:30.663-04:00 pool-3-thread-1  INFO  server.common.lookup.impl.AfdLookupClientPoolableObjectFactoryImpl Existing Lookup Service connection valid: true

2017-07-05T12:39:30.672-04:00 pool-3-thread-1  INFO  server.common.lookup.impl.LookupServiceInfoProviderImpl com.vmware.cis:cs.identity found!

2017-07-05T12:39:30.672-04:00 pool-3-thread-1  ERROR server.common.sso.impl.SsoAdminProviderImpl Refetch STS certificates failed

Only problem is my Web Client doesn't work and therefore I am not able refresh the STS certificate...

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Enthusiast
Enthusiast

No snapshot, and I am on the latest patches.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Virtuoso
Virtuoso

Yeah, I see they provided the way only via web client. Not sure if any commands are available for this.

0 Kudos
Highlighted
Enthusiast
Enthusiast

Haven't found any. Also haven't heard back from VMware Support. Just over 4 hours since I submitted a case.

https://tactsol.com https://vmware.solutions
0 Kudos
Highlighted
Virtuoso
Virtuoso

Keep calling them, I know how much time they spend on a single case :smileygrin:, have gone through this.

0 Kudos