I have successfully upgraded my VCSA 6 to 6.5 - after the upgrade I was able to login and use the vsphere web client without issue.
I then upgraded my single ESXi 6 host to 6.5, which was successful. After booting up the VCSA, after logging into the Vsphere client I get the following error:-
Could not connect to one or more vCenter Server systems:
At this point the client is unusable. I have tried stopping and restarting all services via SSH, and rebooting the VCSA, however nothing has helped.
I have also tried following the instructions here “Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere W... - However I only see 1 Service ID, so there doesn't appear to be a duplicate entry..
Many thanks
Can you attach the sceenshot of the error? Are you able to login the webclient? and if possible attach the vcsupport bundle.
Hi - This is a screenshot of the error.
I am able to login to the ESXi host web GUI. This issue occurs with the vSphere web client.
Many thanks
Hi,
Disable Ad-Block to your Browser.
Thanks,
Hello all,
To add to this thread -- I experienced the exact same issue yesterday for no obvious reason. I was using the vCSA (Connected to several VMs), then out of nowhere it dropped. The appliance is online and supposedly it is "Good" and "Healthy". I have rebooted the appliance, but no avail.
This is the output from cli:
Command> service-control --status --all
StartPending:
vmware-sps vmware-updatemgr
Running:
applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
pschealth vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-psc-client vmware-rbd-watchdog vmware-vcha vmware-vpxd
I can see that there are some services that shouldn't be in a stopped state, so I ran below:
Command> service-control --start --all
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
2016-11-30T23:14:40.820Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']
2016-11-30T23:14:40.824Z Done running command
2016-11-30T23:14:40.826Z Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.826Z Running command: ['/sbin/service', u'lwsmd', 'status']
2016-11-30T23:14:40.846Z Done running command
Successfully started service lwsmd
2016-11-30T23:14:40.849Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']
2016-11-30T23:14:40.854Z Done running command
2016-11-30T23:14:40.856Z Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.856Z Running command: ['/sbin/service', u'vmafdd', 'status']
2016-11-30T23:14:40.875Z Done running command
Successfully started service vmafdd
2016-11-30T23:14:40.878Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']
2016-11-30T23:14:40.883Z Done running command
2016-11-30T23:14:40.885Z Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.885Z Running command: ['/sbin/service', u'vmdird', 'status']
2016-11-30T23:14:40.904Z Done running command
Successfully started service vmdird
2016-11-30T23:14:40.907Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']
2016-11-30T23:14:40.912Z Done running command
2016-11-30T23:14:40.914Z Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.914Z Running command: ['/sbin/service', u'vmcad', 'status']
2016-11-30T23:14:40.933Z Done running command
Successfully started service vmcad
2016-11-30T23:14:40.936Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']
2016-11-30T23:14:40.941Z Done running command
2016-11-30T23:14:40.943Z Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.943Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
2016-11-30T23:14:40.962Z Done running command
Successfully started service vmware-sts-idmd
2016-11-30T23:14:40.964Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']
2016-11-30T23:14:40.969Z Done running command
2016-11-30T23:14:40.971Z Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:40.971Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
2016-11-30T23:14:40.991Z Done running command
Successfully started service vmware-stsd
2016-11-30T23:14:40.994Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']
2016-11-30T23:14:40.998Z Done running command
2016-11-30T23:14:41.000Z Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:41.001Z Running command: ['/sbin/service', u'vmdnsd', 'status']
2016-11-30T23:14:41.023Z Done running command
Successfully started service vmdnsd
2016-11-30T23:14:41.025Z Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']
2016-11-30T23:14:41.030Z Done running command
2016-11-30T23:14:41.032Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
2016-11-30T23:14:41.032Z Running command: ['/sbin/service', u'vmware-psc-client', 'status']
2016-11-30T23:14:41.052Z Done running command
Successfully started service vmware-psc-client
Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start pschealth services. Error: Service crashed while starting
/var/log/vmware/psc-client/catalina.2016-12-01.log:
01-Dec-2016 09:24:53.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-bio-7081"]
01-Dec-2016 09:24:53.844 WARNING [main] com.springsource.tcserver.serviceability.rmi.JmxSocketListener.init Not initializing due to port being set to -1
01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Pivotal tc Runtime 3.1.4.SR1/8.0.33.A.RELEASE
01-Dec-2016 09:24:53.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war
01-Dec-2016 09:24:54.233 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
01-Dec-2016 09:24:54.288 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war has finished in 369 ms
01-Dec-2016 09:24:54.291 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war
01-Dec-2016 09:24:54.392 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
01-Dec-2016 09:24:54.394 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war has finished in 102 ms
01-Dec-2016 09:24:54.399 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/psc.war
/var/log/vmware/psc-client/localhost.2016-11-30.log
30-Nov-2016 03:30:25.252 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'
30-Nov-2016 03:30:25.494 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
30-Nov-2016 14:30:55.483 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
30-Nov-2016 14:30:55.577 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
30-Nov-2016 14:31:00.159 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'
30-Nov-2016 15:06:20.403 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
30-Nov-2016 15:06:20.579 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
30-Nov-2016 15:06:25.827 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'
30-Nov-2016 04:15:17.420 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'
30-Nov-2016 04:15:17.458 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
30-Nov-2016 04:16:25.727 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
30-Nov-2016 04:16:26.596 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
30-Nov-2016 04:17:00.499 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'
30-Nov-2016 22:24:30.373 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'
30-Nov-2016 22:24:30.544 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
30-Nov-2016 22:25:14.320 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
30-Nov-2016 22:25:14.412 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
30-Nov-2016 22:25:19.200 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'
/var/log/vmware/psc-client/psc-client.log | grep -i WARN:
[2016-11-30 22:25:17.621] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter
[2016-11-30 22:25:17.622] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionListener
[2016-11-30 22:25:17.623] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionObjectFilter
[2016-11-30 22:25:17.940] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider
[2016-11-30 22:25:17.941] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider
[2016-11-30 22:25:17.943] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.PropertyProviderBean
[2016-11-30 22:25:17.944] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.DataProviderAdapter
[2016-11-30 22:25:17.948] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.provider.ExpressionProvider
[2016-11-30 22:25:18.523] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.vim.security.sso.SsoServiceProvider
[2016-11-30 22:25:18.645] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.mvc.objectnavigator.interpreter.NavigationRequestInterpreter
[2016-11-30 22:25:19.057] [WARN ] localhost-startStop-1 org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy - Retained attributes is deprecated. Override the extractAttributes() method instead.
[2016-11-30 22:25:47.705] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter
[2016-11-30 22:25:47.707] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
[2016-11-30 22:25:47.716] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
[2016-11-30 22:25:47.751] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.impl.JeePluginLoader - Directory/usr/lib/vmware-psc-client/webapps/ROOT is not a plugin (no plugin.xml file).
[2016-11-30 22:25:47.876] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
[2016-11-30 22:25:47.878] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
[2016-11-30 22:25:47.893] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
[2016-11-30 22:25:47.895] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider
/var/log/vmware/vpxd/vpxd.log | grep -i ERROR:
2016-11-30T23:14:41.713Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file : /etc/motd)
2016-11-30T23:14:41.717Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument
2016-11-30T23:14:43.900Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: FileIO error: Could not find file : /var/lib/vmware/hpMetadataCache.xml
2016-11-30T23:14:44.521Z error vpxd[7F2FE8923700] [Originator@6876 sub=vmomi.soapStub[5]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
2016-11-30T23:14:44.648Z error vpxd[7F2FE31E3700] [Originator@6876 sub=vmomi.soapStub[7]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for embeddedEsx version e.x.p: vpx-upgrade-eesx-1-linux-*
2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for esx version e.x.p: vpx-upgrade-esx-7-linux-*
2016-11-30T23:14:44.992Z error vpxd[7F2FE12A5700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2016-11-30T23:14:45.014Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"
2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"
2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"
2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"
2016-11-30T23:14:45.147Z error vpxd[7F2FE32E5700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2016-11-30T23:14:45.404Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: FileIO error: Permission denied for file : /etc/vmware-vpx/vpxd.cfg.tmp
2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.
2016-11-30T23:14:45.735Z error vpxd[7F2FE1428700] [Originator@6876 sub=vmomi.soapStub[17]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
2016-11-30T23:14:45.833Z error vpxd[7F2FE3EFD700] [Originator@6876 sub=vmomi.soapStub[19]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'
/var/log/vmware/vpxd/vpxd.log | grep -i WARN:
2016-11-30T23:14:41.709Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-mob-pipe. Removing...
2016-11-30T23:14:43.385Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] Init [LocalizationManagerMo::Init()] took 1666 ms
2016-11-30T23:14:43.506Z warning vpxd[7F2FE8186700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:44.369Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-webserver-pipe. Removing...
2016-11-30T23:14:44.377Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file or VECS infois not found or is not valid for extension com.vmware.vim.stats.report
2016-11-30T23:14:44.378Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file or VECS infois not found or is not valid for extension com.vmware.vim.vcha
2016-11-30T23:14:44.390Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file or VECS infois not found or is not valid for extension hostdiag
2016-11-30T23:14:44.521Z warning vpxd[7F2FE8923700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:44.648Z warning vpxd[7F2FE31E3700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:44.801Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3217 ms
2016-11-30T23:14:44.992Z warning vpxd[7F2FE12A5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:45.119Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')
2016-11-30T23:14:45.147Z warning vpxd[7F2FE32E5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:45.187Z warning vpxd[7F2FE34E9700] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')
2016-11-30T23:14:45.347Z warning vpxd[7F2FE13A7700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.
2016-11-30T23:14:45.735Z warning vpxd[7F2FE1428700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:45.833Z warning vpxd[7F2FE3EFD700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'
2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'
2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'
2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'
2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'
2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'
2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'
2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'
2016-11-30T23:14:51.549Z warning vpxd[7F2FE2040700] [Originator@6876 sub=VpxProfiler opID=StatsTruncateExpiredPartitions-618fb492] StatsTruncateExpiredPartitions [TotalTime] took 6230 ms
2016-11-30T23:14:53.264Z warning vpxd[7F2FE2AD5700] [Originator@6876 sub=OMM opID=HostSync-host-34-11775afc-SWI-65bb0468] Cannot download vm config of [vim.VirtualMachine:vm-1166,MikroTikCHR-6.37.2]: Received SOAP response fault from [<cs p:00007f2fcc58f600, TCP:esxi02.domain.local:443>]: downloadVMXConfig
Checked disk space as well (Just incase):
Filesystem | Size | Used | Avail | Use% | Mounted on |
devtmpfs | 4.9G | 0 | 4.9G | 0% | /dev |
tmpfs | 4.9G | 12K | 4.9G | 1% | /dev/shm |
tmpfs | 4.9G | 700K | 4.9G | 1% | /run |
tmpfs | 4.9G | 0 | 4.9G | 0% | /sys/fs/cgroup |
/dev/sda3 | 11G | 4.0G | 6.1G | 40% | / |
tmpfs | 4.9G | 11M | 4.9G | 1% | /tmp |
/dev/mapper/netdump_vg-netdump | 985M | 1.3M | 932M | 1% | /storage/netdump |
/dev/mapper/autodeploy_vg-autodeploy | 9.8G | 23M | 9.2G | 1% | /storage/autodeploy |
/dev/mapper/core_vg-core | 25G | 1.4G | 22G | 6% | /storage/core |
/dev/mapper/log_vg-log | 9.8G | 241M | 9.0G | 3% | /storage/log |
/dev/sda1 | 120M | 28M | 87M | 24% | /boot |
/dev/mapper/imagebuilder_vg-imagebuilder | 9.8G | 23M | 9.2G | 1% | /storage/imagebuilder |
/dev/mapper/dblog_vg-dblog | 15G | 134M | 14G | 1% | /storage/dblog |
/dev/mapper/seat_vg-seat | 9.8G | 75M | 9.2G | 1% | /storage/seat |
/dev/mapper/db_vg-db | 9.8G | 130M | 9.1G | 2% | /storage/db |
/dev/mapper/updatemgr_vg-updatemgr | 99G | 64M | 94G | 1% | /storage/updatemgr |
This is a fresh 6.5 lab environment as well.
Thanks in advance!
We have exactly the same situation. Dropped out out of the blue. Also had 2 out of 3 hosts on ESXi 6.5. Both ESXi hosts upgraded from 6.0 to 6.5.
Also we migrated from VCSA 6.0 to 6.5. Rebooted before and all was fine.
VPXD logs say something interessting:
...
2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Execution elapsed time: 0 ms
2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Statement diagnostic data from driver is 23505:0:7:ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query
...
2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:7
2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110
2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] DoHostSync:host-9 [DoHostSyncTime] took 2670 ms
2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] InvtHostSyncLRO::StartWork [HostSyncTime] took 2670 ms
2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=vpxCommon opID=HostSync-host-9-1d836128] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
2016-12-14T21:53:03.765Z panic vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-4602587, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B9EA8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B4E90]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B7E5B]
--> backtrace[03] libvmacore.so[0x002B7F36]: Vmacore::PanicExit(char const*)
--> backtrace[04] vpxd[0x00722E78]
--> backtrace[05] vpxd[0x00738629]
--> backtrace[06] vpxd[0x0073A837]
--> backtrace[07] vpxd[0x0073FB68]
--> backtrace[08] libvmacore.so[0x0023ED6F]
--> backtrace[09] libvmacore.so[0x0023F056]
--> backtrace[10] libvmacore.so[0x002BF67B]
--> backtrace[11] libpthread.so.0[0x00007454]
--> backtrace[12] libc.so.6[0x000E8E2D]
--> [backtrace end]
What is going on?
Has anybody a suggestion?
In the end I ended up creating a new vCenter VM and removing the old one.
Clearly things go amiss during the upgrade, even though the installer states the process is successful.
Regards
Hi ,
I also faced the same issue .
The error can be cleared , if you remove the certificate it won't pop up the error , but vcenter inventory was not showing there .
I got this issue after a week from the installation , one week it was smooth .
You may check the NTP configuration and try to migrate to another esxi host and check .
There isn't enough information in the log snippet. The issue looks similar to vCenter – VdbODBCError – unique constraint pk_vpx_vm_virtual_device – fix | vMan where you can see possible workaround
I have done some more digging into this issue, and it seems to crop up if I upgrade the VCSA first and then update my hosts afterwards. I recently performed an update of VCSA 6.5 to 6.5a and rebooted and the web GUI worked without issue. Once I upgraded my ESXi hosts from 6.5 to 6.5a the problem occurred.
Something seems to break with respect to communication between the VCSA and the hosts after upgrading the hosts.
Hi All,
I have exact same problem. Two vCenter servers in linked mode with external PSCs. One day suddenly one of Vcenter servers stopped to respond. The issue that we have is not related to doubled DB entry as other suggested that.In logs I see exact same entries as misan.
Additionally I have checked time on both VCs and PSCs and replication between PSCs, all looks good. I have tried to re-generate certificate on PSC where affected VCs is connecting to, but this has failed. I have also performed upgrade to VCSA 6.5c build number 5400, no luck, still cannot start vpxd service. Hopefully this is a test lab, but I would like to avoid reinstallilng it as it runs dvswitch with bunch of VMs connected to it and Primary NSX controller for Cross-vCenter implementation. Hope that someone from VMware will take a look on this. At this point I do not recommend to go into production with VCSA 6.5.
Hi All,
today we had the same issue as described above.
To remember:
Could not connect to one or more vCenter Server systems:
On VCSA Shell:
service-control --status vmware-vpxd shows "stopped"
service-control --start vmware-vpxd starts the service starts for a couple of seconds and stops again
We contacted VMware Support and they analyzed the core dump logs (/var/core) and could figure out a problem with a single VM in our environment.
There was a problem with a network adapter on that VM (DB entry NULL)
Solution that worked for us:
Find VM
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres (Login to postgres)
select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name=''; (get affected VM)
select * from vpx_entity where id = '<YOURID>'; (get affected VM Name)
Fix VM / vpxd (for our specific issue with database issue)
Remove Network Adapter, save config, readd Network Adapter
service-control --start vmware-vpxd
You should now be able to login to your vCenter.
So the NULL Value in the database seem to be a really rare issue. In case of "Could not connect to.....:" just Find VM unregister it from vCenter and Reregister.
Hope this helps somebody!
Regards,
Gabriel
Had the same issue as you although I didn't interpret solution very clearly. You state to remove the network adapter from the VM which I wrongfully assumed you had to do from the database instead of the local webui/management of the esxi server. I went ahead and removed the problem VM from the inventory of the esxi server and a restart of vpxd on VCSA had everything back up and running instantly.
What broke this for me I believe is that I copied a VM from Workstation 12 onto a datastore and registered that VM via VCSA. Shortly after starting the problem VM, VCSA did freak out and have this problem. Now I'm in the process of using vCenter Converter with hopes I don't have to repeat this!
That worked for me. Thanks!
This works for me to.
Thanks for writing this solution.
Regards
Pascal
Gabriel
Thanks a million for posting this, (shame the original poster didn't give you the points)
We had this on our semi production enviroment and you saved me a call to support.
In my case I think this came from modifing the vm from the vmrc client through the html5 web
A couple of heads ups
When runing the select command the ip address is that of the host, so it's easy to find where the vms is located
VCDB=# select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name='';
id | local_file_name | ip_address
-----+-----------------------------------------------------------------------------------+--------------
608 | /vmfs/volumes/5953bbbb7-05cbbb38-82f4-00bbbfafb56/NTvmXXXX/NTvmXXXX.vmx | 10.111.210.90
(1 row)
It wasn't possible to modify the vm, so we created a local vSwitch and port group and the vm assigned itself to that
thanks again
Russ
TLDR; If you have any v5.x Host Profiles, delete them.
We recently upgraded our v5.5 Windows vCenter with embedded SSO to a v6.5 vCSA with an external PSC. Everything seemed stable until we updated our ESXi hosts to v6.5, at which point interacting with the hosts in the Flash client (at least this seemed to be the most reliable trigger) would cause the vpxd service to crash.
None of the fixes in this thread worked for us, but a VMware tech stopped by today and we informed her that we were having this problem. She immediately told us to delete our v5.x Host Profiles, as they had recently seen this problem at another site. We had one v5.x Host Profile, and deleted it. After some testing, deleting the Host Profile appears to have resolved our vpxd service crashes.
This solved my problem as well
I was still waiting for VMWare support to come back to me and found this page
It seems that vCenter is a little delicate if it is completely stopped with some bad data
The cause of it for me was an ip address conflict between a VM guest and VM Host on different site to the vCenter server
A colleague got this solution for at least improving web client performance.
Manually increasing the heap memory on vCenter Server components in vCenter 6.x (2150757)
https://kb.vmware.com/s/article/2150757
hope it reduces the frustration of using the web client.