VMware Cloud Community
misan
Enthusiast
Enthusiast
‎11-21-2016 05:54 PM

VCSA 6.5 Web GUI Error

I have successfully upgraded my VCSA 6 to 6.5 - after the upgrade I was able to login and use the vsphere web client without issue.

I then upgraded my single ESXi 6 host to 6.5, which was successful.  After booting up the VCSA, after logging into the Vsphere client I get the following error:-

Could not connect to one or more vCenter Server systems:

https://FQDN:443/sdk

At this point the client is unusable.  I have tried stopping and restarting all services via SSH, and rebooting the VCSA, however nothing has helped.

I have also tried following the instructions here “Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere W... - However I only see 1 Service ID, so there doesn't appear to be a duplicate entry..

Many thanks

Reply
21 Replies
sanjeebkumar
VMware Employee
VMware Employee
‎11-21-2016 08:09 PM

Can you attach the sceenshot of the error? Are you able to login the webclient? and if possible attach the vcsupport bundle.

Reply
0 Kudos
misan
Enthusiast
Enthusiast
‎11-22-2016 05:56 AM

Hi - This is a screenshot of the error.

I am able to login to the ESXi host web GUI.   This issue occurs with the vSphere web client.

screen.png

Many thanks

Reply
0 Kudos
Marmotte94
Enthusiast
Enthusiast
‎11-22-2016 10:10 AM

Hi,

Disable Ad-Block to your Browser.

Thanks,

Please, visit my blog http://www.purplescreen.eu/
Reply
seanagius
Contributor
Contributor
‎11-30-2016 03:39 PM

Hello all,

To add to this thread -- I experienced the exact same issue yesterday for no obvious reason. I was using the vCSA (Connected to several VMs), then out of nowhere it dropped. The appliance is online and supposedly it is "Good" and "Healthy". I have rebooted the appliance, but no avail.

pastedImage_92.png

pastedImage_99.png

This is the output from cli:

Command> service-control --status --all

StartPending:

vmware-sps vmware-updatemgr

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-psc-client vmware-rbd-watchdog vmware-vcha vmware-vpxd

I can see that there are some services that shouldn't be in a stopped state, so I ran below:

Command> service-control --start --all

Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

2016-11-30T23:14:40.820Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']

2016-11-30T23:14:40.824Z   Done running command

2016-11-30T23:14:40.826Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.826Z   Running command: ['/sbin/service', u'lwsmd', 'status']

2016-11-30T23:14:40.846Z   Done running command

Successfully started service lwsmd

2016-11-30T23:14:40.849Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']

2016-11-30T23:14:40.854Z   Done running command

2016-11-30T23:14:40.856Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.856Z   Running command: ['/sbin/service', u'vmafdd', 'status']

2016-11-30T23:14:40.875Z   Done running command

Successfully started service vmafdd

2016-11-30T23:14:40.878Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']

2016-11-30T23:14:40.883Z   Done running command

2016-11-30T23:14:40.885Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.885Z   Running command: ['/sbin/service', u'vmdird', 'status']

2016-11-30T23:14:40.904Z   Done running command

Successfully started service vmdird

2016-11-30T23:14:40.907Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']

2016-11-30T23:14:40.912Z   Done running command

2016-11-30T23:14:40.914Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.914Z   Running command: ['/sbin/service', u'vmcad', 'status']

2016-11-30T23:14:40.933Z   Done running command

Successfully started service vmcad

2016-11-30T23:14:40.936Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']

2016-11-30T23:14:40.941Z   Done running command

2016-11-30T23:14:40.943Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.943Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']

2016-11-30T23:14:40.962Z   Done running command

Successfully started service vmware-sts-idmd

2016-11-30T23:14:40.964Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']

2016-11-30T23:14:40.969Z   Done running command

2016-11-30T23:14:40.971Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:40.971Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']

2016-11-30T23:14:40.991Z   Done running command

Successfully started service vmware-stsd

2016-11-30T23:14:40.994Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']

2016-11-30T23:14:40.998Z   Done running command

2016-11-30T23:14:41.000Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:41.001Z   Running command: ['/sbin/service', u'vmdnsd', 'status']

2016-11-30T23:14:41.023Z   Done running command

Successfully started service vmdnsd

2016-11-30T23:14:41.025Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']

2016-11-30T23:14:41.030Z   Done running command

2016-11-30T23:14:41.032Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2016-11-30T23:14:41.032Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']

2016-11-30T23:14:41.052Z   Done running command

Successfully started service vmware-psc-client

Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start pschealth services. Error: Service crashed while starting

/var/log/vmware/psc-client/catalina.2016-12-01.log:

01-Dec-2016 09:24:53.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-bio-7081"]

01-Dec-2016 09:24:53.844 WARNING [main] com.springsource.tcserver.serviceability.rmi.JmxSocketListener.init Not initializing due to port being set to -1

01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina

01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Pivotal tc Runtime 3.1.4.SR1/8.0.33.A.RELEASE

01-Dec-2016 09:24:53.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war

01-Dec-2016 09:24:54.233 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

01-Dec-2016 09:24:54.288 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war has finished in 369 ms

01-Dec-2016 09:24:54.291 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war

01-Dec-2016 09:24:54.392 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

01-Dec-2016 09:24:54.394 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war has finished in 102 ms

01-Dec-2016 09:24:54.399 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/psc.war

/var/log/vmware/psc-client/localhost.2016-11-30.log

30-Nov-2016 03:30:25.252 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

30-Nov-2016 03:30:25.494 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

30-Nov-2016 14:30:55.483 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

30-Nov-2016 14:30:55.577 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

30-Nov-2016 14:31:00.159 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

30-Nov-2016 15:06:20.403 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

30-Nov-2016 15:06:20.579 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

30-Nov-2016 15:06:25.827 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

30-Nov-2016 04:15:17.420 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

30-Nov-2016 04:15:17.458 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

30-Nov-2016 04:16:25.727 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

30-Nov-2016 04:16:26.596 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

30-Nov-2016 04:17:00.499 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

30-Nov-2016 22:24:30.373 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

30-Nov-2016 22:24:30.544 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

30-Nov-2016 22:25:14.320 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

30-Nov-2016 22:25:14.412 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

30-Nov-2016 22:25:19.200 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

/var/log/vmware/psc-client/psc-client.log | grep -i WARN:

[2016-11-30 22:25:17.621] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

[2016-11-30 22:25:17.622] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionListener

[2016-11-30 22:25:17.623] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionObjectFilter

[2016-11-30 22:25:17.940] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

[2016-11-30 22:25:17.941] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

[2016-11-30 22:25:17.943] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.PropertyProviderBean

[2016-11-30 22:25:17.944] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.DataProviderAdapter

[2016-11-30 22:25:17.948] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.provider.ExpressionProvider

[2016-11-30 22:25:18.523] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.vim.security.sso.SsoServiceProvider

[2016-11-30 22:25:18.645] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.mvc.objectnavigator.interpreter.NavigationRequestInterpreter

[2016-11-30 22:25:19.057] [WARN ] localhost-startStop-1 org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy - Retained attributes is deprecated. Override the extractAttributes() method instead.

[2016-11-30 22:25:47.705] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

[2016-11-30 22:25:47.707] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

[2016-11-30 22:25:47.716] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

[2016-11-30 22:25:47.751] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.impl.JeePluginLoader - Directory/usr/lib/vmware-psc-client/webapps/ROOT is not a plugin (no plugin.xml file).

[2016-11-30 22:25:47.876] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

[2016-11-30 22:25:47.878] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

[2016-11-30 22:25:47.893] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

[2016-11-30 22:25:47.895] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

/var/log/vmware/vpxd/vpxd.log | grep -i ERROR:

2016-11-30T23:14:41.713Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)

2016-11-30T23:14:41.717Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

2016-11-30T23:14:43.900Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: FileIO error: Could not find file  : /var/lib/vmware/hpMetadataCache.xml

2016-11-30T23:14:44.521Z error vpxd[7F2FE8923700] [Originator@6876 sub=vmomi.soapStub[5]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

2016-11-30T23:14:44.648Z error vpxd[7F2FE31E3700] [Originator@6876 sub=vmomi.soapStub[7]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for embeddedEsx version e.x.p: vpx-upgrade-eesx-1-linux-*

2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for esx version e.x.p: vpx-upgrade-esx-7-linux-*

2016-11-30T23:14:44.992Z error vpxd[7F2FE12A5700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

2016-11-30T23:14:45.014Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"

2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

2016-11-30T23:14:45.147Z error vpxd[7F2FE32E5700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

2016-11-30T23:14:45.404Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: FileIO error: Permission denied for file  : /etc/vmware-vpx/vpxd.cfg.tmp

2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

2016-11-30T23:14:45.735Z error vpxd[7F2FE1428700] [Originator@6876 sub=vmomi.soapStub[17]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

2016-11-30T23:14:45.833Z error vpxd[7F2FE3EFD700] [Originator@6876 sub=vmomi.soapStub[19]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

/var/log/vmware/vpxd/vpxd.log | grep -i WARN:

2016-11-30T23:14:41.709Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-mob-pipe. Removing...

2016-11-30T23:14:43.385Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] Init [LocalizationManagerMo::Init()] took 1666 ms

2016-11-30T23:14:43.506Z warning vpxd[7F2FE8186700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:44.369Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-webserver-pipe. Removing...

2016-11-30T23:14:44.377Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.stats.report

2016-11-30T23:14:44.378Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.vcha

2016-11-30T23:14:44.390Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension hostdiag

2016-11-30T23:14:44.521Z warning vpxd[7F2FE8923700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:44.648Z warning vpxd[7F2FE31E3700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:44.801Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3217 ms

2016-11-30T23:14:44.992Z warning vpxd[7F2FE12A5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:45.119Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

2016-11-30T23:14:45.147Z warning vpxd[7F2FE32E5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:45.187Z warning vpxd[7F2FE34E9700] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

2016-11-30T23:14:45.347Z warning vpxd[7F2FE13A7700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

2016-11-30T23:14:45.735Z warning vpxd[7F2FE1428700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:45.833Z warning vpxd[7F2FE3EFD700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

2016-11-30T23:14:51.549Z warning vpxd[7F2FE2040700] [Originator@6876 sub=VpxProfiler opID=StatsTruncateExpiredPartitions-618fb492] StatsTruncateExpiredPartitions [TotalTime] took 6230 ms

2016-11-30T23:14:53.264Z warning vpxd[7F2FE2AD5700] [Originator@6876 sub=OMM opID=HostSync-host-34-11775afc-SWI-65bb0468] Cannot download vm config of [vim.VirtualMachine:vm-1166,MikroTikCHR-6.37.2]: Received SOAP response fault from [<cs p:00007f2fcc58f600, TCP:esxi02.domain.local:443>]: downloadVMXConfig

Checked disk space as well (Just incase):

     

FilesystemSizeUsedAvailUse%Mounted on
devtmpfs4.9G04.9G0%/dev
tmpfs4.9G12K4.9G1%/dev/shm
tmpfs4.9G700K4.9G1%/run
tmpfs4.9G04.9G0%/sys/fs/cgroup
/dev/sda311G4.0G6.1G40%/
tmpfs4.9G11M4.9G1%/tmp
/dev/mapper/netdump_vg-netdump985M1.3M932M1%/storage/netdump
/dev/mapper/autodeploy_vg-autodeploy9.8G23M9.2G1%/storage/autodeploy
/dev/mapper/core_vg-core25G1.4G22G6%/storage/core
/dev/mapper/log_vg-log9.8G241M9.0G3%/storage/log
/dev/sda1120M28M87M24%/boot
/dev/mapper/imagebuilder_vg-imagebuilder9.8G23M9.2G1%/storage/imagebuilder
/dev/mapper/dblog_vg-dblog15G134M14G1%/storage/dblog
/dev/mapper/seat_vg-seat9.8G75M9.2G1%/storage/seat
/dev/mapper/db_vg-db9.8G130M9.1G2%/storage/db
/dev/mapper/updatemgr_vg-updatemgr99G64M94G1%/storage/updatemgr

This is a fresh 6.5 lab environment as well.

Thanks in advance!

Reply
0 Kudos
Arce23
Contributor
Contributor
‎12-14-2016 02:08 PM

We have exactly the same situation. Dropped out out of the blue. Also had 2 out of 3 hosts on ESXi 6.5.  Both ESXi hosts upgraded from 6.0 to 6.5.

Also we migrated from VCSA 6.0 to 6.5.  Rebooted before and all was fine.

VPXD logs say something interessting:

...

2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Execution elapsed time: 0 ms

2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Statement diagnostic data from driver is 23505:0:7:ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query

...

2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:7

2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110

2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] DoHostSync:host-9 [DoHostSyncTime] took 2670 ms

2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] InvtHostSyncLRO::StartWork [HostSyncTime] took 2670 ms

2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=vpxCommon opID=HostSync-host-9-1d836128] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

2016-12-14T21:53:03.765Z panic vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128]

-->

--> Panic: Unrecoverable VmRootError. Panic!

--> Backtrace:

--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-4602587, tag: vpxd, cpu: x86_64, os: linux, buildType: release

--> backtrace[00] libvmacore.so[0x002B9EA8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)

--> backtrace[01] libvmacore.so[0x001B4E90]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] libvmacore.so[0x002B7E5B]

--> backtrace[03] libvmacore.so[0x002B7F36]: Vmacore::PanicExit(char const*)

--> backtrace[04] vpxd[0x00722E78]

--> backtrace[05] vpxd[0x00738629]

--> backtrace[06] vpxd[0x0073A837]

--> backtrace[07] vpxd[0x0073FB68]

--> backtrace[08] libvmacore.so[0x0023ED6F]

--> backtrace[09] libvmacore.so[0x0023F056]

--> backtrace[10] libvmacore.so[0x002BF67B]

--> backtrace[11] libpthread.so.0[0x00007454]

--> backtrace[12] libc.so.6[0x000E8E2D]

--> [backtrace end]

What is going on?

Has anybody a suggestion?

Reply
0 Kudos
misan
Enthusiast
Enthusiast
‎12-14-2016 05:05 PM

In the end I ended up creating a new vCenter VM and removing the old one.

Clearly things go amiss during the upgrade, even though the installer states the process is successful.

Regards

Reply
0 Kudos
RAJ_RAJ
Expert
Expert
‎12-14-2016 09:29 PM

Hi ,

I also faced the same issue .

The error can be cleared , if you remove  the certificate it won't pop up the error , but vcenter inventory was not showing there .

I got this issue after a week from the installation , one week it was smooth .


You may check the NTP configuration  and try to migrate to another esxi host and check .

RAJESH RADHAKRISHNAN VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335 Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
Reply
0 Kudos
admin
Immortal
Immortal
‎12-15-2016 12:39 AM

There isn't enough information in the log snippet. The issue looks similar to vCenter – VdbODBCError – unique constraint pk_vpx_vm_virtual_device – fix | vMan where you can see possible workaround

Reply
misan
Enthusiast
Enthusiast
‎02-25-2017 08:30 AM

I have done some more digging into this issue, and it seems to crop up if I upgrade the VCSA first and then update my hosts afterwards.   I recently performed an update of VCSA 6.5 to 6.5a and rebooted and the web GUI worked without issue.  Once I upgraded my ESXi hosts from 6.5 to 6.5a the problem occurred.

Something seems to break with respect to communication between the VCSA and the hosts after upgrading the hosts. 

Reply
0 Kudos
Danooo
Contributor
Contributor
‎04-18-2017 04:40 AM

Hi All,

I have exact same problem. Two vCenter servers in linked mode with external PSCs. One day suddenly one of Vcenter servers stopped to respond. The issue that we have is not related to doubled DB entry as other suggested that.In logs I see exact same entries as misan.

Additionally I have checked time on both VCs and PSCs and replication between PSCs, all looks good. I have tried to re-generate certificate on PSC where affected VCs is connecting to, but this has failed. I have also performed upgrade to VCSA 6.5c build number 5400, no luck, still cannot start vpxd service. Hopefully this is a test lab, but I would like to avoid reinstallilng it as it runs dvswitch with bunch of VMs connected to it and Primary NSX controller for Cross-vCenter implementation. Hope that someone from VMware will take a look on this. At this point I do not recommend to go into production with VCSA 6.5.

Reply
0 Kudos
Barra87
Contributor
Contributor
‎06-23-2017 12:59 PM

Hi All,

today we had the same issue as described above.

To remember:

Could not connect to one or more vCenter Server systems:

https://FQDN:443/sdk

On VCSA Shell:

service-control --status vmware-vpxd shows "stopped"

service-control --start vmware-vpxd starts the service starts for a couple of seconds and stops again

We contacted VMware Support and they analyzed the core dump logs (/var/core) and could figure out a problem with a single VM in our environment.

There was a problem with a network adapter on that VM (DB entry NULL)

Solution that worked for us:

Find VM

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres (Login to postgres)

select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name=''; (get affected VM)

select * from vpx_entity where id = '<YOURID>'; (get affected VM Name)

Fix VM / vpxd (for our specific issue with database issue)

Remove Network Adapter, save config, readd Network Adapter

service-control --start vmware-vpxd

You should now be able to login to your vCenter.

So the NULL Value in the database seem to be a really rare issue. In case of "Could not connect to.....:" just Find VM unregister it from vCenter and Reregister.

Hope this helps somebody!

Regards,

Gabriel

Reply
idontwanttouset
Contributor
Contributor
‎07-11-2017 12:53 PM

Had the same issue as you although I didn't interpret solution very clearly. You state to remove the network adapter from the VM which I wrongfully assumed you had to do from the database instead of the local webui/management of the esxi server. I went ahead and removed the problem VM from the inventory of the esxi server and a restart of vpxd on VCSA had everything back up and running instantly.

What broke this for me I believe is that I copied a VM from Workstation 12 onto a datastore and registered that VM via VCSA. Shortly after starting the problem VM, VCSA did freak out and have this problem. Now I'm in the process of using vCenter Converter with hopes I don't have to repeat this!

Reply
0 Kudos
aaronklep
Contributor
Contributor
‎07-14-2017 09:36 AM

That worked for me.  Thanks!

Reply
0 Kudos
Bledenash
Contributor
Contributor
‎07-18-2017 04:14 AM

This works for me to.

Thanks for writing this solution.

Regards

Pascal

Reply
0 Kudos
roconnor
Enthusiast
Enthusiast
‎08-09-2017 09:13 AM

Gabriel

Thanks a million for posting this, (shame the original poster didn't give you the points)

We had this on our semi production enviroment and you saved me a call to support.

In my case I think this came from modifing the vm from the vmrc client through the html5 web

A couple of heads ups

When runing the select command the ip address is that of the host, so it's easy to find where the vms is located

VCDB=# select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name='';

id  |                                  local_file_name                                                                               |  ip_address

-----+-----------------------------------------------------------------------------------+--------------

608 | /vmfs/volumes/5953bbbb7-05cbbb38-82f4-00bbbfafb56/NTvmXXXX/NTvmXXXX.vmx    | 10.111.210.90

(1 row)

It wasn't possible to modify the vm, so we created a local vSwitch and port group and the vm assigned itself to that

thanks again

Russ

Reply
lmaxfield
Contributor
Contributor
‎08-17-2017 01:40 PM

TLDR; If you have any v5.x Host Profiles, delete them.

We recently upgraded our v5.5 Windows vCenter with embedded SSO to a v6.5 vCSA with an external PSC. Everything seemed stable until we updated our ESXi hosts to v6.5, at which point interacting with the hosts in the Flash client (at least this seemed to be the most reliable trigger) would cause the vpxd service to crash.

None of the fixes in this thread worked for us, but a VMware tech stopped by today and we informed her that we were having this problem. She immediately told us to delete our v5.x Host Profiles, as they had recently seen this problem at another site. We had one v5.x Host Profile, and deleted it. After some testing, deleting the Host Profile appears to have resolved our vpxd service crashes.

Reply
0 Kudos
geojen
Contributor
Contributor
‎04-12-2018 04:17 PM

This solved my problem as well

I was still waiting for VMWare support to come back to me and found this page

It seems that vCenter is a little delicate if it is completely stopped with some bad data

The cause of it for me was an ip address conflict between a VM guest and VM Host on different site to the vCenter server

Reply
0 Kudos
Tottom
Contributor
Contributor
‎08-31-2018 03:49 AM

A colleague got this solution for at least improving web client performance.

Manually increasing the heap memory on vCenter Server components in vCenter 6.x (2150757)

https://kb.vmware.com/s/article/2150757

hope it reduces the frustration of using the web client.

Reply
0 Kudos