1 2 Previous Next 19 Replies Latest reply on Aug 31, 2018 3:49 AM by Tottom

    VCSA 6.5 Web GUI Error

    misan Novice

      I have successfully upgraded my VCSA 6 to 6.5 - after the upgrade I was able to login and use the vsphere web client without issue.

       

      I then upgraded my single ESXi 6 host to 6.5, which was successful.  After booting up the VCSA, after logging into the Vsphere client I get the following error:-

       

      Could not connect to one or more vCenter Server systems:

      https://FQDN:443/sdk

       

      At this point the client is unusable.  I have tried stopping and restarting all services via SSH, and rebooting the VCSA, however nothing has helped.

       

      I have also tried following the instructions here “Failed to verify the SSL certificate for one or more vCenter Server Systems” error in the vSphere Web Client (2050273) … - However I only see 1 Service ID, so there doesn't appear to be a duplicate entry..

       

      Many thanks

        • 1. Re: VCSA 6.5 Web GUI Error
          sanjeebkumar Enthusiast
          VMware Employees

          Can you attach the sceenshot of the error? Are you able to login the webclient? and if possible attach the vcsupport bundle.

          • 2. Re: VCSA 6.5 Web GUI Error
            misan Novice

            Hi - This is a screenshot of the error.

             

            I am able to login to the ESXi host web GUI.   This issue occurs with the vSphere web client.

             

            screen.png

            Many thanks

            • 3. Re: VCSA 6.5 Web GUI Error
              Marmotte94 Enthusiast
              vExpert

              Hi,

              Disable Ad-Block to your Browser.

               

              Thanks,

              • 4. Re: VCSA 6.5 Web GUI Error
                seanagius Lurker

                Hello all,

                 

                To add to this thread -- I experienced the exact same issue yesterday for no obvious reason. I was using the vCSA (Connected to several VMs), then out of nowhere it dropped. The appliance is online and supposedly it is "Good" and "Healthy". I have rebooted the appliance, but no avail.

                 

                 

                 

                This is the output from cli:

                Command> service-control --status --all

                StartPending:

                vmware-sps vmware-updatemgr

                Running:

                applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

                Stopped:

                pschealth vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-psc-client vmware-rbd-watchdog vmware-vcha vmware-vpxd

                 

                I can see that there are some services that shouldn't be in a stopped state, so I ran below:

                Command> service-control --start --all

                Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

                2016-11-30T23:14:40.820Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']

                2016-11-30T23:14:40.824Z   Done running command

                2016-11-30T23:14:40.826Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.826Z   Running command: ['/sbin/service', u'lwsmd', 'status']

                2016-11-30T23:14:40.846Z   Done running command

                Successfully started service lwsmd

                2016-11-30T23:14:40.849Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']

                2016-11-30T23:14:40.854Z   Done running command

                2016-11-30T23:14:40.856Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.856Z   Running command: ['/sbin/service', u'vmafdd', 'status']

                2016-11-30T23:14:40.875Z   Done running command

                Successfully started service vmafdd

                2016-11-30T23:14:40.878Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']

                2016-11-30T23:14:40.883Z   Done running command

                2016-11-30T23:14:40.885Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.885Z   Running command: ['/sbin/service', u'vmdird', 'status']

                2016-11-30T23:14:40.904Z   Done running command

                Successfully started service vmdird

                2016-11-30T23:14:40.907Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']

                2016-11-30T23:14:40.912Z   Done running command

                2016-11-30T23:14:40.914Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.914Z   Running command: ['/sbin/service', u'vmcad', 'status']

                2016-11-30T23:14:40.933Z   Done running command

                Successfully started service vmcad

                2016-11-30T23:14:40.936Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']

                2016-11-30T23:14:40.941Z   Done running command

                2016-11-30T23:14:40.943Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.943Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']

                2016-11-30T23:14:40.962Z   Done running command

                Successfully started service vmware-sts-idmd

                2016-11-30T23:14:40.964Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']

                2016-11-30T23:14:40.969Z   Done running command

                2016-11-30T23:14:40.971Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:40.971Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']

                2016-11-30T23:14:40.991Z   Done running command

                Successfully started service vmware-stsd

                2016-11-30T23:14:40.994Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']

                2016-11-30T23:14:40.998Z   Done running command

                2016-11-30T23:14:41.000Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:41.001Z   Running command: ['/sbin/service', u'vmdnsd', 'status']

                2016-11-30T23:14:41.023Z   Done running command

                Successfully started service vmdnsd

                2016-11-30T23:14:41.025Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']

                2016-11-30T23:14:41.030Z   Done running command

                2016-11-30T23:14:41.032Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

                2016-11-30T23:14:41.032Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']

                2016-11-30T23:14:41.052Z   Done running command

                Successfully started service vmware-psc-client

                Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start pschealth services. Error: Service crashed while starting

                 

                /var/log/vmware/psc-client/catalina.2016-12-01.log:

                01-Dec-2016 09:24:53.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-bio-7081"]

                01-Dec-2016 09:24:53.844 WARNING [main] com.springsource.tcserver.serviceability.rmi.JmxSocketListener.init Not initializing due to port being set to -1

                01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina

                01-Dec-2016 09:24:53.886 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Pivotal tc Runtime 3.1.4.SR1/8.0.33.A.RELEASE

                01-Dec-2016 09:24:53.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war

                01-Dec-2016 09:24:54.233 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

                01-Dec-2016 09:24:54.288 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/ROOT.war has finished in 369 ms

                01-Dec-2016 09:24:54.291 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war

                01-Dec-2016 09:24:54.392 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

                01-Dec-2016 09:24:54.394 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/lib/vmware-psc-client/webapps/csp-ui.war has finished in 102 ms

                01-Dec-2016 09:24:54.399 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/lib/vmware-psc-client/webapps/psc.war

                 

                /var/log/vmware/psc-client/localhost.2016-11-30.log

                30-Nov-2016 03:30:25.252 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

                30-Nov-2016 03:30:25.494 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

                30-Nov-2016 14:30:55.483 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

                30-Nov-2016 14:30:55.577 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

                30-Nov-2016 14:31:00.159 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

                30-Nov-2016 15:06:20.403 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

                30-Nov-2016 15:06:20.579 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

                30-Nov-2016 15:06:25.827 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

                30-Nov-2016 04:15:17.420 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

                30-Nov-2016 04:15:17.458 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

                30-Nov-2016 04:16:25.727 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

                30-Nov-2016 04:16:26.596 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

                30-Nov-2016 04:17:00.499 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

                30-Nov-2016 22:24:30.373 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Destroying Spring FrameworkServlet 'spring'

                30-Nov-2016 22:24:30.544 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext

                30-Nov-2016 22:25:14.320 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath

                30-Nov-2016 22:25:14.412 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

                30-Nov-2016 22:25:19.200 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring'

                 

                /var/log/vmware/psc-client/psc-client.log | grep -i WARN:

                [2016-11-30 22:25:17.621] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

                [2016-11-30 22:25:17.622] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionListener

                [2016-11-30 22:25:17.623] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionObjectFilter

                [2016-11-30 22:25:17.940] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

                [2016-11-30 22:25:17.941] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.mutation.MutationProvider

                [2016-11-30 22:25:17.943] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.PropertyProviderBean

                [2016-11-30 22:25:17.944] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.DataProviderAdapter

                [2016-11-30 22:25:17.948] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.data.query.provider.ExpressionProvider

                [2016-11-30 22:25:18.523] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.vim.security.sso.SsoServiceProvider

                [2016-11-30 22:25:18.645] [WARN ] localhost-startStop-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.mvc.objectnavigator.interpreter.NavigationRequestInterpreter

                [2016-11-30 22:25:19.057] [WARN ] localhost-startStop-1 org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy - Retained attributes is deprecated. Override the extractAttributes() method instead.

                [2016-11-30 22:25:47.705] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionFilter

                [2016-11-30 22:25:47.707] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                [2016-11-30 22:25:47.716] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                [2016-11-30 22:25:47.751] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.impl.JeePluginLoader - Directory/usr/lib/vmware-psc-client/webapps/ROOT is not a plugin (no plugin.xml file).

                [2016-11-30 22:25:47.876] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                [2016-11-30 22:25:47.878] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                [2016-11-30 22:25:47.893] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                [2016-11-30 22:25:47.895] [WARN ] pluginLoader-thread-pool-1 com.vmware.vise.extensionfw.TomcatServiceRegistry - No Services registered for interface com.vmware.vise.extensionfw.ExtensionPersistenceProvider

                 

                /var/log/vmware/vpxd/vpxd.log | grep -i ERROR:

                2016-11-30T23:14:41.713Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(FileIO error: Could not find file  : /etc/motd)

                2016-11-30T23:14:41.717Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=MoOptionMgr] [OptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: vmodl.fault.InvalidArgument

                2016-11-30T23:14:43.900Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: FileIO error: Could not find file  : /var/lib/vmware/hpMetadataCache.xml

                2016-11-30T23:14:44.521Z error vpxd[7F2FE8923700] [Originator@6876 sub=vmomi.soapStub[5]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

                2016-11-30T23:14:44.648Z error vpxd[7F2FE31E3700] [Originator@6876 sub=vmomi.soapStub[7]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

                2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for embeddedEsx version e.x.p: vpx-upgrade-eesx-1-linux-*

                2016-11-30T23:14:44.735Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=HostUpgrader] [VpxdHostUpgrader] Could not find vpxa bundle for esx version e.x.p: vpx-upgrade-esx-7-linux-*

                2016-11-30T23:14:44.992Z error vpxd[7F2FE12A5700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

                2016-11-30T23:14:45.014Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

                2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"

                2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

                2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

                2016-11-30T23:14:45.121Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-7f64b63f-05d8-4db5-ad02-d7a56fcd0631"

                2016-11-30T23:14:45.147Z error vpxd[7F2FE32E5700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

                2016-11-30T23:14:45.404Z error vpxd[7F2FF6AAB800] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: FileIO error: Permission denied for file  : /etc/vmware-vpx/vpxd.cfg.tmp

                2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

                2016-11-30T23:14:45.735Z error vpxd[7F2FE1428700] [Originator@6876 sub=vmomi.soapStub[17]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

                2016-11-30T23:14:45.833Z error vpxd[7F2FE3EFD700] [Originator@6876 sub=vmomi.soapStub[19]] initial service state request failed, disabling pings. error=HTTP Status:500 'Internal Server Error'

                 

                /var/log/vmware/vpxd/vpxd.log | grep -i WARN:

                2016-11-30T23:14:41.709Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-mob-pipe. Removing...

                2016-11-30T23:14:43.385Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] Init [LocalizationManagerMo::Init()] took 1666 ms

                2016-11-30T23:14:43.506Z warning vpxd[7F2FE8186700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:44.369Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxUtil] [VpxVmomi] Found previous domain socket /var/run/vmware/vpxd-webserver-pipe. Removing...

                2016-11-30T23:14:44.377Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.stats.report

                2016-11-30T23:14:44.378Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension com.vmware.vim.vcha

                2016-11-30T23:14:44.390Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=MoExtensionMgr] [ExtensionManagerMo] Public key file  or VECS infois not found or is not valid for extension hostdiag

                2016-11-30T23:14:44.521Z warning vpxd[7F2FE8923700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:44.648Z warning vpxd[7F2FE31E3700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:44.801Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3217 ms

                2016-11-30T23:14:44.992Z warning vpxd[7F2FE12A5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:45.119Z warning vpxd[7F2FF6AAB800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

                2016-11-30T23:14:45.147Z warning vpxd[7F2FE32E5700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:45.187Z warning vpxd[7F2FE34E9700] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')

                2016-11-30T23:14:45.347Z warning vpxd[7F2FE13A7700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:45.433Z info vpxd[7F2FF6AAB800] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.

                2016-11-30T23:14:45.735Z warning vpxd[7F2FE1428700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:45.833Z warning vpxd[7F2FE3EFD700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.hostdiag.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.rbd.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.762Z warning vpxd[7F2FE2346700] [Originator@6876 sub=Default opID=74f7f84d] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.hostdiag.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.rbd.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:47.954Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

                2016-11-30T23:14:47.955Z warning vpxd[7F2FE89A4700] [Originator@6876 sub=Default opID=62961005] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

                2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

                2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.449Z warning vpxd[7F2FE2952700] [Originator@6876 sub=Default opID=71bcbbfa] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vim.vcha.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.hostdiag.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.com.vmware.content.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.cl.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.rbd.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.client..summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.com.vmware.vim.viClientScripts.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vmcam.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.com.vmware.vcenter.iso.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcenter.iso.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.com.vmware.ovf.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.ovf.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.HTTP.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vcIntegrity.server.cis.common.skipreg.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.client.vsphere-client-serenity.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.com.vmware.vsphere.client.vsan.summary'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.label'

                2016-11-30T23:14:48.749Z warning vpxd[7F2FE0E1C700] [Originator@6876 sub=Default opID=5cda4ce4] Default resource used for extension description 'extension.com.vmware.vsan.health.server.cis.common.skipreg.summary'

                2016-11-30T23:14:51.549Z warning vpxd[7F2FE2040700] [Originator@6876 sub=VpxProfiler opID=StatsTruncateExpiredPartitions-618fb492] StatsTruncateExpiredPartitions [TotalTime] took 6230 ms

                2016-11-30T23:14:53.264Z warning vpxd[7F2FE2AD5700] [Originator@6876 sub=OMM opID=HostSync-host-34-11775afc-SWI-65bb0468] Cannot download vm config of [vim.VirtualMachine:vm-1166,MikroTikCHR-6.37.2]: Received SOAP response fault from [<cs p:00007f2fcc58f600, TCP:esxi02.domain.local:443>]: downloadVMXConfig

                 

                Checked disk space as well (Just incase):

                     

                FilesystemSizeUsedAvailUse%Mounted on
                devtmpfs4.9G04.9G0%/dev
                tmpfs4.9G12K4.9G1%/dev/shm
                tmpfs4.9G700K4.9G1%/run
                tmpfs4.9G04.9G0%/sys/fs/cgroup
                /dev/sda311G4.0G6.1G40%/
                tmpfs4.9G11M4.9G1%/tmp
                /dev/mapper/netdump_vg-netdump985M1.3M932M1%/storage/netdump
                /dev/mapper/autodeploy_vg-autodeploy9.8G23M9.2G1%/storage/autodeploy
                /dev/mapper/core_vg-core25G1.4G22G6%/storage/core
                /dev/mapper/log_vg-log9.8G241M9.0G3%/storage/log
                /dev/sda1120M28M87M24%/boot
                /dev/mapper/imagebuilder_vg-imagebuilder9.8G23M9.2G1%/storage/imagebuilder
                /dev/mapper/dblog_vg-dblog15G134M14G1%/storage/dblog
                /dev/mapper/seat_vg-seat9.8G75M9.2G1%/storage/seat
                /dev/mapper/db_vg-db9.8G130M9.1G2%/storage/db
                /dev/mapper/updatemgr_vg-updatemgr99G64M94G1%/storage/updatemgr

                 

                This is a fresh 6.5 lab environment as well.

                 

                Thanks in advance!

                • 5. Re: VCSA 6.5 Web GUI Error
                  Arce23 Novice

                  We have exactly the same situation. Dropped out out of the blue. Also had 2 out of 3 hosts on ESXi 6.5.  Both ESXi hosts upgraded from 6.0 to 6.5.

                  Also we migrated from VCSA 6.0 to 6.5.  Rebooted before and all was fine.

                  VPXD logs say something interessting:

                  ...

                  2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

                  2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Execution elapsed time: 0 ms

                  2016-12-14T21:53:03.755Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] Statement diagnostic data from driver is 23505:0:7:ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

                  --> Error while executing the query

                  ...

                  2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:7

                  2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110

                  2016-12-14T21:53:03.756Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

                  --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

                  2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] DoHostSync:host-9 [DoHostSyncTime] took 2670 ms

                  2016-12-14T21:53:03.762Z warning vpxd[7F01D0EB0700] [Originator@6876 sub=VpxProfiler opID=HostSync-host-9-1d836128] InvtHostSyncLRO::StartWork [HostSyncTime] took 2670 ms

                  2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=vpxCommon opID=HostSync-host-9-1d836128] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...

                  2016-12-14T21:53:03.762Z error vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

                  --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

                  2016-12-14T21:53:03.765Z panic vpxd[7F01D0EB0700] [Originator@6876 sub=Default opID=HostSync-host-9-1d836128]

                  -->

                  --> Panic: Unrecoverable VmRootError. Panic!

                  --> Backtrace:

                  --> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-4602587, tag: vpxd, cpu: x86_64, os: linux, buildType: release

                  --> backtrace[00] libvmacore.so[0x002B9EA8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)

                  --> backtrace[01] libvmacore.so[0x001B4E90]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

                  --> backtrace[02] libvmacore.so[0x002B7E5B]

                  --> backtrace[03] libvmacore.so[0x002B7F36]: Vmacore::PanicExit(char const*)

                  --> backtrace[04] vpxd[0x00722E78]

                  --> backtrace[05] vpxd[0x00738629]

                  --> backtrace[06] vpxd[0x0073A837]

                  --> backtrace[07] vpxd[0x0073FB68]

                  --> backtrace[08] libvmacore.so[0x0023ED6F]

                  --> backtrace[09] libvmacore.so[0x0023F056]

                  --> backtrace[10] libvmacore.so[0x002BF67B]

                  --> backtrace[11] libpthread.so.0[0x00007454]

                  --> backtrace[12] libc.so.6[0x000E8E2D]

                  --> [backtrace end]

                   

                  What is going on?

                  Has anybody a suggestion?

                  • 6. Re: VCSA 6.5 Web GUI Error
                    misan Novice

                    In the end I ended up creating a new vCenter VM and removing the old one.

                     

                    Clearly things go amiss during the upgrade, even though the installer states the process is successful.

                     

                    Regards

                    • 7. Re: VCSA 6.5 Web GUI Error
                      RAJ_RAJ Expert
                      vExpert

                      Hi ,

                       

                      I also faced the same issue .

                       

                      The error can be cleared , if you remove  the certificate it won't pop up the error , but vcenter inventory was not showing there .

                       

                      I got this issue after a week from the installation , one week it was smooth .


                      You may check the NTP configuration  and try to migrate to another esxi host and check .

                      RAJESH RADHAKRISHNAN
                      VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA
                      https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335
                      Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
                      • 8. Re: VCSA 6.5 Web GUI Error
                        Enthusiast

                        There isn't enough information in the log snippet. The issue looks similar to vCenter – VdbODBCError – unique constraint pk_vpx_vm_virtual_device – fix | vMan where you can see possible workaround

                        • 10. Re: VCSA 6.5 Web GUI Error
                          misan Novice

                          I have done some more digging into this issue, and it seems to crop up if I upgrade the VCSA first and then update my hosts afterwards.   I recently performed an update of VCSA 6.5 to 6.5a and rebooted and the web GUI worked without issue.  Once I upgraded my ESXi hosts from 6.5 to 6.5a the problem occurred.

                           

                          Something seems to break with respect to communication between the VCSA and the hosts after upgrading the hosts. 

                          • 11. Re: VCSA 6.5 Web GUI Error
                            Danooo Novice

                            Hi All,

                            I have exact same problem. Two vCenter servers in linked mode with external PSCs. One day suddenly one of Vcenter servers stopped to respond. The issue that we have is not related to doubled DB entry as other suggested that.In logs I see exact same entries as misan.

                            Additionally I have checked time on both VCs and PSCs and replication between PSCs, all looks good. I have tried to re-generate certificate on PSC where affected VCs is connecting to, but this has failed. I have also performed upgrade to VCSA 6.5c build number 5400, no luck, still cannot start vpxd service. Hopefully this is a test lab, but I would like to avoid reinstallilng it as it runs dvswitch with bunch of VMs connected to it and Primary NSX controller for Cross-vCenter implementation. Hope that someone from VMware will take a look on this. At this point I do not recommend to go into production with VCSA 6.5.

                            • 12. Re: VCSA 6.5 Web GUI Error
                              Barra87 Novice

                              Hi All,

                               

                              today we had the same issue as described above.

                               

                              To remember:

                              Could not connect to one or more vCenter Server systems:

                              https://FQDN:443/sdk

                               

                              On VCSA Shell:

                              service-control --status vmware-vpxd shows "stopped"

                              service-control --start vmware-vpxd starts the service starts for a couple of seconds and stops again

                               

                              We contacted VMware Support and they analyzed the core dump logs (/var/core) and could figure out a problem with a single VM in our environment.

                              There was a problem with a network adapter on that VM (DB entry NULL)

                               

                              Solution that worked for us:

                               

                              Find VM

                              /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres (Login to postgres)

                              select v.id, v.local_file_name, h.IP_ADDRESS FROM VPX_VM v, VPX_NIC n, VPX_HOST h WHERE v.id=n.entity_id and v.host_id=h.id and n.network_name=''; (get affected VM)

                              select * from vpx_entity where id = '<YOURID>'; (get affected VM Name)

                               

                              Fix VM / vpxd (for our specific issue with database issue)

                              Remove Network Adapter, save config, readd Network Adapter

                              service-control --start vmware-vpxd

                              You should now be able to login to your vCenter.

                               

                              So the NULL Value in the database seem to be a really rare issue. In case of "Could not connect to.....:" just Find VM unregister it from vCenter and Reregister.

                               

                              Hope this helps somebody!

                               

                              Regards,

                              Gabriel

                              5 people found this helpful
                              • 13. Re: VCSA 6.5 Web GUI Error
                                idontwanttousethis Lurker

                                Had the same issue as you although I didn't interpret solution very clearly. You state to remove the network adapter from the VM which I wrongfully assumed you had to do from the database instead of the local webui/management of the esxi server. I went ahead and removed the problem VM from the inventory of the esxi server and a restart of vpxd on VCSA had everything back up and running instantly.

                                 

                                What broke this for me I believe is that I copied a VM from Workstation 12 onto a datastore and registered that VM via VCSA. Shortly after starting the problem VM, VCSA did freak out and have this problem. Now I'm in the process of using vCenter Converter with hopes I don't have to repeat this!

                                • 14. Re: VCSA 6.5 Web GUI Error
                                  aaronklep Novice

                                  That worked for me.  Thanks!

                                  1 2 Previous Next