9 Replies Latest reply on Mar 1, 2019 8:31 AM by ao89kuzmin

    vCenter crashed after updating

    nasco Novice

      I've updated vCenter to the available update from Update Manager via WebGUI. After rebooting, I cannot access anything. both https://myserver or https://myserver:5480 as well. I've tried to Google and change password of root account, DB password, postgres password, etc... but nothing helped. Below is some logs, pls help me to solve this because this is the only vCenter in our system.

      root@localhost [ ~ ]# service-control --status --all

      Running:

      applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

      Stopped:

      pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

      root@localhost [ ~ ]#

       

       

      root@localhost [ ~ ]# vi /var/log/vmware/vpxd/vpxd.log

      2018-06-17T17:30:15.861+07:00 error vpxd[7FDE06A68800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager

      2018-06-17T17:30:15.861+07:00 warning vpxd[7FDE06A68800] [Originator@6876 sub=VpxProfiler] Start [VpxdAuthorize::Start()] took 186628 ms

      2018-06-17T17:30:15.861+07:00 warning vpxd[7FDE06A68800] [Originator@6876 sub=VpxProfiler] ServerApp::Start [TotalTime] took 187161 ms

      2018-06-17T17:30:15.861+07:00 error vpxd[7FDE06A68800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down

      2018-06-17T17:30:15.861+07:00 info vpxd[7FDE06A68800] [Originator@6876 sub=SupportMgr] Wrote uptime information

      2018-06-17T17:31:39.507+07:00 info vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_CM_CALL_FAILED)

      2018-06-17T17:31:39.510+07:00 warning vpxd[7FDDD0B16700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fddec0341c0, h:12, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:18090'>>, e: system:111(Connection refused)

      2018-06-17T17:31:39.510+07:00 error vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused

      2018-06-17T17:31:39.510+07:00 warning vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused

      2018-06-17T17:31:39.569+07:00 warning vpxd[7FDDF8845700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fddd41381f0, h:26, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:18090'>>, e: system:111(Connection refused)

      2018-06-17T17:31:39.570+07:00 error vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused

      2018-06-17T17:31:39.570+07:00 warning vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused

      2018-06-17T17:31:39.645+07:00 warning vpxd[7FDDF8845700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fddd4127750, h:26, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:18090'>>, e: system:111(Connection refused)

      2018-06-17T17:31:39.645+07:00 error vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused

      2018-06-17T17:31:39.645+07:00 warning vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused

      2018-06-17T17:31:39.724+07:00 warning vpxd[7FDDD172E700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fddc83348a0, h:26, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:18090'>>, e: system:111(Connection refused)

      2018-06-17T17:31:39.729+07:00 error vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] [CisConnection]: ComponentManager->LoginByToken failed: Connection refused

      2018-06-17T17:31:39.729+07:00 warning vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Connection refused

      2018-06-17T17:31:39.729+07:00 warning vpxd[7FDDD3AF5700] [Originator@6876 sub=HostGateway] Ignoring exception during refresh of HostGateway cache: N7Vmacore15SystemExceptionE(Connection refused)

      --> [context]zKq7AVECAAAAAAHCfgAPdnB4ZAAAJGcrbGlidm1hY29yZS5zbwAA9B4bAD5vGABwKCsA5R0jAKgmJgCaKCYABUcmAOZSJgA36SMAo7QjAIq3IwDbvysBRHQAbGlicHRocmVhZC5zby4wAAItjA5saWJjLnNvLjYA[/context]

      2018-06-17T17:32:15.886+07:00 info vpxd[7FDE06A68800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

       

       

       

      root@localhost [ ~ ]# service --status-all

      /etc/init.d/netconsole: line 19: .: /etc/sysconfig/network: is a directory

      netconsole module not loaded

      ● vami-lighttp.service

         Loaded: loaded (/etc/init.d/vami-lighttp; disabled; vendor preset: enabled)

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      running (standalone: 1326)

      ● vmafdd.service - LSB: Start and Stop vmafd

         Loaded: loaded (/etc/rc.d/init.d/vmafdd; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmafdd.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf

         Active: active (exited) since Sun 2018-06-17 17:15:41 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1290 ExecStart=/etc/rc.d/init.d/vmafdd start (code=exited, status=0/SUCCESS)

          Tasks: 0

         Memory: 0B

            CPU: 0

       

      Jun 17 17:15:39 localhost.localdom systemd[1]: Starting LSB: Start and Stop vmafd...

      Jun 17 17:15:41 localhost.localdom vmafdd[1290]: Starting VMware afd Servicedone

      Jun 17 17:15:41 localhost.localdom systemd[1]: Started LSB: Start and Stop vmafd.

      running (standalone: 1420)

      ● vmcad.service - LSB: Start and Stop vmca

         Loaded: loaded (/etc/rc.d/init.d/vmcad; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmcad.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf

         Active: active (exited) since Sun 2018-06-17 17:15:43 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1385 ExecStart=/etc/rc.d/init.d/vmcad start (code=exited, status=0/SUCCESS)

          Tasks: 0

         Memory: 0B

            CPU: 0

       

      Jun 17 17:15:41 localhost.localdom systemd[1]: Starting LSB: Start and Stop vmca...

      Jun 17 17:15:43 localhost.localdom vmcad[1385]: Starting VMware Certificate Servicedone

      Jun 17 17:15:43 localhost.localdom systemd[1]: Started LSB: Start and Stop vmca.

      Error: LW_ERROR_NO_SUCH_SERVICE (41204)

      No service with the specified name exists

      ● vmcam.service - LSB: Start and Stop vmcam

         Loaded: loaded (/etc/rc.d/init.d/vmcam; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmcam.service.d

                 └─dependency.conf

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      running (standalone: 1424)

      ● vmdird.service - LSB: Start and Stop vmdir

         Loaded: loaded (/etc/rc.d/init.d/vmdird; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmdird.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf

         Active: active (exited) since Sun 2018-06-17 17:15:45 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1387 ExecStart=/etc/rc.d/init.d/vmdird start (code=exited, status=0/SUCCESS)

          Tasks: 0

         Memory: 0B

            CPU: 0

       

      Jun 17 17:15:41 localhost.localdom systemd[1]: Starting LSB: Start and Stop vmdir...

      Jun 17 17:15:45 localhost.localdom systemd[1]: Started LSB: Start and Stop vmdir.

      Jun 17 17:15:45 localhost.localdom vmdird[1387]: Starting VMware Directory Servicedone

      running (standalone: 1593)

      ● vmdnsd.service - LSB: Start and Stop vmdns

         Loaded: loaded (/etc/rc.d/init.d/vmdnsd; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmdnsd.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dependency.conf

         Active: active (exited) since Sun 2018-06-17 17:15:46 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1569 ExecStart=/etc/rc.d/init.d/vmdnsd start (code=exited, status=0/SUCCESS)

          Tasks: 0

         Memory: 0B

            CPU: 0

       

      Jun 17 17:15:45 localhost.localdom systemd[1]: Starting LSB: Start and Stop vmdns...

      Jun 17 17:15:46 localhost.localdom vmdnsd[1569]: Starting VMware Domain Name Servicedone

      Jun 17 17:15:46 localhost.localdom systemd[1]: Started LSB: Start and Stop vmdns.

      msg_unused

      ● vmware-imagebuilder.service - LSB: VMware Image Builder Server

         Loaded: loaded (/etc/rc.d/init.d/vmware-imagebuilder; bad; vendor preset: enabled)

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      watchdog  is not running

      webserver is not running

      logdumper is not running

      ● vmware-netdumper.service - LSB: VMware network core dump server

         Loaded: loaded (/etc/rc.d/init.d/vmware-netdumper; bad; vendor preset: enabled)

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      ensure environment variables are set

      ensure environment variables are set

      Checking for service vmware-psc-clientrunning

      ● vmware-psc-client.service - LSB: VMWare PSC Client

         Loaded: loaded (/etc/rc.d/init.d/vmware-psc-client; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmware-psc-client.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf

         Active: active (running) since Sun 2018-06-17 17:15:45 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 847 ExecStart=/etc/rc.d/init.d/vmware-psc-client start (code=exited, status=0/SUCCESS)

          Tasks: 69

         Memory: 479.7M

            CPU: 44.479s

         CGroup: /system.slice/vmware-psc-client.service

                 ├─934 vmware-psc-client -procname vmware-psc-client -home /usr/java/jre-vmware -server -Xmx160m -XX:CompressedClassSpaceSize=128m -Xss256k -XX:ParallelGCThreads=1 -pidfile /var/log/vmware/psc-client/tcserver.pid -Djava.util.logging.config.file=/usr/lib/vmware-psc-client/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.map.althashing.threshold=512 -Xss228K -Xmx512m -XX:MaxPermSize=128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/vmware/psc-client/ -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/var/log/vmware/psc-client/gclogFile -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=5M -XX:ErrorFile=/var/log/vmware/psc-client/hs_err_lotus_pid%p.log -ea -Djava.security.properties=/etc/vmware/java/vmware-override-java.security -Djava.endorsed.dirs=/var/opt/apache-tomcat-8.5.13/endorsed -classpath /usr/lib/vmware/common-jars/httpasyncclient-4.0.1.jar:/var/opt/apache-tomcat-8.5.13/bin/bootstrap.jar:/var/opt/apache-tomcat-8.5.13/bin/tomcat-juli.jar -Dcatalina.base=/usr/lib/vmware-psc-client -Dcatalina.home=/var/opt/apache-tomcat-8.5.13 -Djava.io.tmpdir=/usr/lib/vmware-psc-client/temp org.apache.catalina.startup.Bootstrap start

                 └─936 vmware-psc-client -procname vmware-psc-client -home /usr/java/jre-vmware -server -Xmx160m -XX:CompressedClassSpaceSize=128m -Xss256k -XX:ParallelGCThreads=1 -pidfile /var/log/vmware/psc-client/tcserver.pid -Djava.util.logging.config.file=/usr/lib/vmware-psc-client/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.map.althashing.threshold=512 -Xss228K -Xmx512m -XX:MaxPermSize=128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/vmware/psc-client/ -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/var/log/vmware/psc-client/gclogFile -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=5M -XX:ErrorFile=/var/log/vmware/psc-client/hs_err_lotus_pid%p.log -ea -Djava.security.properties=/etc/vmware/java/vmware-override-java.security -Djava.endorsed.dirs=/var/opt/apache-tomcat-8.5.13/endorsed -classpath /usr/lib/vmware/common-jars/httpasyncclient-4.0.1.jar:/var/opt/apache-tomcat-8.5.13/bin/bootstrap.jar:/var/opt/apache-tomcat-8.5.13/bin/tomcat-juli.jar -Dcatalina.base=/usr/lib/vmware-psc-client -Dcatalina.home=/var/opt/apache-tomcat-8.5.13 -Djava.io.tmpdir=/usr/lib/vmware-psc-client/temp org.apache.catalina.startup.Bootstrap start

       

      Jun 17 17:15:23 localhost.localdom systemd[1]: Starting LSB: VMWare PSC Client...

      Jun 17 17:15:24 localhost.localdom vmware-psc-client[847]: ensure environment variables are set

      Jun 17 17:15:45 localhost.localdom vmware-psc-client[847]: Starting vmware-psc-client ..................done

      Jun 17 17:15:45 localhost.localdom systemd[1]: Started LSB: VMWare PSC Client.

      Checking for /usr/bin/rbd-watchdog-linux: unused

      ● vmware-rbd-watchdog.service - LSB: VMware AutoDeploy

         Loaded: loaded (/etc/rc.d/init.d/vmware-rbd-watchdog; bad; vendor preset: enabled)

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      VMware Appliance Stats Monitor is running.

      VMware HTTP Reverse Proxy is running.

      ensure environment variables are set

      ensure environment variables are set

      Checking for service vmware-stsd

      running

      ● vmware-stsd.service - LSB: VMWare Security Token Service

         Loaded: loaded (/etc/rc.d/init.d/vmware-stsd; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmware-stsd.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf

         Active: active (running) since Sun 2018-06-17 17:16:34 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1731 ExecStart=/etc/rc.d/init.d/vmware-stsd start (code=exited, status=0/SUCCESS)

          Tasks: 97

         Memory: 602.2M

            CPU: 1min 22.519s

         CGroup: /system.slice/vmware-stsd.service

                 ├─1755 vmware-stsd -procname vmware-stsd -home /usr/java/jre-vmware -server -pidfile /var/log/vmware/sso/tcserver.pid -errfile /var/log/vmware/sso/utils/vmware-stsd.err -Djava.util.logging.config.file=/usr/lib/vmware-sso/vmware-sts/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx378m -XX:CompressedClassSpaceSize=160m -Xss256k -XX:ParallelGCThreads=1 -XX:MaxPermSize=160m -XX:ThreadStackSize=228k -Djdk.map.althashing.threshold=512 -Xss228K -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/vmware/sso/ -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/var/log/vmware/sso/gclogFile -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=5M -XX:ErrorFile=/var/log/vmware/sso/hs_err_sts_pid%p.log -XX:HeapDumpPath=/var/log/vmware/sso/ -Xmx160m -XX:MaxPermSize=160m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1 -ea -XX:CompressedClassSpaceSize=128m -ea -Djava.security.egd=file:/dev/./urandom -Djava.security.properties=/usr/lib/vmware-sso/vmware-sts/conf/vmware-identity-override-java.security -Djava.ext.dirs=/usr/java/jre-vmware/lib/ext:/usr/java/packages/lib/ext:/opt/vmware/jre_ext -Djava.endorsed.dirs=/var/opt/apache-tomcat-8.5.13/endorsed -classpath /var/opt/apache-tomcat-8.5.13/bin/bootstrap.jar:/var/opt/apache-tomcat-8.5.13/bin/tomcat-juli.jar -Dcatalina.base=/usr/lib/vmware-sso/vmware-sts -Dcatalina.home=/var/opt/apache-tomcat-8.5.13 -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true -Djava.io.tmpdir=/usr/lib/vmware-sso/vmware-sts/temp org.apache.catalina.startup.Bootstrap start

                 └─1758 vmware-stsd -procname vmware-stsd -home /usr/java/jre-vmware -server -pidfile /var/log/vmware/sso/tcserver.pid -errfile /var/log/vmware/sso/utils/vmware-stsd.err -Djava.util.logging.config.file=/usr/lib/vmware-sso/vmware-sts/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx378m -XX:CompressedClassSpaceSize=160m -Xss256k -XX:ParallelGCThreads=1 -XX:MaxPermSize=160m -XX:ThreadStackSize=228k -Djdk.map.althashing.threshold=512 -Xss228K -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/vmware/sso/ -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/var/log/vmware/sso/gclogFile -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=5M -XX:ErrorFile=/var/log/vmware/sso/hs_err_sts_pid%p.log -XX:HeapDumpPath=/var/log/vmware/sso/ -Xmx160m -XX:MaxPermSize=160m -XX:ThreadStackSize=256 -XX:ParallelGCThreads=1 -ea -XX:CompressedClassSpaceSize=128m -ea -Djava.security.egd=file:/dev/./urandom -Djava.security.properties=/usr/lib/vmware-sso/vmware-sts/conf/vmware-identity-override-java.security -Djava.ext.dirs=/usr/java/jre-vmware/lib/ext:/usr/java/packages/lib/ext:/opt/vmware/jre_ext -Djava.endorsed.dirs=/var/opt/apache-tomcat-8.5.13/endorsed -classpath /var/opt/apache-tomcat-8.5.13/bin/bootstrap.jar:/var/opt/apache-tomcat-8.5.13/bin/tomcat-juli.jar -Dcatalina.base=/usr/lib/vmware-sso/vmware-sts -Dcatalina.home=/var/opt/apache-tomcat-8.5.13 -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true -Djava.io.tmpdir=/usr/lib/vmware-sso/vmware-sts/temp org.apache.catalina.startup.Bootstrap start

       

      Jun 17 17:15:58 localhost.localdom systemd[1]: Starting LSB: VMWare Security Token Service...

      Jun 17 17:15:58 localhost.localdom vmware-stsd[1731]: ensure environment variables are set

      Jun 17 17:16:34 localhost.localdom vmware-stsd[1731]: Starting vmware-stsd.  Request for http://localhost:7080/afd failed after 1 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 1 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached  Increasing the timeout window to 2 seconds..  Request for http://localhost:7080/afd failed after 2 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached  Increasing the timeout window to 3 seconds.........done

      Jun 17 17:16:34 localhost.localdom systemd[1]: Started LSB: VMWare Security Token Service.

      ensure environment variables are set

      ensure environment variables are set

      Checking for VMware STS IDM Server ...running

      ● vmware-sts-idmd.service - LSB: Start and Stop vmware-sts-idmd

         Loaded: loaded (/etc/rc.d/init.d/vmware-sts-idmd; bad; vendor preset: enabled)

        Drop-In: /etc/systemd/system/vmware-sts-idmd.service.d

                 └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf

         Active: active (running) since Sun 2018-06-17 17:15:58 ICT; 7h ago

           Docs: man:systemd-sysv-generator(8)

        Process: 1568 ExecStart=/etc/rc.d/init.d/vmware-sts-idmd start (code=exited, status=0/SUCCESS)

          Tasks: 24

         Memory: 155.1M

            CPU: 1min 34.356s

         CGroup: /system.slice/vmware-sts-idmd.service

                 ├─1595 vmware-sts-idmd -procname vmware-sts-idmd -wait 120 -server -Xmx168m -XX:CompressedClassSpaceSize=160m -Xss256k -XX:ParallelGCThreads=1 -XX:ErrorFile=/var/log/vmware/sso/hs_err_idm_pid%p.log -XX:HeapDumpPath=/var/log/vmware/sso -home /usr/java/jre-vmware -pidfile /var/run/vmware-sts-idmd.pid -errfile /var/log/vmware/sso/vmware-sts-idmd.err -cp /opt/vmware/lib64/commons-lang-2.5.jar:/opt/vmware/lib64/commons-logging-1.1.1.jar:/opt/vmware/lib64/log4j-api-2.2.jar:/opt/vmware/lib64/log4j-1.2-api-2.0.2.jar:/opt/vmware/lib64/log4j-core-2.2.jar:/opt/vmware/lib64/log4j-1.2.12rsa-1.jar:/opt/vmware/lib64/jna.jar:/opt/vmware/lib64/commons-daemon-1.0.8.jar:/opt/vmware/lib64/bcpkix-jdk15on-147.jar:/opt/vmware/lib64/bcprov-jdk15on-147.jar:/opt/vmware/lib64/vmware-identity-platform.jar:/opt/vmware/lib64/vmware-identity-idm-interface.jar:/opt/vmware/lib64/vmware-identity-idm-server.jar:/opt/vmware/lib64/vmware-identity-diagnostics.jar:/opt/vmware/lib64/bcprov-jdk16-145.jar:/opt/vmware/lib64/authentication-framework.jar:/opt/vmware/lib64/afd-heartbeat-service.jar:/opt/vmware/lib64/httpclient-4.5.1.jar:/opt/vmware/lib64/authapi-8.5.jar:/opt/vmware/lib64/cryptoj-6.1.jar:/opt/vmware/lib64/xmlsec-1.5.5.jar:/opt/vmware/lib64 -Djava.rmi.server.codebase=file:////opt/vmware/lib64/vmware-identity-idm-interface.jar file:////opt/vmware/lib64/vmware-identity-idm-server.jar -Djava.security.policy=/opt/vmware/share/config/server.policy -Dvmware.log.dir=/var/log/vmware/sso -Dlog4j.configurationFile=file:///opt/vmware/share/config/log4j2.xml com.vmware.identity.idm.server.IdmServer

                 └─1596 vmware-sts-idmd -procname vmware-sts-idmd -wait 120 -server -Xmx168m -XX:CompressedClassSpaceSize=160m -Xss256k -XX:ParallelGCThreads=1 -XX:ErrorFile=/var/log/vmware/sso/hs_err_idm_pid%p.log -XX:HeapDumpPath=/var/log/vmware/sso -home /usr/java/jre-vmware -pidfile /var/run/vmware-sts-idmd.pid -errfile /var/log/vmware/sso/vmware-sts-idmd.err -cp /opt/vmware/lib64/commons-lang-2.5.jar:/opt/vmware/lib64/commons-logging-1.1.1.jar:/opt/vmware/lib64/log4j-api-2.2.jar:/opt/vmware/lib64/log4j-1.2-api-2.0.2.jar:/opt/vmware/lib64/log4j-core-2.2.jar:/opt/vmware/lib64/log4j-1.2.12rsa-1.jar:/opt/vmware/lib64/jna.jar:/opt/vmware/lib64/commons-daemon-1.0.8.jar:/opt/vmware/lib64/bcpkix-jdk15on-147.jar:/opt/vmware/lib64/bcprov-jdk15on-147.jar:/opt/vmware/lib64/vmware-identity-platform.jar:/opt/vmware/lib64/vmware-identity-idm-interface.jar:/opt/vmware/lib64/vmware-identity-idm-server.jar:/opt/vmware/lib64/vmware-identity-diagnostics.jar:/opt/vmware/lib64/bcprov-jdk16-145.jar:/opt/vmware/lib64/authentication-framework.jar:/opt/vmware/lib64/afd-heartbeat-service.jar:/opt/vmware/lib64/httpclient-4.5.1.jar:/opt/vmware/lib64/authapi-8.5.jar:/opt/vmware/lib64/cryptoj-6.1.jar:/opt/vmware/lib64/xmlsec-1.5.5.jar:/opt/vmware/lib64 -Djava.rmi.server.codebase=file:////opt/vmware/lib64/vmware-identity-idm-interface.jar file:////opt/vmware/lib64/vmware-identity-idm-server.jar -Djava.security.policy=/opt/vmware/share/config/server.policy -Dvmware.log.dir=/var/log/vmware/sso -Dlog4j.configurationFile=file:///opt/vmware/share/config/log4j2.xml com.vmware.identity.idm.server.IdmServer

       

      Jun 17 17:15:45 localhost.localdom systemd[1]: Starting LSB: Start and Stop vmware-sts-idmd...

      Jun 17 17:15:45 localhost.localdom vmware-sts-idmd[1568]: ensure environment variables are set

      Jun 17 17:15:58 localhost.localdom vmware-sts-idmd[1568]: Starting VMware STS IDM Server ...done

      Jun 17 17:15:58 localhost.localdom systemd[1]: Started LSB: Start and Stop vmware-sts-idmd.

      msg_unused

      ● vmware-updatemgr.service - LSB: VMware Update Manager Server

         Loaded: loaded (/etc/rc.d/init.d/vmware-updatemgr; bad; vendor preset: enabled)

         Active: inactive (dead)

           Docs: man:systemd-sysv-generator(8)

      vAPI Endpoint is not running.

      <30>Jun 18 00:28:44 vmware-vcha: stopped

      pg_ctl: server is running (PID: 2545)

      /opt/vmware/vpostgres/9.4/bin/postgres "-D" "/storage/db/vpostgres"

      vmware-vpxd is stopped

      VMware vService Manager is not running.

      root@localhost [ ~ ]#

       

      I've tried many ways but cannot start service to access GUI

        • 1. Re: vCenter crashed after updating
          daphnissov Guru
          Community WarriorsvExpert

          So how did you update this vCenter again? What version is this? From the output you provided, it appears that either a config got wiped out or it had issues to start with. The hostname should never be "localhost" which is what it shows. Were you wise and snapshot this vCenter before proceeding with the update? What was the source version and what was the destination version?

          • 2. Re: vCenter crashed after updating
            nasco Novice

            Unfortunately, I did not take snapshot before updating. The current version of vCenter is 6.5.0 4944578 build and the update available as I remember is 6.5 update 2 build 8307201

            The hostname is not localhost, I don't know why the hostname is like that now. Honestly, I've never been logged in to CLI after installing vCenter. Everything is ok until updating new version yesterday

            • 3. Re: vCenter crashed after updating
              daphnissov Guru
              vExpertCommunity Warriors

              Your options are either restore from backup or open a support case to see if they can fix what sounds like a partial upgrade situation.

              • 4. Re: vCenter crashed after updating
                nasco Novice

                Can I disassociate our 2 ESXi servers from vCenter via vSphere Client and install a new vCenter? Is there any risk from this action?

                • 5. Re: vCenter crashed after updating
                  daphnissov Guru
                  vExpertCommunity Warriors

                  You don't need to, unless they're using a distributed switch. In that case, there's not much you can do without vCenter being available.

                  • 6. Re: vCenter crashed after updating
                    nasco Novice

                    Hi daphnissov,

                    I don't understand what you mean "don't need to". My situation now is running without vCenter and I've not configured distributied switch before in the past. I need vCenter to manage some advance task which cannot be done via vSphere Client. And I'm so tired of troubleshooting to fix vCenter working again. Can I disassociate our ESXi servers with vCenter (via vSphere Client) and create a new vCenter and then associate the ESXi servers with the new vCenter?

                    • 7. Re: vCenter crashed after updating
                      daphnissov Guru
                      Community WarriorsvExpert

                      As I said, you do not need to disassociate ESXi hosts with a now-defunct vCenter. You simply join them to a new vCenter. If you're not using a vDS then there's little concern. This is all assuming you can't/don't want to open a support case to rectify your problem, of course.

                      • 8. Re: vCenter crashed after updating
                        DukeSniper Lurker

                        I had the same problem when updating to the latest hotfix. Update crashed with some unidentified error, and after reboot, both VAMI and VCenter were dead. I managed to at least revive the VCenter part by manually applying the update from the integrated shell (software-update install --staged) and rebooting the Appliance. I am still figuring out how to revive the VAMI UI.

                         

                        UPDATE: It's all up and running again. What I had to do was find out which services were missing, unmask and then restart those:

                         

                        for servicename in $(systemctl list-unit-files | grep masked | tr -s ' ' | cut -f 1 -d ' ' | grep \.service$ | cut -d '.' -f 1) ; do systemctl unmask ${servicename} ; systemctl enable ${servicename} ; service ${servicename} start ; done

                         

                        now everything seems to be working again, even surviving reboots. However, I'm quite certain that there are still some things that are f*cked, so I'm gonna try to backup and restore the VCSA now.

                         

                        UPDATE 2: still struggling. Integrated backup is complaining about some services not running.

                        • 9. Re: vCenter crashed after updating
                          ao89kuzmin Lurker

                          Hi!

                           

                          I had same problem. I upgraded my vcenter appliance 6.5U2 from Build 9451637 to Build 11347054.

                           

                           

                          After rebooting, I saw this error:

                          (https://myvcenter.local) - 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055d5a4458ea0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

                           

                           

                          I checked vCenter Server Appliance Health(https://myvcenter.local:5480) and finded that: "Single Sign-On - Status Not Running"

                           

                           

                          I checked vcenter appliance services (service --status-all) and finded this:

                          ● vmware-stsd.service - LSB: VMWare Security Token Service

                          ....

                             Active: failed (Result: timeout) since Fri 2019-03-01 19:47:49 +05; 2min 9s ago

                          ....

                          Failed to start LSB: VMWare Security Token Service.

                          vmware-stsd.service: Unit entered failed state.

                          vmware-stsd.service: Failed with result 'timeout'.

                          Increasing the timeout window to 10 seconds..  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.  Request for http://localhost:7080/afd failed after 10 seconds. Status: /usr/bin/curl status. Response: 000. Host: ;; connection timed out; no servers could be reached.

                          ....

                          Starting LSB: Start and Stop vmware-sts-idmd...

                          vmware-sts-idmd[2953]: ensure environment variables are set

                          vmware-sts-idmd[2953]: Starting VMware STS IDM Server ...failed

                          vmware-sts-idmd.service: Control process exited, code=exited status=1

                          Failed to start LSB: Start and Stop vmware-sts-idmd.

                          vmware-sts-idmd.service: Unit entered failed state.

                          vmware-sts-idmd.service: Failed with result 'exit-code'.

                           

                           

                          Then i tried restarting services, but it did't help me:

                          service-control --stop --all

                          service-control --start --all

                           

                           

                          Then i tried to find problem with LDAP and SSO service.

                          I checked the log (vmware-sts-idmd.log) and finded it:

                          cat /var/log/vmware/sso/localhost.2019-03-01.log

                          ...

                          Caused by: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

                          LDAP error [code: 49]

                                  at com.vmware.identity.interop.ldap.LdapErrorChecker$28.RaiseLdapError(LdapErrorChecker.java:413)

                                  at com.vmware.identity.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:1090)

                                  at com.vmware.identity.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1241)

                                  at com.vmware.identity.interop.ldap.OpenLdapClientLibrary.ldap_sasl_srp_bind_s(OpenLdapClientLibrary.java:769)

                                  at com.vmware.identity.interop.ldap.LdapConnection.bindSaslSrpConnection(LdapConnection.java:670)

                                  at com.vmware.vim.lookup.impl.LdapStorage$LdapQuery.execute(LdapStorage.java:2030)

                          ...

                           

                           

                          Then:

                          cat /var/log/vmware/sso/vmware-sts-idmd.log

                          ...

                          com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

                          Caused by: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

                          Caused by: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

                          ...

                           

                           

                          Then:

                          cat /var/log/vmware/vmdird/vmdird-syslog.log

                          ...

                          VmDirSendLdapResult: Request (Bind), Error (49), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)

                          Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=vcenter.vsphere.local,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL

                          SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

                          ...

                           

                           

                          The problem is invalid credentials.

                           

                           

                          I finded this article and it helped me:

                          https://kb.vmware.com/s/article/2147280