7 Replies Latest reply on Jun 18, 2020 8:07 AM by Sukanyad

    vCenter 6.7 down

    techwater Lurker

      I had to reboot this server for a maintenance.  since then the services are not coming up.  Is it because of /storage/archive being 100% full ?

       

      root@XXXXXXX [ ~ ]# service-control --status

      Running:

      lwsmd vmafdd vmcad vmdird vmdnsd vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-eam vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vsphere-client vsphere-ui

      Stopped:

      applmgmt pschealth vmcam vmonapi vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsan-dps

       

      2020-03-09T14:51:41.420Z Done running command

      2020-03-09T14:51:41.421Z Successfully started service vmware-vmon

      2020-03-09T14:51:41.421Z Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']

      2020-03-09T14:51:41.430Z Done running command

      Successfully started service vmware-vmon

      2020-03-09T14:51:41.431Z Running command: ['/usr/lib/vmware-vmon/vmon-cli', '--batchstart', 'ALL']

      2020-03-09T14:58:49.248Z Done running command

      Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start topologysvc, vmonapi, applmgmt, cm services. Error: Operation timed out

       

       

      2020-05-15T05:13:14.486-04:00 | ERROR | state-manager1 | DefaultStateManager | Could not initialize endpoint runtime state.

      com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)

              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)

              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

      Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

              at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:292)

              at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:206)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)

              at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)

              ... 10 more

      Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

              at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)

              at com.sun.proxy.$Proxy38.list(Unknown Source)

              at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:288)

              ... 13 more

       

      # df -h

      Filesystem                                Size  Used Avail Use% Mounted on

      devtmpfs                                  9.8G     0  9.8G   0% /dev

      tmpfs                                     9.8G  864K  9.8G   1% /dev/shm

      tmpfs                                     9.8G  684K  9.8G   1% /run

      tmpfs                                     9.8G     0  9.8G   0% /sys/fs/cgroup

      /dev/sda3                                  11G  6.1G  4.0G  61% /

      tmpfs                                     9.8G  912K  9.8G   1% /tmp

      /dev/mapper/dblog_vg-dblog                 15G  470M   14G   4% /storage/dblog

      /dev/mapper/core_vg-core                   50G  831M   46G   2% /storage/core

      /dev/mapper/db_vg-db                      9.8G  324M  8.9G   4% /storage/db

      /dev/sda1                                 120M   34M   78M  31% /boot

      /dev/mapper/netdump_vg-netdump            985M  1.3M  916M   1% /storage/netdump

      /dev/mapper/log_vg-log                    9.8G  6.6G  2.7G  71% /storage/log

      /dev/mapper/autodeploy_vg-autodeploy      9.8G   23M  9.2G   1% /storage/autodeploy

      /dev/mapper/imagebuilder_vg-imagebuilder  9.8G   23M  9.2G   1% /storage/imagebuilder

      /dev/mapper/archive_vg-archive             50G   47G   20M 100% /storage/archive

      /dev/mapper/updatemgr_vg-updatemgr         99G   64M   94G   1% /storage/updatemgr

      /dev/mapper/seat_vg-seat                   25G  1.1G   23G   5% /storage/seat

       

      2020-05-26T08:13:43.671-04:00 warning rhttpproxy[79242] [Originator@6876 sub=Proxy Req 04617 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

      2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f4ff80059e0, h:19, <TCP '127.0.0.1 : 46684'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

      2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04618 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

      2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50380166f0, h:21, <TCP '127.0.0.1 : 46706'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

      2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04620 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

      2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50500134d0, h:19, <TCP '127.0.0.1 : 46850'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

      2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Proxy Req 04642 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

        • 1. Re: vCenter 6.7 down
          daphnissov Guru
          Community WarriorsvExpert

          For break/fix issues, please open a support case with VMware.

          • 2. Re: vCenter 6.7 down
            Vijay2027 Expert
            vExpert

            Looks like sts cert is expired. Log a SR with GSS please.

            • 3. Re: vCenter 6.7 down
              techwater Lurker

              I think the cert is fine

               

              # python checksts.py

               

               

              2 VALID CERTS

              ================

               

               

                      LEAF CERTS:

               

               

                      [] Certificate 58:8D:B0:E5:BC:B0:48:04:76:03:83:43:09:0D:E5:23:AE:F5:86:02 will expire in 2089 days (6 years).

               

               

                      ROOT CERTS:

               

               

                      [] Certificate 8B:E7:0F:8A:CF:B0:65:74:37:D5:39:0B:1A:2E:FD:E3:61:CD:8E:66 will expire in 2089 days (6 years).

               

               

              0 EXPIRED CERTS

              ================

               

               

                      LEAF CERTS:

               

               

                      None

               

               

                      ROOT CERTS:

               

               

                      None

              • 4. Re: vCenter 6.7 down
                techwater Lurker

                almost every log file on the server has some error.  Can somebody please guide me in which sequence I see go through the various log files.

                • 5. Re: vCenter 6.7 down
                  techwater Lurker

                  there is nothing listening on 8089

                   

                  2020-05-26T08:13:43.671-04:00 warning rhttpproxy[79242] [Originator@6876 sub=Proxy Req 04617 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

                  2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f4ff80059e0, h:19, <TCP '127.0.0.1 : 46684'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

                  2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04618 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

                  2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50380166f0, h:21, <TCP '127.0.0.1 : 46706'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

                  2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04620 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

                  2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50500134d0, h:19, <TCP '127.0.0.1 : 46850'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)

                  2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Proxy Req 04642 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

                  • 6. Re: vCenter 6.7 down
                    daphnissov Guru
                    Community WarriorsvExpert

                    Support case. Open one.

                    • 7. Re: vCenter 6.7 down
                      Sukanyad Enthusiast
                      VMware Employees

                      /storage/archive being 100% full This a know issue and doesnt affect any vcenter operation .so its not the cause.