3 Replies Latest reply on Sep 11, 2020 12:21 PM by LLCMEP

    VCenter error start

    LLCMEP Lurker

      После установки обновлений на сервер 2016 перестал запускать VCenter. При подключение по веб выдает следующее:

      503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000001b07949a9f0] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

       

      На самом сервер при проверке сервисов выдает следующее:

      C:\Program Files\VMware\vCenter Server\bin>service-control --status

      Running:

      EsxAgentManager VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS VServiceManager rhttpproxy vPostgres vapiEndpoint vmon vmonapi vmsyslogcollector vmware-cis-config vmware-license vmware-perfcharts vmwareServiceControlAgent vpxd-svcs vsphere-ui vspherewebclientsvc

      Stopped:

      VMWareCAMService content-library mbcs vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vpxd

      StartPending:

      vimPBSM vsan-health

       

      При попытке запуска:

      C:\Program Files\VMware\vCenter Server\bin>service-control --start --all

      Operation not cancellable. Please wait for it to finish...

      Performing start operation on service vmware-cis-config...

      Successfully started service vmware-cis-config

      Performing start operation on service VMWareAfdService...

      Successfully started service VMWareAfdService

      Performing start operation on service VMWareDirectoryService...

      Successfully started service VMWareDirectoryService

      Performing start operation on service VMWareCertificateService...

      Successfully started service VMWareCertificateService

      Performing start operation on service VMwareIdentityMgmtService...

      Successfully started service VMwareIdentityMgmtService

      Performing start operation on service VMwareSTS...

      Successfully started service VMwareSTS

      Performing start operation on service VMwareDNSService...

      Successfully started service VMwareDNSService

      Performing start operation on profile: ALL...

      Successfully started service vmon

      Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, content-library, vpxd, vsan-health services. Error: Operation timed out

       

      Ребят помогите куда копать пожалуйста!

        • 1. Re: VCenter error start
          Finikiez Master
          vExpert

          Добрый день!

           

          Насколько большая инфраструктура в плане количество хостов\виртуальных машин\смежных сервисов, которые используют vcenter?

           

          В целом vcenter на ОС Windows умер и всем нужно переходить на VCSA (linux-based appliance).

          Поэтому в крайнем случае рекомендуется установить новый аплайнс и переподключить хосты к нему.

           

          Если говорить про траблшутинг, то надо начать с лога vpxd (основной сервис) и прочитать в конце текстового файла, почему он не запускается.

          Либо откатить накаченные обновления как первый шаг.

          • 2. Re: VCenter error start
            Finikiez Master
            vExpert

            Плюс проверить, не сдохли ли у вас сертификаты.

             

            Из последних самых известных проблем - https://kb.vmware.com/s/article/79248

            • 3. Re: VCenter error start
              LLCMEP Lurker

              Так проблема в том что 6 хостов и около 300 виртуалок. Бекап сервака восстановили до момента обновлений. Не помогает. В логах vphd-svcs постоянно скачют warnings:

              2020-09-11T22:13:21.096+03:00 [Thread-9  WARN  com.vmware.cis.server.util.VpxdClient  opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

              2020-09-11T22:13:21.097+03:00 [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 31

              2020-09-11T22:13:43.170+03:00 [Thread-9  WARN  com.vmware.cis.server.util.VpxdClient  opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

              2020-09-11T22:13:43.171+03:00 [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 32

              2020-09-11T22:14:05.244+03:00 [Thread-9  WARN  com.vmware.cis.server.util.VpxdClient  opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

              2020-09-11T22:14:05.245+03:00 [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 33

              2020-09-11T22:14:27.319+03:00 [Thread-9  WARN  com.vmware.cis.server.util.VpxdClient  opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

              2020-09-11T22:14:27.320+03:00 [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 34

              2020-09-11T22:14:49.393+03:00 [Thread-9  WARN  com.vmware.cis.server.util.VpxdClient  opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

              2020-09-11T22:14:49.394+03:00 [Thread-9  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 35

               

              А вот vphd-svcs-access сыпиться:

              [11/Sep/2020:19:33:59 +0300] 0:0:0:0:0:0:0:1 6055 tomcat-exec-170 200 "Apache-HttpClient/4.5.3 (Java/1.8.0_212)" GET /invsvc/kv-resource HTTP/1.1

              [11/Sep/2020:19:33:59 +0300] 0:0:0:0:0:0:0:1 2763 tomcat-exec-138 200 "Apache-HttpClient/4.5.3 (Java/1.8.0_212)" GET /invsvc/authz-resource HTTP/1.1

              [11/Sep/2020:19:33:59 +0300] 0:0:0:0:0:0:0:1 13089 tomcat-exec-260 200 "Apache-HttpClient/4.5.3 (Java/1.8.0_212)" GET /invsvc/invsvc-resource HTTP/1.1

              [11/Sep/2020:19:38:10 +0300] 127.0.0.1 595 tomcat-exec-284 200 "vAPI/2.100.0 Java/1.8.0_212 (Windows Server 2016; 10.0; amd64)" POST /invsvc/vapi HTTP/1.1

              [11/Sep/2020:22:10:41 +0300] 127.0.0.1 595 tomcat-exec-276 200 "vAPI/2.100.0 Java/1.8.0_212 (Windows Server 2016; 10.0; amd64)" POST /invsvc/vapi HTTP/1.1

              [11/Sep/2020:22:14:43 +0300] 127.0.0.1 595 tomcat-exec-268 200 "vAPI/2.100.0 Java/1.8.0_212 (Windows Server 2016; 10.0; amd64)" POST /invsvc/vapi HTTP/1.1

              [11/Sep/2020:22:18:44 +0300] 127.0.0.1 595 tomcat-exec-256 200 "vAPI/2.100.0 Java/1.8.0_212 (Windows Server 2016; 10.0; amd64)" POST /invsvc/vapi HTTP/1.1

              [11/Sep/2020:22:19:31 +0300] 0:0:0:0:0:0:0:1 304 tomcat-exec-252 200 "Python-urllib/3.5" GET /invsvc/invsvc-health HTTP/1.1