3 Replies Latest reply on Dec 2, 2019 7:25 AM by msripada

    Update to 6.7u3 failed

    KasperKWJ Lurker

      Hi all,

       

      Recently I've been trying to patch some ESXi servers to 6.7u3. The patching is successful. However after the server reboot, the ESXi is unable to exit maintenance mode. vCenter always throw the error message "Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections" whenever I try to exit maintenance mode.

       

      On the esxi hostd.log:

       

      warning hostd[2101206] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000008c848b1d50, h:21, <TCP '127.0.0.1 : 57259'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

       

      error hostd[2101206] [Originator@6876 sub=vmomi.soapStub[1]] Resetting stub adapter for server <cs p:0000008c834451e0, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

      --> [context]zKq7AVICAgAAACEo5QAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

       

      Been trying to figure out what port 9095 is used for, but to no avail. I'm thinking it might be related to vSAN. We are using VMware vSAN on the cluster. Can anyone help with this issue?

        • 1. Re: Update to 6.7u3 failed
          msripada Expert
          vExpert

          so post ESXi host upgrade to 6.7 update 3, host is disconnected..

          can you share the output of /etc/init.d/hostd status and /etc/init.d/vpxa status?

           

          You can collect the log bundle using shell and can share us.. mostly hostd failed to start or unresponsive but need log bundle to validate

           

          thanks,

          MS

          • 2. Re: Update to 6.7u3 failed
            KasperKWJ Lurker

            The hostd logs are just repetitions of what is pasted below. It did mention that port 9095 was not running, but we could not determine which service was using that port.

             

            Hostd logs:

             

            2019-11-29T01:53:43.390Z warning hostd[2101652] [Originator@6876 sub=SessionOrientedStub[0] opID=7c4de4b3] Invoke method vim.host.VsanSystemEx.GetVsanRuntimeInfo failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

            --> [context]zKq7AVICAgAAAASD2gAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

            2019-11-29T01:53:43.391Z warning hostd[2101652] [Originator@6876 sub=Hostsvc.HaHost opID=7c4de4b3] Fail to get VsanSystem runtime info.Maybe VSAN isn't started? Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

            2019-11-29T01:53:43.391Z warning hostd[2101652] [Originator@6876 sub=PropertyProvider opID=7c4de4b3] It took 10011677 microseconds, 00:00:10.011677 (hh:mm:ss.us), to get property summary for vim.HostSystem:ha-host

            2019-11-29T01:53:43.390Z error hostd[2101562] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter for server <cs p:0000007a374eae00, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

            --> [context]zKq7AVICAgAAAASD2gAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

            2019-11-29T01:53:43.460Z warning hostd[2101562] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000007a38e02800, h:28, <TCP '127.0.0.1 : 56351'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

            2019-11-29T01:53:45.462Z warning hostd[2101985] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000007a391228e0, h:28, <TCP '127.0.0.1 : 47015'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

            2019-11-29T01:53:45.463Z warning hostd[2102313] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00000079b9b88720, h:19, <TCP '127.0.0.1 : 38140'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

            2019-11-29T01:53:45.463Z error hostd[2102313] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter for server <cs p:0000007a374eae00, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

             

            vpxa.logs:

             

            2019-11-29T01:52:19.725Z - time the service was last started, Section for VMware ESX, pid=2101804, version=6.7.0, build=14320388, option=Release

            2019-11-29T01:52:19.725Z info -[2101804] [Originator@6876 sub=Default] Glibc malloc guards disabled.

            2019-11-29T01:52:19.725Z info -[2101804] [Originator@6876 sub=Default] Initialized SystemFactory

            2019-11-29T01:52:19.748Z warning vpxa[2101804] [Originator@6876 sub=vpxLro] [VpxLRO] 98 requested for task.maxThreads exceedsmax allowed; capping to 32

            2019-11-29T01:52:19.763Z error vpxa[2101804] [Originator@6876 sub=Default] [VimXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

            2019-11-29T01:52:19.763Z warning vpxa[2101804] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

            2019-11-29T01:52:19.763Z error vpxa[2101804] [Originator@6876 sub=Default] Can not discover support for WSDL namespace vim25

            2019-11-29T01:52:19.763Z warning vpxa[2101804] [Originator@6876 sub=hostdcnx] Could not resolve version for authenticating to host agent

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

            2019-11-29T01:52:20.569Z warning vpxa[2101804] [Originator@6876 sub=Default] Document root directory not found at /etc/vmware/vpxa/docRoot

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Http Service started: <server <acceptor p:0x000000699d9d4900, h:12, <TCP '127.0.0.1 : 8089'>>, >

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

            2019-11-29T01:52:20.569Z warning vpxa[2101804] [Originator@6876 sub=Default] Document root directory not found at /etc/vmware/vpxa/docRoot

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

            2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Http Service started: <server <acceptor p:0x000000699d9fe4b0, h:13, <UNIX '/var/run/vmware/proxy-vpxa'>>, >

            2019-11-29T01:54:03.810Z warning vpxa[2101822] [Originator@6876 sub=VpxProfiler] InvokeWithOpId [TotalTime] took 84042 ms

            2019-11-29T01:54:13.838Z warning vpxa[2101818] [Originator@6876 sub=vpxaAuthorize opID=lro-2541221-7ded7eb1-43] No auth data found for privileged operation:method=setServer

            2019-11-29T01:55:04.361Z warning vpxa[2101815] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

            2019-11-29T01:55:04.362Z warning vpxa[2101815] [Originator@6876 sub=VpxProfiler] InvokeWithOpId [TotalTime] took 44591 ms

            2019-11-29T01:55:14.392Z warning vpxa[2101825] [Originator@6876 sub=VpxProfiler opID=SWI-3113c398] InvokeWithOpId [TotalTime] took 10034 ms

            2019-11-29T01:55:24.367Z warning vpxa[2101815] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

            2019-11-29T01:55:44.369Z warning vpxa[2101824] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

            • 3. Re: Update to 6.7u3 failed
              msripada Expert
              vExpert

              i believe 9095 might be vsan service.. can you check the vsan health service log once?

               

              Are you using vsan?

               

              Thanks,

              MS