VMware Cloud Community
KasperKWJ
Contributor
Contributor

Update to 6.7u3 failed

Hi all,

Recently I've been trying to patch some ESXi servers to 6.7u3. The patching is successful. However after the server reboot, the ESXi is unable to exit maintenance mode. vCenter always throw the error message "Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections" whenever I try to exit maintenance mode.

On the esxi hostd.log:

warning hostd[2101206] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000008c848b1d50, h:21, <TCP '127.0.0.1 : 57259'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

error hostd[2101206] [Originator@6876 sub=vmomi.soapStub[1]] Resetting stub adapter for server <cs p:0000008c834451e0, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVICAgAAACEo5QAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

Been trying to figure out what port 9095 is used for, but to no avail. I'm thinking it might be related to vSAN. We are using VMware vSAN on the cluster. Can anyone help with this issue?

Reply
0 Kudos
3 Replies
msripada
Virtuoso
Virtuoso

so post ESXi host upgrade to 6.7 update 3, host is disconnected..

can you share the output of /etc/init.d/hostd status and /etc/init.d/vpxa status?

You can collect the log bundle using shell and can share us.. mostly hostd failed to start or unresponsive but need log bundle to validate

thanks,

MS

Reply
0 Kudos
KasperKWJ
Contributor
Contributor

The hostd logs are just repetitions of what is pasted below. It did mention that port 9095 was not running, but we could not determine which service was using that port.

Hostd logs:

2019-11-29T01:53:43.390Z warning hostd[2101652] [Originator@6876 sub=SessionOrientedStub[0] opID=7c4de4b3] Invoke method vim.host.VsanSystemEx.GetVsanRuntimeInfo failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVICAgAAAASD2gAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

2019-11-29T01:53:43.391Z warning hostd[2101652] [Originator@6876 sub=Hostsvc.HaHost opID=7c4de4b3] Fail to get VsanSystem runtime info.Maybe VSAN isn't started? Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2019-11-29T01:53:43.391Z warning hostd[2101652] [Originator@6876 sub=PropertyProvider opID=7c4de4b3] It took 10011677 microseconds, 00:00:10.011677 (hh:mm:ss.us), to get property summary for vim.HostSystem:ha-host

2019-11-29T01:53:43.390Z error hostd[2101562] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter for server <cs p:0000007a374eae00, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVICAgAAAASD2gAOaG9zdGQAACxONmxpYnZtYWNvcmUuc28AALC+GwCQnBcAiskoANkbLQCAHi0A3SItAOEqLQAVaSkAteUoAJPpKACrxDYBO30AbGlicHRocmVhZC5zby4wAAJ9nw5saWJjLnNvLjYA[/context]

2019-11-29T01:53:43.460Z warning hostd[2101562] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000007a38e02800, h:28, <TCP '127.0.0.1 : 56351'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

2019-11-29T01:53:45.462Z warning hostd[2101985] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0000007a391228e0, h:28, <TCP '127.0.0.1 : 47015'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

2019-11-29T01:53:45.463Z warning hostd[2102313] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00000079b9b88720, h:19, <TCP '127.0.0.1 : 38140'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

2019-11-29T01:53:45.463Z error hostd[2102313] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter for server <cs p:0000007a374eae00, TCP:localhost.localdomain:9095> : service state request failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

vpxa.logs:

2019-11-29T01:52:19.725Z - time the service was last started, Section for VMware ESX, pid=2101804, version=6.7.0, build=14320388, option=Release

2019-11-29T01:52:19.725Z info -[2101804] [Originator@6876 sub=Default] Glibc malloc guards disabled.

2019-11-29T01:52:19.725Z info -[2101804] [Originator@6876 sub=Default] Initialized SystemFactory

2019-11-29T01:52:19.748Z warning vpxa[2101804] [Originator@6876 sub=vpxLro] [VpxLRO] 98 requested for task.maxThreads exceedsmax allowed; capping to 32

2019-11-29T01:52:19.763Z error vpxa[2101804] [Originator@6876 sub=Default] [VimXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2019-11-29T01:52:19.763Z warning vpxa[2101804] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2019-11-29T01:52:19.763Z error vpxa[2101804] [Originator@6876 sub=Default] Can not discover support for WSDL namespace vim25

2019-11-29T01:52:19.763Z warning vpxa[2101804] [Originator@6876 sub=hostdcnx] Could not resolve version for authenticating to host agent

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

2019-11-29T01:52:20.569Z warning vpxa[2101804] [Originator@6876 sub=Default] Document root directory not found at /etc/vmware/vpxa/docRoot

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Http Service started: <server <acceptor p:0x000000699d9d4900, h:12, <TCP '127.0.0.1 : 8089'>>, >

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

2019-11-29T01:52:20.569Z warning vpxa[2101804] [Originator@6876 sub=Default] Document root directory not found at /etc/vmware/vpxa/docRoot

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for nonChunkingAgents

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Using default for agentsNeedingContentLength

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Max buffered response size is 104857600 bytes

2019-11-29T01:52:20.569Z info vpxa[2101804] [Originator@6876 sub=SoapAdapter.HTTPService] Http Service started: <server <acceptor p:0x000000699d9fe4b0, h:13, <UNIX '/var/run/vmware/proxy-vpxa'>>, >

2019-11-29T01:54:03.810Z warning vpxa[2101822] [Originator@6876 sub=VpxProfiler] InvokeWithOpId [TotalTime] took 84042 ms

2019-11-29T01:54:13.838Z warning vpxa[2101818] [Originator@6876 sub=vpxaAuthorize opID=lro-2541221-7ded7eb1-43] No auth data found for privileged operation:method=setServer

2019-11-29T01:55:04.361Z warning vpxa[2101815] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

2019-11-29T01:55:04.362Z warning vpxa[2101815] [Originator@6876 sub=VpxProfiler] InvokeWithOpId [TotalTime] took 44591 ms

2019-11-29T01:55:14.392Z warning vpxa[2101825] [Originator@6876 sub=VpxProfiler opID=SWI-3113c398] InvokeWithOpId [TotalTime] took 10034 ms

2019-11-29T01:55:24.367Z warning vpxa[2101815] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

2019-11-29T01:55:44.369Z warning vpxa[2101824] [Originator@6876 sub=hostdstats] Host to vpxd translation is empty, dropping results

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

i believe 9095 might be vsan service.. can you check the vsan health service log once?

Are you using vsan?

Thanks,

MS

Reply
0 Kudos