One of my 5.1 ESX hosts is disconnected in vCenter, usually happens after reboot (splitter removal requires reboot)
The symptom is that all esxcli commands fail with Connect to localhost failed: Connection failure
Searching for Error messages in logs did not come with something useful
Less /var/log/vpxa.log
2014-04-30T12:34:58.059Z [34242B90 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-04-30T12:35:18.061Z [343E1B90 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-04-30T12:35:18.080Z [343E1B90 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-04-30T12:35:18.080Z [343E1B90 warning 'Default'] Closing Response processing in unexpected state: 3
2014-04-30T12:35:18.100Z [343E1B90 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-04-30T12:35:18.100Z [343E1B90 warning 'Default'] Closing Response processing in unexpected state: 3
2014-04-30T12:35:18.100Z [43E1B90 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
Less var/log/vmkernel.log
2014-04-30T13:41:34.248Z cpu0:8781)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x41240040dfc0, 0) to dev "naa.60012480000300002d5b4dc4ae97c8d9" on path "vmhba35:C0:T9:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2014-04-30T13:41:34.390Z cpu0:8208)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x41240040dfc0, 0) to dev "naa.60012480000300002d5b4dc4ae97c8d9" on path "vmhba35:C0:T9:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2014-04-30T13:41:34.533Z cpu0:8208)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x41240040dfc0, 0) to dev "naa.60012480000300002d5b4dc4ae97c8d9" on path "vmhba35:C0:T9:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2014-04-30T13:41:34.576Z cpu0:8208)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60012480000300002d5b4dc4ae97c8d9" state in doubt; requested fast path state update...
2014-04-30T13:41:34.581Z cpu4:8746)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.600601607bc12d00f27438edcbaae311" - issuing command 0x412441eeac00
2014-04-30T13:41:34.581Z cpu8:8200)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0x28 (0x412441eeac00) to dev "naa.600601607bc12d00f27438edcbaae311" failed on path "vmhba2:C0:T1:L27" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.
2014-04-30T13:41:34.581Z cpu8:8200)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.600601607bc12d00f27438edcbaae311": awaiting fast path state update before retrying failed command again...
2014-04-30T13:41:34.676Z cpu0:8208)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x41240040dfc0, 0) to dev "naa.60012480000300002d5b4dc4ae97c8d9" on path "vmhba35:C0:T9:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
2014-04-30T13:41:34.819Z cpu0:8602)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x12 (0x41240040dfc0, 0) to dev "naa.60012480000300002d5b4dc4ae97c8d9" on path "vmhba35:C0:T9:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x24 0x0. Act:EVAL
I’ve tried several reboots, services.sh restart, /etc/init.d/hostd restart, /etc/init.d/hostd restart, /etc/init.d/vpxa restart. No change
