1 2 3 Previous Next 32 Replies Latest reply on May 3, 2016 7:26 PM by dsalnikov

vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 14, 2014 11:02 AM

Hello all,

Last week i have not been successful in upgrading the VMware environment from 5.5 to 5.5U1. Initial installation went fine, configured the first ESXi host and created a host profile. So far so good, no major issues. When installing the remainder of the 5 other blade servers and applying the host profile all seems to be working well whereas things went sideways when the entire environment was shutdown and restarted.

Te setup looks as following:

Dell 1955 blade center servers (6x)

Cisco 3030 switches connected to the blades

vCenter server as a VM on the ESXi cluster

vCenter upgrade manager as a VM

Distributed Virtual Switch configured

During the shutdown procedure i move all the vCenter components to a single host before powering down the vCenter server (Windows 2008). Then i shutdown the ESXi host using the management client as vCenter is no longer available.

Power up cycle is to start this ESXi host first, connect to it and then powerup the vCenter components. Once all is running DRS and Power management is configured to automatic after which the simulation VM's are started.

With 5.5 all was working well and no issues at all. When i installed 5.5 U1 i could no longer connect to the ESXi server and it gave an error 503 when connecting to it. Going through the log files showed that components on the file system could not be found after which i decided to do a clean install.

After reinstalling the whole lot i connected to a ESXi host and enabled the network and added the iScsi adapter. At that stage the system becomes unresponsive and the session between the ESXi host and the client is disconnected.

Running several tests i concluded that something must be wrong with the virtual disk system components, iScsi or vSan. 5 of the 6 hosts will be rolled back to vmware 5.5 as vmware 5.5U1 is not stable...

Can use some help troubleshooting this issue as i will leave 1 server in the current state it is to identify the root cause as i'm stuck at the moment. Later on i will post some log file sections with the errors i identified. (Restarting the hostd does not resolve the problem)

Regards,

Martin Meuwese

1. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:30 AM (in response to M.Meuwese)

Actions that cause the VMware 5.5u1 to stop functioning:

1. Login to the vSphere client
2. Enable iScsi
Now the system is stuck and will no longer respond to any commands, even from the server console pressing f2 or f12 will yield no result. It is completely stuck. Only a reboot can revive the system....
Connecting to the console is back online though the vSphere client can't connect and throws an error 'An unknown connection error occurred. (The server could not interpret the client's request. (the remote server returned an error: (503) Server Unavailable.))
Log files show the following:

VMKERNEL LOG:
Like Show 0 Likes(0)

Actions
2. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:36 AM (in response to M.Meuwese)

2014-03-16T17:15:06.550Z cpu1:36726)UserDump: 1820: Dumping cartel 36726 (from world 36726) to file /var/core/hostd-zdump.003 ...
2014-03-16T17:17:01.560Z cpu2:36726)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:17:03.392Z cpu2:37378)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:17:03.392Z cpu2:37378)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:17:03.392Z cpu2:37378)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:17:03.465Z cpu2:37378)VC: 2059: Device rescan time 485 msec (total number of devices 6)
2014-03-16T17:17:03.465Z cpu2:37378)VC: 2062: Filesystem probe time 461 msec (devices probed 5 of 6)
2014-03-16T17:17:03.465Z cpu2:37378)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:17:04.035Z cpu3:37378)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:17:04.557Z cpu0:37378)UserDump: 1820: Dumping cartel 37378 (from world 37378) to file /var/core/hostd-zdump.000 ...
2014-03-16T17:18:54.306Z cpu2:37378)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:18:55.961Z cpu3:38000)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:18:55.961Z cpu3:38000)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:18:55.961Z cpu3:38000)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:18:56.080Z cpu3:38000)VC: 2059: Device rescan time 369 msec (total number of devices 6)
2014-03-16T17:18:56.080Z cpu3:38000)VC: 2062: Filesystem probe time 459 msec (devices probed 5 of 6)
2014-03-16T17:18:56.080Z cpu3:38000)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:18:56.669Z cpu1:38000)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:18:56.838Z cpu1:38000)UserDump: 1820: Dumping cartel 38000 (from world 38000) to file /var/core/hostd-zdump.001 ...
2014-03-16T17:20:47.344Z cpu0:38000)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:20:49.646Z cpu4:38673)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:20:49.646Z cpu4:38673)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:20:49.646Z cpu4:38673)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:20:49.647Z cpu4:38673)VC: 2059: Device rescan time 541 msec (total number of devices 6)
2014-03-16T17:20:49.647Z cpu4:38673)VC: 2062: Filesystem probe time 800 msec (devices probed 5 of 6)
2014-03-16T17:20:49.647Z cpu4:38673)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:20:50.213Z cpu4:38673)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:20:50.387Z cpu4:38673)UserDump: 1820: Dumping cartel 38673 (from world 38673) to file /var/core/hostd-zdump.002 ...
2014-03-16T17:22:42.225Z cpu6:38673)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:22:44.287Z cpu7:39308)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:22:44.287Z cpu7:39308)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:22:44.287Z cpu7:39308)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:22:44.408Z cpu7:39308)VC: 2059: Device rescan time 639 msec (total number of devices 6)
2014-03-16T17:22:44.408Z cpu7:39308)VC: 2062: Filesystem probe time 602 msec (devices probed 5 of 6)
2014-03-16T17:22:44.408Z cpu7:39308)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:22:44.955Z cpu7:39308)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:22:45.213Z cpu7:39308)UserDump: 1820: Dumping cartel 39308 (from world 39308) to file /var/core/hostd-zdump.003 ...
2014-03-16T17:24:32.615Z cpu0:39308)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:24:34.538Z cpu3:39918)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:24:34.538Z cpu3:39918)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:24:34.538Z cpu3:39918)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:24:34.661Z cpu3:39918)VC: 2059: Device rescan time 515 msec (total number of devices 6)
2014-03-16T17:24:34.661Z cpu3:39918)VC: 2062: Filesystem probe time 574 msec (devices probed 5 of 6)
2014-03-16T17:24:34.661Z cpu3:39918)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:24:35.297Z cpu3:39918)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:24:35.567Z cpu3:39918)UserDump: 1820: Dumping cartel 39918 (from world 39918) to file /var/core/hostd-zdump.000 ...
2014-03-16T17:26:24.277Z cpu6:39918)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:26:25.875Z cpu6:40534)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:26:25.875Z cpu6:40534)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:26:25.875Z cpu6:40534)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:26:25.997Z cpu6:40534)VC: 2059: Device rescan time 344 msec (total number of devices 6)
2014-03-16T17:26:25.997Z cpu6:40534)VC: 2062: Filesystem probe time 451 msec (devices probed 5 of 6)
2014-03-16T17:26:25.997Z cpu6:40534)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:26:26.525Z cpu6:40534)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:26:26.783Z cpu6:40534)UserDump: 1820: Dumping cartel 40534 (from world 40534) to file /var/core/hostd-zdump.001 ...
2014-03-16T17:28:27.631Z cpu3:40534)UserDump: 1944: Userworld coredump complete.
2014-03-16T17:28:29.530Z cpu4:41281)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:28:29.530Z cpu4:41281)Vol3: 714: Couldn't read volume header from control: Not supported
2014-03-16T17:28:29.530Z cpu4:41281)FSS: 5092: No FS driver claimed device 'control': Not supported
2014-03-16T17:28:30.069Z cpu4:41281)VC: 2059: Device rescan time 558 msec (total number of devices 6)
2014-03-16T17:28:30.069Z cpu4:41281)VC: 2062: Filesystem probe time 856 msec (devices probed 5 of 6)
2014-03-16T17:28:30.069Z cpu4:41281)VC: 2064: Refresh open volume time 0 msec
2014-03-16T17:28:30.614Z cpu6:41281)User: 2888: wantCoreDump : hostd -enabled : 1
2014-03-16T17:28:31.132Z cpu6:41281)UserDump: 1820: Dumping cartel 41281 (from world 41281) to file /var/core/hostd-zdump.002 ...
Like Show 0 Likes(0)

Actions
3. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:39 AM (in response to M.Meuwese)

VPXA.LOG

2014-03-16T17:36:19.602Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:36:19.602Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:36:19.602Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:36:39.603Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:36:39.618Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:36:39.618Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:36:39.631Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:36:39.632Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:36:39.632Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:36:59.634Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:36:59.646Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:36:59.646Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:36:59.659Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:36:59.659Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:36:59.659Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:37:19.659Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:37:19.674Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:37:19.675Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:19.686Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:37:19.687Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:19.687Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:37:39.689Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:37:39.703Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:37:39.703Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:39.718Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:37:39.718Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:39.718Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:37:59.718Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:37:59.731Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:37:59.731Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:59.760Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:37:59.760Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:37:59.760Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
2014-03-16T17:38:19.761Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
2014-03-16T17:38:19.774Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)
2014-03-16T17:38:19.774Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:38:19.788Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)
2014-03-16T17:38:19.788Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3
2014-03-16T17:38:19.789Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent
Like Show 0 Likes(0)

Actions
4. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:41 AM (in response to M.Meuwese)

Firewall log:

2014-03-16T17:23:14.001Z: Abandoned event (esx.audit.net.firewall.config.changed) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.config.changed) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.port.hooked) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.port.hooked) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.clear.coredump.configured2) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.dcui.enabled) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.host.boot) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.clear.coredump.configured2) after 6 failures.
2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.dcui.login.failed) after 6 failures.
2014-03-16T17:23:14.002Z: Failed to send event (esx.problem.hostd.core.dumped); 5 failures so far.
2014-03-16T17:24:32.615Z: [UserWorldCorrelator] 994329446us: [vob.uw.core.dumped] /bin/hostd(39308) /var/core/hostd-zdump.003
2014-03-16T17:26:14.000Z: Abandoned event (esx.problem.hostd.core.dumped) after 6 failures.
2014-03-16T17:26:24.277Z: [UserWorldCorrelator] 1105991600us: [vob.uw.core.dumped] /bin/hostd(39918) /var/core/hostd-zdump.000
2014-03-16T17:27:26.545Z: [GenericCorrelator] 1168260001us: [vob.user.ssh.enabled] SSH access has been enabled
2014-03-16T17:27:26.545Z: [UserLevelCorrelator] 1168260001us: [vob.user.ssh.enabled] SSH access has been enabled
2014-03-16T17:27:26.546Z: [UserLevelCorrelator] 1168260365us: [esx.audit.ssh.enabled] SSH access has been enabled.
2014-03-16T17:27:26.546Z: An event (esx.audit.ssh.enabled) could not be sent immediately to hostd; queueing for retry.
2014-03-16T17:28:27.631Z: [UserWorldCorrelator] 1229345745us: [vob.uw.core.dumped] /bin/hostd(40534) /var/core/hostd-zdump.001
2014-03-16T17:30:22.243Z: [UserWorldCorrelator] 1343957129us: [vob.uw.core.dumped] /bin/hostd(41281) /var/core/hostd-zdump.002
2014-03-16T17:30:26.001Z: Failed to send event (esx.audit.ssh.enabled); 2 failures so far.
2014-03-16T17:32:13.164Z: [UserWorldCorrelator] 1454878605us: [vob.uw.core.dumped] /bin/hostd(41897) /var/core/hostd-zdump.003
2014-03-16T17:33:26.001Z: Failed to send event (esx.audit.ssh.enabled); 3 failures so far.
2014-03-16T17:34:12.282Z: [UserWorldCorrelator] 1573996718us: [vob.uw.core.dumped] /bin/hostd(42551) /var/core/hostd-zdump.000
2014-03-16T17:35:56.867Z: [UserWorldCorrelator] 1678581191us: [vob.uw.core.dumped] /bin/hostd(43206) /var/core/hostd-zdump.001
2014-03-16T17:36:26.001Z: Failed to send event (esx.audit.ssh.enabled); 4 failures so far.
2014-03-16T17:37:49.302Z: [UserWorldCorrelator] 1791016363us: [vob.uw.core.dumped] /bin/hostd(43796) /var/core/hostd-zdump.002
2014-03-16T17:39:26.000Z: Failed to send event (esx.audit.ssh.enabled); 5 failures so far.
2014-03-16T17:39:53.565Z: [UserWorldCorrelator] 1915279514us: [vob.uw.core.dumped] /bin/hostd(44418) /var/core/hostd-zdump.003
Like Show 0 Likes(0)

Actions
5. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:44 AM (in response to M.Meuwese)

Hostd.log

2014-03-14T16:29:32.452Z [5BA81B70 error 'Solo.VmwareCLI' opID=hostd-6425 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-14T16:29:33.923Z [5BE40B70 error 'Solo.VmwareCLI' opID=hostd-d756 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-14T16:29:34.278Z [5BE40B70 error 'Solo.VmwareCLI' opID=hostd-40c5 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-14T16:30:27.373Z [FFF5BB70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x5b629400, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-14T16:30:27.373Z [5B481B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe7bdc0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-14T16:36:02.878Z [FFD315B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-14T17:01:08.348Z [FFA955B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-14T17:01:15.626Z [3AB80B70 error 'Default'] prctl(PR_SET_NAME, '') failed; errno: 22(Invalid argument)
2014-03-14T17:01:15.745Z [FFA955B0 error 'Statssvc.vim.PerformanceManager'] Error while loading stats depot: "FileIO error: Could not find file : /var/lib/vmware/hostd/stats/hostAgentStats.idMap". Possible file corruption!
2014-03-14T17:01:15.746Z [FFA955B0 error 'Statssvc.vim.PerformanceManager'] Unable to load stats depot hostAgentStats in directory /var/lib/vmware/hostd/stats/ : FileIO error: Could not find file : /var/lib/vmware/hostd/stats/hostAgentStats.idMap
2014-03-14T17:01:17.450Z [FFC40B70 error 'Solo.VmwareCLI' opID=hostd-288b user=root] GetPrimitiveParam: Cannot find (help)
2014-03-14T17:01:18.933Z [FFAF8B70 error 'Solo.VmwareCLI' opID=hostd-ca86 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-14T17:01:19.292Z [3A2C2B70 error 'Solo.VmwareCLI' opID=hostd-42be user=root] GetPrimitiveParam: Cannot find (help)
ResourceGroup: Skipping resource group: ResourceGroupImpl (3834) in group 21. Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information
2014-03-14T17:02:12.370Z [3A2C2B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x3a72e700, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-14T17:02:12.370Z [3AE40B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe94e48, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-14T17:07:20.017Z [FFB975B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-14T17:09:04.135Z [FFDAE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-14T17:10:49.473Z [FFAF35B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T16:06:52.666Z [FF8A25B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T16:06:59.924Z [776DAB70 error 'Default'] prctl(PR_SET_NAME, '') failed; errno: 22(Invalid argument)
2014-03-16T16:07:00.046Z [FF8A25B0 error 'Statssvc.vim.PerformanceManager'] Error while loading stats depot: "FileIO error: Could not find file : /var/lib/vmware/hostd/stats/hostAgentStats.idMap". Possible file corruption!
2014-03-16T16:07:00.046Z [FF8A25B0 error 'Statssvc.vim.PerformanceManager'] Unable to load stats depot hostAgentStats in directory /var/lib/vmware/hostd/stats/ : FileIO error: Could not find file : /var/lib/vmware/hostd/stats/hostAgentStats.idMap
2014-03-16T16:07:01.744Z [772EAB70 error 'Solo.VmwareCLI' opID=hostd-5fd2 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-16T16:07:03.225Z [FF905B70 error 'Solo.VmwareCLI' opID=hostd-fae0 user=root] GetPrimitiveParam: Cannot find (help)
2014-03-16T16:07:03.586Z [772EAB70 error 'Solo.VmwareCLI' opID=hostd-7d5e user=root] GetPrimitiveParam: Cannot find (help)
2014-03-16T16:07:56.662Z [FF987B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x77301be8, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-16T16:07:56.662Z [FF946B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x775026e0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-16T16:13:38.441Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x774037e8, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:18:38.448Z [772EAB70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x7750b420, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:23:38.456Z [772A9B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa00668, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:27:17.024Z [FF905B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0xffc0a718, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-16T16:28:38.462Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x775026a8, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:33:38.472Z [FF987B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x774667f8, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:38:38.480Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa06358, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:43:38.488Z [772EAB70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc2e428, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:48:38.497Z [FF946B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc0a718, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:53:07.520Z [FF946B70 error 'Solo.HttpSvc.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe5e6a8, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2014-03-16T16:53:38.506Z [FF987B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa01878, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T16:58:38.514Z [77D80B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc19560, h:54, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T17:03:38.522Z [77D80B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x77c352c0, h:56, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-03-16T17:09:07.509Z [FFA685B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:11:18.098Z [FF9A05B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:13:12.660Z [FFF435B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:15:04.433Z [FFC2D5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:17:02.503Z [FFEFE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:18:55.233Z [FFEEE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:20:48.289Z [FFD825B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:22:43.148Z [FFC555B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:24:33.553Z [FF9995B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:26:25.182Z [FFF215B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:28:28.635Z [FF9D95B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:30:23.165Z [FFDC95B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:32:14.092Z [FFBF05B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:34:13.200Z [FFCCE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:35:57.746Z [FFB845B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:37:50.235Z [FFCE65B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:39:54.468Z [FFE285B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
2014-03-16T17:42:15.881Z [FF94E5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.
Like Show 0 Likes(0)

Actions
6. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 16, 2014 10:45 AM (in response to M.Meuwese)

rhttplog

2014-03-16T17:44:24.686Z [FF946B70 verbose 'Proxy Req 01417'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:38948'>>)>
2014-03-16T17:44:24.687Z [FF8A1D70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb037d8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:24.687Z [FF8A1D70 warning 'Proxy Req 01417'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:28.266Z [FF987B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb00620, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:42886'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2014-03-16T17:44:28.277Z [FF946B70 verbose 'Proxy Req 01419'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:60784'>>)>
2014-03-16T17:44:28.278Z [FF987B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb038c8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:28.278Z [FF987B70 warning 'Proxy Req 01419'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:31.854Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb01260, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:52406'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2014-03-16T17:44:31.868Z [FF946B70 verbose 'Proxy Req 01421'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:13831'>>)>
2014-03-16T17:44:31.869Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb03a08, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:31.869Z [FF905B70 warning 'Proxy Req 01421'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:35.441Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb01260, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:14593'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2014-03-16T17:44:35.453Z [FF987B70 verbose 'Proxy Req 01423'] New proxy client <SSL(<io_obj p:0xffc1b7f8, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:36746'>>)>
2014-03-16T17:44:35.454Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc2dad8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:35.454Z [FF905B70 warning 'Proxy Req 01423'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:39.032Z [FF946B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:47755'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2014-03-16T17:44:39.043Z [FF987B70 verbose 'Proxy Req 01425'] New proxy client <SSL(<io_obj p:0xffc012f0, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:61960'>>)>
2014-03-16T17:44:39.044Z [FF946B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc1b420, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:39.045Z [FF946B70 warning 'Proxy Req 01425'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:40.312Z [FF946B70 verbose 'Proxy Req 01426'] New proxy client <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:53555'>>)>
2014-03-16T17:44:40.313Z [FF946B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffa1d708, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:40.313Z [FF946B70 warning 'Proxy Req 01426'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:40.327Z [FF987B70 verbose 'Proxy Req 01427'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:37426'>>)>
2014-03-16T17:44:40.328Z [FF987B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb03ac0, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:40.328Z [FF987B70 warning 'Proxy Req 01427'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
2014-03-16T17:44:42.617Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:39162'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
2014-03-16T17:44:42.630Z [FF987B70 verbose 'Proxy Req 01429'] New proxy client <SSL(<io_obj p:0xffc2d230, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:36602'>>)>
2014-03-16T17:44:42.631Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc2db30, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)
2014-03-16T17:44:42.631Z [FF905B70 warning 'Proxy Req 01429'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).
Like Show 0 Likes(0)

Actions
7. Re: vmware 5.5 u1 - iScsi adapter crashes the system

amiterjain Mar 16, 2014 10:59 PM (in response to M.Meuwese)

Hi Martin,

Could you please check the hardware compatibility list. I could not see Dell 1955 blade center server support in 5.5U1 HCL.
Like Show 0 Likes(0)

Actions
8. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 17, 2014 2:35 AM (in response to amiterjain)

Will do. Probably now time to start replacing it.

Martin

Sent from my iPhone
Like Show 0 Likes(0)

Actions
9. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 17, 2014 8:46 AM (in response to M.Meuwese)

Today i received a reply that the 1955 DELL Blade servers are no longer in the HCL which could be the cause for the issues to surface. I will close this thread as it is running on non-supported hardware and revert back to 5.5 1331820 as this has been running stable for a long time.
Like Show 0 Likes(0)

Actions
10. Re: vmware 5.5 u1 - iScsi adapter crashes the system

pasikappext Mar 18, 2014 10:23 AM (in response to M.Meuwese)

Hello,

I don't think this problem is related to hardware. I'm seeing the exact same hostd crash caused by sw-iSCSI on ESXi 5.5U1 and this is with supported (on-HCL) hardware.

Any ideas how to fix this?
Like Show 0 Likes(0)

Actions
11. Re: vmware 5.5 u1 - iScsi adapter crashes the system

pasikappext Mar 18, 2014 12:31 PM (in response to pasikappext)

More information.. to reproduce:

1) Install ESXi 5.5 U1 host.
2) Verify you can connect to the host with vSphere Client.
3) ssh into the host and run "esxcli iscsi software set --enabled=true". Or enable sw-iSCSI client using the vSphere Client.
4) Reboot the ESXi host.
5) Notice you can't connect to the host with vSphere Client anymore.
6) ssh into the host and check the process list with "ps". You'll notice "hostd" is not there. "hostd" now crashes on every ESXi host reboot.
Like Show 0 Likes(0)

Actions
12. Re: vmware 5.5 u1 - iScsi adapter crashes the system

pasikappext Mar 18, 2014 1:57 PM (in response to pasikappext)

Ok, so it looks the hostd iscsi crash only happens when installing the ESXi 5.5u1 host from "VMware-ESXi-5.5U1-RollupISO.iso".
If I install the host from "VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso" then iscsi seems to work OK!

I wonder what's wrong with the RollupISO ..
Like Show 0 Likes(0)

Actions
13. Re: vmware 5.5 u1 - iScsi adapter crashes the system

M.Meuwese Mar 18, 2014 2:15 PM (in response to pasikappext)

I Will verify tomorrow and post my findings once done. Let's hope it is that simple...

Sent from my iPhone
Like Show 0 Likes(0)

Actions
14. Re: vmware 5.5 u1 - iScsi adapter crashes the system

Wingsup Mar 18, 2014 4:21 PM (in response to M.Meuwese)

I just wanted to throw this out here, I had the same issue in my lab, older unsupported hardware also. The rollup iso caused the same issue with an iscsi crash for me, and the the version mentioned by user pasikappext works for me. "VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso
Like Show 0 Likes(0)

Actions