VMware Cloud Community
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

vmware 5.5 u1 - iScsi adapter crashes the system

Hello all,

Last week i have not been successful in upgrading the VMware environment from 5.5 to 5.5U1. Initial installation went fine, configured the first ESXi host and created a host profile. So far so good, no major issues. When installing the remainder of the 5 other blade servers and applying the host profile all seems to be working well whereas things went sideways when the entire environment was shutdown and restarted.

Te setup looks as following:

Dell 1955 blade center servers (6x)

Cisco 3030 switches connected to the blades

vCenter server as a VM on the ESXi cluster

vCenter upgrade manager as a VM

Distributed Virtual Switch configured

During the shutdown procedure i move all the vCenter components to a single host before powering down the vCenter server (Windows 2008). Then i shutdown the ESXi host using the management client as vCenter is no longer available.

Power up cycle is to start this ESXi host first, connect to it and then powerup the vCenter components. Once all is running DRS and Power management is configured to automatic after which the simulation VM's are started.

With 5.5 all was working well and no issues at all. When i installed 5.5 U1 i could no longer connect to the ESXi server and it gave an error 503 when connecting to it. Going through the log files showed that components on the file system could not be found after which i decided to do a clean install.

After reinstalling the whole lot i connected to a ESXi host and enabled the network and added the iScsi adapter. At that stage the system becomes unresponsive and the session between the ESXi host and the client is disconnected.

Running several tests i concluded that something must be wrong with the virtual disk system components, iScsi or vSan. 5 of the 6 hosts will be rolled back to vmware 5.5 as vmware 5.5U1 is not stable...

Can use some help troubleshooting this issue as i will leave 1 server in the current state it is to identify the root cause as i'm stuck at the moment. Later on i will post some log file sections with the errors i identified. (Restarting the hostd does not resolve the problem)

Regards,

Martin Meuwese

Best regards, Martin Meuwese
1 Solution

Accepted Solutions
amiterjain
Contributor
Contributor
Jump to solution

Hi Martin,

Could you please check the hardware compatibility list. I could not see Dell 1955 blade center server support in 5.5U1 HCL.

View solution in original post

Reply
0 Kudos
32 Replies
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

Actions that cause the VMware 5.5u1 to stop functioning:

1. Login to the vSphere client

ESXi5_5u1_crash_001.jpg

2. Enable iScsi

ESXi5_5u1_crash_002.jpg

ESXi5_5u1_crash_003.jpg

ESXi5_5u1_crash_004.jpg

Now the system is stuck and will no longer respond to any commands, even from the server console pressing f2 or f12 will yield no result. It is completely stuck. Only a reboot can revive the system....

ESXi5_5u1_crash_006.jpg

Connecting to the console is back online though the vSphere client can't connect and throws an error 'An unknown connection error occurred. (The server could not interpret the client's request. (the remote server returned an error: (503) Server Unavailable.))

ESXi5_5u1_crash_007.jpg

Log files show the following:

VMKERNEL LOG:

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

2014-03-16T17:15:06.550Z cpu1:36726)UserDump: 1820: Dumping cartel 36726 (from world 36726) to file /var/core/hostd-zdump.003 ...

2014-03-16T17:17:01.560Z cpu2:36726)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:17:03.392Z cpu2:37378)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:17:03.392Z cpu2:37378)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:17:03.392Z cpu2:37378)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:17:03.465Z cpu2:37378)VC: 2059: Device rescan time 485 msec (total number of devices 6)

2014-03-16T17:17:03.465Z cpu2:37378)VC: 2062: Filesystem probe time 461 msec (devices probed 5 of 6)

2014-03-16T17:17:03.465Z cpu2:37378)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:17:04.035Z cpu3:37378)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:17:04.557Z cpu0:37378)UserDump: 1820: Dumping cartel 37378 (from world 37378) to file /var/core/hostd-zdump.000 ...

2014-03-16T17:18:54.306Z cpu2:37378)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:18:55.961Z cpu3:38000)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:18:55.961Z cpu3:38000)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:18:55.961Z cpu3:38000)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:18:56.080Z cpu3:38000)VC: 2059: Device rescan time 369 msec (total number of devices 6)

2014-03-16T17:18:56.080Z cpu3:38000)VC: 2062: Filesystem probe time 459 msec (devices probed 5 of 6)

2014-03-16T17:18:56.080Z cpu3:38000)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:18:56.669Z cpu1:38000)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:18:56.838Z cpu1:38000)UserDump: 1820: Dumping cartel 38000 (from world 38000) to file /var/core/hostd-zdump.001 ...

2014-03-16T17:20:47.344Z cpu0:38000)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:20:49.646Z cpu4:38673)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:20:49.646Z cpu4:38673)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:20:49.646Z cpu4:38673)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:20:49.647Z cpu4:38673)VC: 2059: Device rescan time 541 msec (total number of devices 6)

2014-03-16T17:20:49.647Z cpu4:38673)VC: 2062: Filesystem probe time 800 msec (devices probed 5 of 6)

2014-03-16T17:20:49.647Z cpu4:38673)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:20:50.213Z cpu4:38673)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:20:50.387Z cpu4:38673)UserDump: 1820: Dumping cartel 38673 (from world 38673) to file /var/core/hostd-zdump.002 ...

2014-03-16T17:22:42.225Z cpu6:38673)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:22:44.287Z cpu7:39308)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:22:44.287Z cpu7:39308)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:22:44.287Z cpu7:39308)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:22:44.408Z cpu7:39308)VC: 2059: Device rescan time 639 msec (total number of devices 6)

2014-03-16T17:22:44.408Z cpu7:39308)VC: 2062: Filesystem probe time 602 msec (devices probed 5 of 6)

2014-03-16T17:22:44.408Z cpu7:39308)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:22:44.955Z cpu7:39308)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:22:45.213Z cpu7:39308)UserDump: 1820: Dumping cartel 39308 (from world 39308) to file /var/core/hostd-zdump.003 ...

2014-03-16T17:24:32.615Z cpu0:39308)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:24:34.538Z cpu3:39918)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:24:34.538Z cpu3:39918)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:24:34.538Z cpu3:39918)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:24:34.661Z cpu3:39918)VC: 2059: Device rescan time 515 msec (total number of devices 6)

2014-03-16T17:24:34.661Z cpu3:39918)VC: 2062: Filesystem probe time 574 msec (devices probed 5 of 6)

2014-03-16T17:24:34.661Z cpu3:39918)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:24:35.297Z cpu3:39918)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:24:35.567Z cpu3:39918)UserDump: 1820: Dumping cartel 39918 (from world 39918) to file /var/core/hostd-zdump.000 ...

2014-03-16T17:26:24.277Z cpu6:39918)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:26:25.875Z cpu6:40534)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:26:25.875Z cpu6:40534)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:26:25.875Z cpu6:40534)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:26:25.997Z cpu6:40534)VC: 2059: Device rescan time 344 msec (total number of devices 6)

2014-03-16T17:26:25.997Z cpu6:40534)VC: 2062: Filesystem probe time 451 msec (devices probed 5 of 6)

2014-03-16T17:26:25.997Z cpu6:40534)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:26:26.525Z cpu6:40534)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:26:26.783Z cpu6:40534)UserDump: 1820: Dumping cartel 40534 (from world 40534) to file /var/core/hostd-zdump.001 ...

2014-03-16T17:28:27.631Z cpu3:40534)UserDump: 1944: Userworld coredump complete.

2014-03-16T17:28:29.530Z cpu4:41281)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:28:29.530Z cpu4:41281)Vol3: 714: Couldn't read volume header from control: Not supported

2014-03-16T17:28:29.530Z cpu4:41281)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-03-16T17:28:30.069Z cpu4:41281)VC: 2059: Device rescan time 558 msec (total number of devices 6)

2014-03-16T17:28:30.069Z cpu4:41281)VC: 2062: Filesystem probe time 856 msec (devices probed 5 of 6)

2014-03-16T17:28:30.069Z cpu4:41281)VC: 2064: Refresh open volume time 0 msec

2014-03-16T17:28:30.614Z cpu6:41281)User: 2888: wantCoreDump : hostd -enabled : 1

2014-03-16T17:28:31.132Z cpu6:41281)UserDump: 1820: Dumping cartel 41281 (from world 41281) to file /var/core/hostd-zdump.002 ...

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

VPXA.LOG

2014-03-16T17:36:19.602Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:36:19.602Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:36:19.602Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:36:39.603Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:36:39.618Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:36:39.618Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:36:39.631Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:36:39.632Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:36:39.632Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:36:59.634Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:36:59.646Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:36:59.646Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:36:59.659Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:36:59.659Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:36:59.659Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:37:19.659Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:37:19.674Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:37:19.675Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:19.686Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:37:19.687Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:19.687Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:37:39.689Z [FFC27B70 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:37:39.703Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:37:39.703Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:39.718Z [FFC27B70 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:37:39.718Z [FFC27B70 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:39.718Z [FFC27B70 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:37:59.718Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:37:59.731Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:37:59.731Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:59.760Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:37:59.760Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:37:59.760Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

2014-03-16T17:38:19.761Z [FFBC31A0 verbose 'hostdcnx'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443

2014-03-16T17:38:19.774Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /sdk/vimServiceVersions.xml: 503 (Service Unavailable)

2014-03-16T17:38:19.774Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:38:19.788Z [FFBC31A0 verbose 'commonvpxXml'] [VpxXml] Error fetching /definitions/import/@namespace from /sdk/vimService?wsdl: 503 (Service Unavailable)

2014-03-16T17:38:19.788Z [FFBC31A0 warning 'Default'] Closing Response processing in unexpected state: 3

2014-03-16T17:38:19.789Z [FFBC31A0 warning 'hostdcnx'] [VpxaHalCnxHostagent] Could not resolve version for authenticating to host agent

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

Firewall log:

2014-03-16T17:23:14.001Z: Abandoned event (esx.audit.net.firewall.config.changed) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.config.changed) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.port.hooked) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.net.firewall.port.hooked) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.clear.coredump.configured2) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.dcui.enabled) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.host.boot) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.clear.coredump.configured2) after 6 failures.

2014-03-16T17:23:14.002Z: Abandoned event (esx.audit.dcui.login.failed) after 6 failures.

2014-03-16T17:23:14.002Z: Failed to send event (esx.problem.hostd.core.dumped); 5 failures so far.

2014-03-16T17:24:32.615Z: [UserWorldCorrelator] 994329446us: [vob.uw.core.dumped] /bin/hostd(39308) /var/core/hostd-zdump.003

2014-03-16T17:26:14.000Z: Abandoned event (esx.problem.hostd.core.dumped) after 6 failures.

2014-03-16T17:26:24.277Z: [UserWorldCorrelator] 1105991600us: [vob.uw.core.dumped] /bin/hostd(39918) /var/core/hostd-zdump.000

2014-03-16T17:27:26.545Z: [GenericCorrelator] 1168260001us: [vob.user.ssh.enabled] SSH access has been enabled

2014-03-16T17:27:26.545Z: [UserLevelCorrelator] 1168260001us: [vob.user.ssh.enabled] SSH access has been enabled

2014-03-16T17:27:26.546Z: [UserLevelCorrelator] 1168260365us: [esx.audit.ssh.enabled] SSH access has been enabled.

2014-03-16T17:27:26.546Z: An event (esx.audit.ssh.enabled) could not be sent immediately to hostd; queueing for retry.

2014-03-16T17:28:27.631Z: [UserWorldCorrelator] 1229345745us: [vob.uw.core.dumped] /bin/hostd(40534) /var/core/hostd-zdump.001

2014-03-16T17:30:22.243Z: [UserWorldCorrelator] 1343957129us: [vob.uw.core.dumped] /bin/hostd(41281) /var/core/hostd-zdump.002

2014-03-16T17:30:26.001Z: Failed to send event (esx.audit.ssh.enabled); 2 failures so far.

2014-03-16T17:32:13.164Z: [UserWorldCorrelator] 1454878605us: [vob.uw.core.dumped] /bin/hostd(41897) /var/core/hostd-zdump.003

2014-03-16T17:33:26.001Z: Failed to send event (esx.audit.ssh.enabled); 3 failures so far.

2014-03-16T17:34:12.282Z: [UserWorldCorrelator] 1573996718us: [vob.uw.core.dumped] /bin/hostd(42551) /var/core/hostd-zdump.000

2014-03-16T17:35:56.867Z: [UserWorldCorrelator] 1678581191us: [vob.uw.core.dumped] /bin/hostd(43206) /var/core/hostd-zdump.001

2014-03-16T17:36:26.001Z: Failed to send event (esx.audit.ssh.enabled); 4 failures so far.

2014-03-16T17:37:49.302Z: [UserWorldCorrelator] 1791016363us: [vob.uw.core.dumped] /bin/hostd(43796) /var/core/hostd-zdump.002

2014-03-16T17:39:26.000Z: Failed to send event (esx.audit.ssh.enabled); 5 failures so far.

2014-03-16T17:39:53.565Z: [UserWorldCorrelator] 1915279514us: [vob.uw.core.dumped] /bin/hostd(44418) /var/core/hostd-zdump.003

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

Hostd.log

2014-03-14T16:29:32.452Z [5BA81B70 error 'Solo.VmwareCLI' opID=hostd-6425 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-14T16:29:33.923Z [5BE40B70 error 'Solo.VmwareCLI' opID=hostd-d756 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-14T16:29:34.278Z [5BE40B70 error 'Solo.VmwareCLI' opID=hostd-40c5 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-14T16:30:27.373Z [FFF5BB70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x5b629400, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-14T16:30:27.373Z [5B481B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe7bdc0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-14T16:36:02.878Z [FFD315B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-14T17:01:08.348Z [FFA955B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-14T17:01:15.626Z [3AB80B70 error 'Default'] prctl(PR_SET_NAME, '') failed; errno: 22(Invalid argument)

2014-03-14T17:01:15.745Z [FFA955B0 error 'Statssvc.vim.PerformanceManager'] Error while loading stats depot: "FileIO error: Could not find file  : /var/lib/vmware/hostd/stats/hostAgentStats.idMap". Possible file corruption!

2014-03-14T17:01:15.746Z [FFA955B0 error 'Statssvc.vim.PerformanceManager'] Unable to load stats depot hostAgentStats in directory /var/lib/vmware/hostd/stats/ : FileIO error: Could not find file  : /var/lib/vmware/hostd/stats/hostAgentStats.idMap

2014-03-14T17:01:17.450Z [FFC40B70 error 'Solo.VmwareCLI' opID=hostd-288b user=root] GetPrimitiveParam: Cannot find (help)

2014-03-14T17:01:18.933Z [FFAF8B70 error 'Solo.VmwareCLI' opID=hostd-ca86 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-14T17:01:19.292Z [3A2C2B70 error 'Solo.VmwareCLI' opID=hostd-42be user=root] GetPrimitiveParam: Cannot find (help)

ResourceGroup: Skipping resource group: ResourceGroupImpl (3834) in group 21. Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information

2014-03-14T17:02:12.370Z [3A2C2B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x3a72e700, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-14T17:02:12.370Z [3AE40B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe94e48, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-14T17:07:20.017Z [FFB975B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-14T17:09:04.135Z [FFDAE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-14T17:10:49.473Z [FFAF35B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T16:06:52.666Z [FF8A25B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T16:06:59.924Z [776DAB70 error 'Default'] prctl(PR_SET_NAME, '') failed; errno: 22(Invalid argument)

2014-03-16T16:07:00.046Z [FF8A25B0 error 'Statssvc.vim.PerformanceManager'] Error while loading stats depot: "FileIO error: Could not find file  : /var/lib/vmware/hostd/stats/hostAgentStats.idMap". Possible file corruption!

2014-03-16T16:07:00.046Z [FF8A25B0 error 'Statssvc.vim.PerformanceManager'] Unable to load stats depot hostAgentStats in directory /var/lib/vmware/hostd/stats/ : FileIO error: Could not find file  : /var/lib/vmware/hostd/stats/hostAgentStats.idMap

2014-03-16T16:07:01.744Z [772EAB70 error 'Solo.VmwareCLI' opID=hostd-5fd2 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-16T16:07:03.225Z [FF905B70 error 'Solo.VmwareCLI' opID=hostd-fae0 user=root] GetPrimitiveParam: Cannot find (help)

2014-03-16T16:07:03.586Z [772EAB70 error 'Solo.VmwareCLI' opID=hostd-7d5e user=root] GetPrimitiveParam: Cannot find (help)

2014-03-16T16:07:56.662Z [FF987B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x77301be8, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-16T16:07:56.662Z [FF946B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x775026e0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-16T16:13:38.441Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x774037e8, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:18:38.448Z [772EAB70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x7750b420, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:23:38.456Z [772A9B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa00668, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:27:17.024Z [FF905B70 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0xffc0a718, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-16T16:28:38.462Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x775026a8, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:33:38.472Z [FF987B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x774667f8, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:38:38.480Z [FF905B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa06358, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:43:38.488Z [772EAB70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc2e428, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:48:38.497Z [FF946B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc0a718, h:45, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:53:07.520Z [FF946B70 error 'Solo.HttpSvc.HTTPService'] Failed to read request; stream: <io_obj p:0x1fe5e6a8, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>, error: N7Vmacore16TimeoutExceptionE(Operation timed out)

2014-03-16T16:53:38.506Z [FF987B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffa01878, h:36, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T16:58:38.514Z [77D80B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0xffc19560, h:54, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T17:03:38.522Z [77D80B70 error 'SoapAdapter.HTTPService.HttpConnection'] Failed to read header on stream <io_obj p:0x77c352c0, h:56, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

2014-03-16T17:09:07.509Z [FFA685B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:11:18.098Z [FF9A05B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:13:12.660Z [FFF435B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:15:04.433Z [FFC2D5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:17:02.503Z [FFEFE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:18:55.233Z [FFEEE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:20:48.289Z [FFD825B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:22:43.148Z [FFC555B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:24:33.553Z [FF9995B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:26:25.182Z [FFF215B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:28:28.635Z [FF9D95B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:30:23.165Z [FFDC95B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:32:14.092Z [FFBF05B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:34:13.200Z [FFCCE5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:35:57.746Z [FFB845B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:37:50.235Z [FFCE65B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:39:54.468Z [FFE285B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

2014-03-16T17:42:15.881Z [FF94E5B0 error 'Hostsvc.NetworkProvider'] Opaque network data cannot be loaded from config file /etc/vmware/hostd/opaqueNetwork.xml: no element found.

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

rhttplog

2014-03-16T17:44:24.686Z [FF946B70 verbose 'Proxy Req 01417'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:38948'>>)>

2014-03-16T17:44:24.687Z [FF8A1D70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb037d8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:24.687Z [FF8A1D70 warning 'Proxy Req 01417'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:28.266Z [FF987B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb00620, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:42886'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

2014-03-16T17:44:28.277Z [FF946B70 verbose 'Proxy Req 01419'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:60784'>>)>

2014-03-16T17:44:28.278Z [FF987B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb038c8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:28.278Z [FF987B70 warning 'Proxy Req 01419'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:31.854Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb01260, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:52406'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

2014-03-16T17:44:31.868Z [FF946B70 verbose 'Proxy Req 01421'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:13831'>>)>

2014-03-16T17:44:31.869Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb03a08, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:31.869Z [FF905B70 warning 'Proxy Req 01421'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:35.441Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffb01260, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:14593'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

2014-03-16T17:44:35.453Z [FF987B70 verbose 'Proxy Req 01423'] New proxy client <SSL(<io_obj p:0xffc1b7f8, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:36746'>>)>

2014-03-16T17:44:35.454Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc2dad8, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:35.454Z [FF905B70 warning 'Proxy Req 01423'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:39.032Z [FF946B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:47755'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

2014-03-16T17:44:39.043Z [FF987B70 verbose 'Proxy Req 01425'] New proxy client <SSL(<io_obj p:0xffc012f0, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:61960'>>)>

2014-03-16T17:44:39.044Z [FF946B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc1b420, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:39.045Z [FF946B70 warning 'Proxy Req 01425'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:40.312Z [FF946B70 verbose 'Proxy Req 01426'] New proxy client <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:53555'>>)>

2014-03-16T17:44:40.313Z [FF946B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffa1d708, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:40.313Z [FF946B70 warning 'Proxy Req 01426'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:40.327Z [FF987B70 verbose 'Proxy Req 01427'] New proxy client <SSL(<io_obj p:0xffb04950, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:37426'>>)>

2014-03-16T17:44:40.328Z [FF987B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffb03ac0, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:40.328Z [FF987B70 warning 'Proxy Req 01427'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

2014-03-16T17:44:42.617Z [FF905B70 warning 'Default'] SSL Handshake failed for stream <SSL(<io_obj p:0xffc2c228, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:39162'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

2014-03-16T17:44:42.630Z [FF987B70 verbose 'Proxy Req 01429'] New proxy client <SSL(<io_obj p:0xffc2d230, h:13, <TCP '127.0.0.1:443'>, <TCP '127.0.0.1:36602'>>)>

2014-03-16T17:44:42.631Z [FF905B70 warning 'Default'] Failed to connect socket; <io_obj p:0xffc2db30, h:14, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8307'>>, e: system:111(Connection refused)

2014-03-16T17:44:42.631Z [FF905B70 warning 'Proxy Req 01429'] Connection to localhost : 8307 failed with error N7Vmacore15SystemExceptionE(Connection refused).

Best regards, Martin Meuwese
Reply
0 Kudos
amiterjain
Contributor
Contributor
Jump to solution

Hi Martin,

Could you please check the hardware compatibility list. I could not see Dell 1955 blade center server support in 5.5U1 HCL.

Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

Will do. Probably now time to start replacing it.

Martin

Sent from my iPhone

Best regards, Martin Meuwese
Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

Today i received a reply that the 1955 DELL Blade servers are no longer in the HCL which could be the cause for the issues to surface. I will close this thread as it is running on non-supported hardware and revert back to 5.5 1331820 as this has been running stable for a long time.

HCL.png

Best regards, Martin Meuwese
Reply
0 Kudos
pasikappext
Contributor
Contributor
Jump to solution

Hello,

I don't think this problem is related to hardware. I'm seeing the exact same hostd crash caused by sw-iSCSI on ESXi 5.5U1 and this is with supported (on-HCL) hardware.

Any ideas how to fix this?

Reply
0 Kudos
pasikappext
Contributor
Contributor
Jump to solution

More information.. to reproduce:

1) Install ESXi 5.5 U1 host.

2) Verify you can connect to the host with vSphere Client.

3) ssh into the host and run "esxcli iscsi software set --enabled=true". Or enable sw-iSCSI client using the vSphere Client.

4) Reboot the ESXi host.

5) Notice you can't connect to the host with vSphere Client anymore.

6) ssh into the host and check the process list with "ps". You'll notice "hostd" is not there. "hostd" now crashes on every ESXi host reboot.

Reply
0 Kudos
pasikappext
Contributor
Contributor
Jump to solution

Ok, so it looks the hostd iscsi crash only happens when installing the ESXi 5.5u1 host from "VMware-ESXi-5.5U1-RollupISO.iso".

If I install the host from "VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso" then iscsi seems to work OK!

I wonder what's wrong with the RollupISO ..

Reply
0 Kudos
M_Meuwese
Enthusiast
Enthusiast
Jump to solution

I Will verify tomorrow and post my findings once done. Let's hope it is that simple...

Sent from my iPhone

Best regards, Martin Meuwese
Reply
0 Kudos
Wingsup
Contributor
Contributor
Jump to solution

I just wanted to throw this out here, I had the same issue in my lab, older unsupported hardware also. The rollup iso caused the same issue with an iscsi crash for me, and the the version  mentioned by user pasikappext works for me. "VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso

Reply
0 Kudos
Michael_Rudloff
Enthusiast
Enthusiast
Jump to solution

Same here, ISCSI initiator crashes the system (using rollup iso). Using Supermicro Hardware.

___ My own knowledge base made public: http://open902.com
Reply
0 Kudos
dpg2vgs
Contributor
Contributor
Jump to solution

Another "me too" on this one. All of our equipment (Dell PowerEdge x20 servers, PowerConnect 70xx switches, and EqualLogic PS6xxx storage running 7.02 firmware) is on the HCL. I spent a fair bit of time with VMware support over the last three days troubleshooting with our spare server and a test volume. The hostd process is usually completely shot, as noted by pasikappext

I found this thread and tried the non-rollup ISO with our usual iSCSI setup script (leveraging Dell's MEM 1.2 TR1091 configuration script, but not installing the module) and had instant success. Subsequent reboot and power-pull tests on the host worked fine, the iSCSI volumes mounted with a Round Robin policy and the test VMs fired-up as they should. Performance seems typical although I haven't run the IOmeter benchmarks yet.

I think VMware ought to pull the rollup ISO or clearly indicate that it may not work on all HCL systems and the referenced ISO is preferred.

Thanks for starting this thread, Martin. Good luck with the non-rollup ISO and the 1955.

AlexanderCapald
Contributor
Contributor
Jump to solution

i have just updated 3/4 of my esx 5.5 hosts to update 1, via the vsphere update manager, and when i got to the iscsi hba properties it works.

Hardware: Supermicro H8QME-2+ 128GB ram

what sequence of events cause yours to crash?

Reply
0 Kudos
dpg2vgs
Contributor
Contributor
Jump to solution

A reboot or power-cycle after the update and the setup of the iSCSI interface would do it, if, and only if, your installation was a fresh installation from the “rollup” ISO.

If you used the update manager you may not have a problem. Just try a reboot and see what happens.

Reply
0 Kudos
Michael_Rudloff
Enthusiast
Enthusiast
Jump to solution

I can't even get the iscsi initiator installed on a fresh installation. Bear in mind, the Rollup ISO does state anyway that it is not to be used for upgrades. Regardless, if I install the Rollup ISO on any server we have, HCL or not HCL validated, the hostd process is trashed as soon as the iscsi initiator is enabled.

___ My own knowledge base made public: http://open902.com
Reply
0 Kudos