vmware 5.5 u1 - iScsi adapter crashes the system

[M_Meuwese]

Hello all,

Last week i have not been successful in upgrading the VMware environment from 5.5 to 5.5U1. Initial installation went fine, configured the first ESXi host and created a host profile. So far so good, no major issues. When installing the remainder of the 5 other blade servers and applying the host profile all seems to be working well whereas things went sideways when the entire environment was shutdown and restarted.

Te setup looks as following:

Dell 1955 blade center servers (6x)

Cisco 3030 switches connected to the blades

vCenter server as a VM on the ESXi cluster

vCenter upgrade manager as a VM

Distributed Virtual Switch configured

During the shutdown procedure i move all the vCenter components to a single host before powering down the vCenter server (Windows 2008). Then i shutdown the ESXi host using the management client as vCenter is no longer available.

Power up cycle is to start this ESXi host first, connect to it and then powerup the vCenter components. Once all is running DRS and Power management is configured to automatic after which the simulation VM's are started.

With 5.5 all was working well and no issues at all. When i installed 5.5 U1 i could no longer connect to the ESXi server and it gave an error 503 when connecting to it. Going through the log files showed that components on the file system could not be found after which i decided to do a clean install.

After reinstalling the whole lot i connected to a ESXi host and enabled the network and added the iScsi adapter. At that stage the system becomes unresponsive and the session between the ESXi host and the client is disconnected.

Running several tests i concluded that something must be wrong with the virtual disk system components, iScsi or vSan. 5 of the 6 hosts will be rolled back to vmware 5.5 as vmware 5.5U1 is not stable...

Can use some help troubleshooting this issue as i will leave 1 server in the current state it is to identify the root cause as i'm stuck at the moment. Later on i will post some log file sections with the errors i identified. (Restarting the hostd does not resolve the problem)

Regards,

Martin Meuwese

[pasikappext]

Fresh ESXi 5.5U1 installations from "VMware-ESXi-5.5U1-RollupISO.iso" are buggy, enabling sw iSCSI there, and rebooting, crashes hostd on every reboot rendering the hosts unmanageable.

Using "VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso" instead works OK.

[AlexanderCapald]

lets compare your esxi 5.5.0 u1 os build numbers, i managed to update all my servers via the vsphere update manager without any issues, iscsi initiator still intact.

[mcorpolo]

I'm seeing the same issue as well.  Building new DL360 Gen 8 servers, built from a 5.5 disk then used Update Manager to patch up.  During setup and scan for iSCSI the machine hangs with the previously mentioned symptoms of not even being able to use F2/F12 at console of server to reboot.

Power cycled the server and on boot it takes FOREVER.  Gave up, unzoned the iSCSI initiators at my array, and the server then quickly finished booting.

Will attempt a fresh install, again, this time from a 5.5u1 fresh media.

[M_Meuwese]

Hello All,

Thank you for all the effort and replies on this post. Today is then finally the day that i have time to try this 1 more time with the recommended CD image. Once completed i will post my findings whereas i suspect all will do just fine.

Regards,

Martin Meuwese

[peetz]

Hi all,

I could reproduce the issue by installing a nested ESXi host with the 5.5 U1 Driver Rollup ISO.

Creating a Software iSCSI adapter will crash the hostd daemon and make the host unmanageable.

I was able to track down the issue to the scsi-teradimm VIB that is included in this ISO.

Once you have removed it by running

esxcli vib remove -n scsi-teradimm

in an ESXi shell and reboot the host, the issue will no longer appear.

Andreas

[dabrams971]

Hi all,

I too spent a few days on this problem, it's defenetly not an HCL problem (tested with HCL and non HCL hardware. I totaly agree with Peetz, I was intrigued by the teradimm adapter which I had never seen before, removing it does resolve the problem. Also installing with the non Rollup ISO, I no longer see teradimm in the storage adapters and everything works perfectlly.

[peetz]

Today VMware has finally addressed the issue by publishing http://kb.vmware.com/kb/2075171

It looks like I was wrong with pointing to the scsi-teradimm VIB ...

Please note: It looks like VMware will not replace the ISO download with a fixed version. Why, VMware?!?

[vTagion]

wish I would have seen this sooner...

[ShoHRock]

Hostd service was crashing when VMware default ESXi ISO was being used. kernel logs: 2014-06-09T13:34:48.548Z cpu4:44586)User: 2888: wantCoreDump : hostd -enabled : 1 2014-06-09T13:34:48.657Z cpu4:44586)UserDump: 1820: Dumping cartel 44586 (from world 44586) to file /var/core/hostd-zdump.003 ... 2014-06-09T13:34:56.405Z cpu5:44586)UserDump: 1944: Userworld coredump complete. Using Dell custom image fixed the issue. This is not the first time using Dell Custom image has helped overcome such issues. Looks like Dell H/w likes Dell Custom image more then VMware default.

[PeteKowalsky]

When I painted myself into this awful corner, this was the ticket out:

# esxcli vib remove -n ima-be2iscsi

Connect to localhost failed: Connection failure

^-- seems to be due to hostd not running, which is the entire issue here.  Catch-22.  What to do?

DO THIS INSTEAD:

# localcli software vib remove --vibname ima-be2iscsi

Removal Result:

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: VMware_bootbank_scsi-blahblahblah.550.0.0.1331820

   VIBs Skipped:

# reboot

DONE!

LESSON LEARNED: NEVER update a host with the Driver Rollup ISO.  ALWAYS USE THE FULL INSTALLER TO UPDATE HOSTS.  Use the driver rollup disc to do fresh installs only.

[admin]

While doing research i came across similar issue, after performing all the advised activities in other forums finally i decided to update 5.5 U1 to the build number 1746018 and then attempted to create software iscsi adapter. After hours of observation hostd didn't crash and it was stable.

Also, i dont believe this is hardware compatibility issue as the host i worked on is HP BL460c G6 which is not compatible however the update fixed the issue.

Please hit like if this has resolved issue in your environment.

[dsalnikov]

PeteKowalskyPeteKowalsky...

I appreciatte your support, the execution of command:

# localcli software vib remove --vibname ima-be2iscsi

Resolve the issue i had in production server! I love you!!!!!!!!!

[dsalnikov]

Martin Meuwese, your excellent description of your problem helps me to solve my problem!

Thanks! an Congratulations!

Diego