VMware Cloud Community
cypherx
Hot Shot
Hot Shot

Direct console access disabled but hostd is crashed and cannot use vSphere client to enable it

Is there a way to enable direct console access to an ESXi 6.0 u2 host server if the hostd process is crashed? 

If hostd is crashed vSphere client cannot connect to adjust the settings of ESXi to enable direct console or ssh.

7 Replies
Nick_Andreev
Expert
Expert

You can use remote management capabilities of your server, such as iDRAC in Dell or iLO in HP, to connect to the console and make necessary changes.

---
If you found my answers helpful please consider marking them as helpful or correct.
VCIX-DCV, VCIX-NV, VCAP-CMA | vExpert '16, '17, '18
Blog: http://niktips.wordpress.com | Twitter: @nick_andreev_au
Reply
0 Kudos
cypherx
Hot Shot
Hot Shot

‌yes tried that but when I pressed F2 it says admin disabled console access.  I never disabled console access!  Is this a new feature of esxi 6?  This was upgraded from 5 a week ago.

Reply
0 Kudos
Nick_Andreev
Expert
Expert

This means that DCUI is disabled. Which can be re-enabled via vCenter only.

Are you able to reboot the server to re-establish connection to vCenter? Because otherwise you'd have to rebuild the host.

---
If you found my answers helpful please consider marking them as helpful or correct.
VCIX-DCV, VCIX-NV, VCAP-CMA | vExpert '16, '17, '18
Blog: http://niktips.wordpress.com | Twitter: @nick_andreev_au
Reply
0 Kudos
NetworkNerd
Enthusiast
Enthusiast

Is there a chance you have vMotion / storage vMotion and could move the VMs to another host temporarily?  I'm not saying this is the best solution, but if you moved all VMs off, you could power off the host using your onboard management controller (like iDRAC or iLO) or even hold down the power button and then power the host on again to see if it boots up and services start as expected.  Then you could turn off lockdown mode for the host through vCenter or even just enable SSH as a back door for future use.  That "might" allow the host to not have to be rebuilt.

Reply
0 Kudos
cypherx
Hot Shot
Hot Shot

Ok here is the issue we ran into.

Seems there is a bug in VMware ESXi 6 with vSphere Replication 6.1.  We have a VM that was replicating to a remote site.  An oops on us is we noticed both disks were replicating (1.2tb).  We didn't need the second disk replicating (its just sql backups that veeam captures anyway).  So we modified the replication schedule for this VM to not include the second disk (in windows it would be drive D:).  I'm not sure if this was the exact chain of events that linked us to a new bug due to be patched next month from VMware.  Aparently whatever 6.0 host this VM is running on, hostd service will in time crash.  Once it crashes there's no way to start it, except for removing the entry of this VM from /etc/vmware/hostd/vmInventory.xml.  If you reboot the ESXi server, as soon as it reads in that inventory and the affected VM files are read, hostd crashes.  With hostd crashed vSphere client cannot connect to the ESXi server either directly or through the vCenter server.  If you cannot use the client to connect you cannot go to the configuration tab and enable services like SSH, DCUI, etc...  So I'm in a catch 22.

The only thing I can think of is await some instruction from VMWare on our open SR# on the original issue (ESXi 6.0 hostd crashing due to bug in VR 6.1).  If curious hostd.log crashes with a backtrace following this: Panic: Assert Failed: "!repDiskInfo->GetDiskReplicationId().empty()" @ bora/vim/hostd/hbrsvc/ReplicationConfig.cpp:769

So maybe the only way to get this ESXi server back up is to RDP to the affected VM (its still running fine actually), shut it down, then in datastore browser or SSH to another ESXi host rename its folder, then reboot this affected ESXi host from the console.  Renaming the VM's folder would make this ESXi host not see it upon reboot, so hopefully hostd would stay started and I could connect to it and re-enable DCUI and SSH (Which I never disabled - maybe upgrading it to 6.0 automatically did that).

I can then re-import the affected VM to another ESXi host, maybe one of mine still running 5.0, 3086167 where this bug does not seem to be affected.  The bug seems to crash hostd on the destination VM during a host vmotion or after about 90 mintues of just running normally on an esxi 6.0 host.

Reply
0 Kudos
cypherx
Hot Shot
Hot Shot

I ended up getting the host back up and enabling esxi shell, dcui console and ssh.

I know the best practices are just to enable those things when needed for security, however if hostd crashes you can't enable them.  I'd rather keep them enabled and just use access lists on the switch so only IT computers can connect to them, paired with strong root passwords.

To get the host back up, I powered off the VM running on it via Microsoft RDP.  Renamed it's folder from another ESXi host SSH, so upon reboot this host wouldn't see it and automatically crash hostd.  Created a new VM with the same settings but attached the disks from the old VM imported to another host.  Everything is running now.

Thanks for your input!

NetworkNerd
Enthusiast
Enthusiast

It sounds like you had an adventure on this one.  I'm glad you got it sorted and can certainly understand why you left ESXi Shell / SSH access enabled.

Reply
0 Kudos