ESXi standalone hostd broken after 7.0 Update 3f upgrade

[bgwallace]

Team,

Seems there were changes made to hostd in the 7.0 Update 3f release and perhaps this is the culprit, but I've been running on this version fine for a couple of weeks and then all of a sudden sometime last night my VMs froze up and after a reboot of the host the ESXi Host Client Web UI will no longer load and VMs won't startup.  When trying to restart hostd, I get the error below.

[root@VM-Server:~] /etc/init.d/hostd stop

hostd is not running.

[root@VM-Server:~] /etc/init.d/hostd start

[ConfigStore:1021793434752:] [1080] Failed to parse JSON String

hostd started.

[root@VM-Server:~] /etc/init.d/hostd restart

watchdog-hostd[1064992]: Unable to terminate watchdog: No running watchdog process for hostd

sh: you need to specify whom to kill

[ConfigStore:273473019008:] [1080] Failed to parse JSON String

hostd started.

It says hostd is started, but it doesn't really start and you can see there is an error coming back from the config store.  This is confirmed from the similar error given when running the following command:

[root@VM-Server:~] /bin/configstorecli config current get -c esx -g services -k hostd

Error Code: 1080

Message: Failed to parse JSON String

I'm unable to run any esxcli commands presumably because hostd isn't running.

[root@VM-Server:~] esxcli --help

Connection failed

I'm hoping someone has an idea on how to get past this issue.  My host is completely unusable until I can fix and I won't be able to perform a simple upgrade when a new version comes out because esxcli commands don't work.

Thank you in advance for your help!

Brad

[bgwallace · Accepted solution]

I tried to find a way to view whatever data was in the ConfigStore for the hostd key so that I could attempt to restore the exact same data that was there before the issue but I couldn't find any answers on how to do this so I eventually decided to simply restore the key data with the default data after getting the idea from the article below.

https://kb.vmware.com/s/article/82227

To do this, I ran the commands below .

[root@VM-Server:~] /bin/configstorecli config default get -c esx -g services -k hostd -outfile tmp.json
Data written to tmp.json

[root@VM-Server:~] /bin/configstorecli config current set -c esx -g services -k hostd -infile tmp.json
Set: completed successfully

[root@VM-Server:~] rm tmp.json
rm: remove 'tmp.json'? Y

[root@VM-Server:~] /etc/init.d/hostd status
hostd is not running.

[root@VM-Server:~] /etc/init.d/hostd start
hostd started.

[root@VM-Server:~] /etc/init.d/hostd status
hostd is running.

[root@VM-Server:~] reboot

I'm not sure if a reboot was necessary. After the server came back up, everything was back to normal! The Web UI would load, the VMs started up and I could run esxcli commands again.