VM Host in Cluster Went into a Disconnected State

[JohanJJF]

Hi Guys

Need some help here as the Company now expects me to do a RCA (Root Cause Analysis) as to why one of our VM Hosts went into a disconnected state. I am no Fundy when it comes to analyzing logs, and VM LOGS are totally hieroglyphics to me. Here is what Happened:

Received an Alert that one of our VM Hosts went into a Disconnected (Isolated State) – Host was unreachable on the Network although all VM’s (Guests) was still Booted and Operational without Network loss. In vSphere client both Hosts were visible in the Cluster and the One host had a small red cross on it with light greyed out text saying disconnected. The Guests on that Host showed the same. When right clicking on the Host and selecting Connect brought up an Error Message saying the Host doesn't exist or could not be found. So I manually shutdown all the VM's with RDP and then Rebooted the Host. After the host was up I could connect it to the Cluster without problems and start up all the VM Guests on that HOST. So basically the Problem is solved but i need to know why and how this happened and provide concrete evidence

Hope you guys can Help!!

[julienvarela]

Hi,

Can you attached the hostd.log file and vmkernel.log of your host.

Regards,

Julien.

[JohanJJF]

Hi Julien

Sorry you have to be more specific since I don't have a Clue where to find these logs (please specify paths/folders), I have exported the logs of that HOST from vSphere Centre and extracted them on my PC but there are so many files and folders I don't know where to find what you need

[julienvarela]

No problem

So, extract the zip file and then you will find the log file into /var/run/log.

Maybe attached the hostd.0.gz ,vpxa.0.gz and vmkernel.0.gz files too. (Depending on your event time).

Then, can you tell me what is the exact time of the issue.

Regards,

Julien.

[JohanJJF]

Mmm I extracted the files and Unfortunately I don't have a "Run" Dir under the "Var" Directory ...... See screenshot.

[julienvarela]

Just a question, what is your ESXi or ESX version.

Check on your log directory because maybe you haven't a persistent crash space.

Otherwise, you can enable SSH on your Host :

Use the vSphere Client to enable local and remote access to the ESXi Shell:

1. Log into a vCenter Server system using the vSphere Client.
2. Select the host in the inventory panel.
3. Click the Configuration tab and click Security Profile.
4. In the Services section, click Properties.
5. Select SSH from this list.
6. Click Options and select Start and stop manually.
   Note: When you select Start and stop manually, the service does not start when you reboot the host. If you want the service to start when you reboot the host, select Start and stop with host.
7. Click Start to enable the service.
8. Click OK.

Then using putty you can connect to your host and located your log files (In /var/log or /var/run/log). Then using Winscp for example you will be able to export it on your pc.

Regards,

Julien