ESX Service Console Stop Working

[raytheman]

I have a ESX 3.5, I can ping the service console IP but I can't connect to it, all the VMs are still up and running though. I have direct access to the server's console (Alt+F1) .

The vSwitch0 are used by the service console and VMs, the fact that all the VMs are still working makes me believe that all I need to do is just re-activating the service console.

Is there a way to re-activate without affecting the VMs? Thanks.

[weinstein5]

Since you have direct access to the service console run the esxcfg-nics -l, esxcfg-vswitch -l and esxcfg-vswif -l commands and post the output - to answer a question yes you will be able to reset the service console networking but first want to see how things are connected - to see more about these commands you can check this site out http://vmware-land.com/esxcfg-help.html

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[Texiwill]

Hello,

If you can login to the service console and there is no OOPS on the screen (LInux Crash) then you most likely have a network issue. If it is a crash, you can only reboot the host to get back the SC. Which means cleanly shutting down the VMs usually....

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links

[raytheman]

OK, now I managed to SSH into the ESX via the service console IP, so there shouldn't be any network issue. However, I still can't connect via VMWare Infrastructure Client, I have tried the following steps with no luck,

root@esx1234 /# service mgmt-vmware status

vmware-hostd is stopped

root@esx1234 /# service mgmt-vmware start

Starting VMware ESX Server Management services:

VMware ESX Server Host Agent is already running

Availability report startup (background)

root@esx1234 /# service mgmt-vmware status

vmware-hostd is stopped

All the VMs are still up and running, so rebooting will be the very last option that I want to take. Thanks.

[Lightbulb]

Anything logged in /var/log/vmware/host.d

Try tail -25 /var/log/vmware/host.d from the cli

[Texiwill]

Hello,

Run the command 'service mgmt-vmware restart' from the SC to fix the problem. Status of stopped means it could have crashed and therefore it will not start properly unless some files are cleaned up. Restart will do that.

Also look in /var/log/vmware/hostd.log for issues. As well as look in /var/core to see if that has files.

You may also need to do: service vmware-vpxa restart

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links

[raytheman]

root@esx1234 root# service mgmt-vmware restart

Stopping VMware ESX Server Management services:

VMware ESX Server Host Agent Services

VMware ESX Server Host Agent Watchdog

VMware ESX Server Host Agent

Starting VMware ESX Server Management services:

VMware ESX Server Host Agent is already running

Availability report startup (background)

root@esx1234 root# service vmware-vpxa restart

Stopping vmware-vpxa:

Starting vmware-vpxa:

Still no luck. Meanwhile, I came across two old posts and appearently the easiest/fastest solution is reboot (even suggested by the support), or restart xinetd service.

Will restart xinetd service affect my VMs?

http://communities.vmware.com/thread/119675

http://communities.vmware.com/message/828264#828264

[Texiwill]

Hello,

I noticed that the watchdog kill failed. Run the following commands, is there any output.... you are looking for defunct or zombie processes, if you have any, you will have to reboot the host to clear the issue.

ps ax | grep D | grep -v grep

ps ax | grep Z | grep -v grep

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links

[raytheman]

Here are the result.

# ps ax | grep D | grep -v grep

PID TTY STAT TIME COMMAND

21604 ? S 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf /smhpd.conf

21620 ? S 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf /smhpd.conf

# ps ax | grep Z | grep -v grep

#

[Texiwill]

Hello,

That is not the problem then..... However, try 'service hpasm restart' to restart HPASM as it sometimes causes issues. Also, do 'service mgmt-vmware stop' then look for watchdog processs and kill it. Since the restart could not kill the watchdog process it may be an issue. Also look for a hostd process that may be hanging out and kill it. Then 'service mgmt-vmware start'

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links

[Rynardts]

I've had the same issue:

[root@esx1234] /--# service mgmt-vmware status

vmware-hostd is stopped

root@esx1234 /# service mgmt-vmware start

Starting VMware ESX Server Management services:

VMware ESX Server Host Agent is already running

Availability report startup (background) OK

root@esx1234 /# service mgmt-vmware status

vmware-hostd is stopped

I found that the PID files weren't cleaned up after the hostd agent was stopped the last time. Therefore when you try to restart the agent with service mgmt-vmware restart, the script finds what it thinks is a "life" PID for the vmware-hostd daemon, and therefore gives the "VMware ESX Server Host Agent is already running" line.

This is what I had to do to get past this:

If the commands "ps -auxwww |grep -i hostd" or "pidof vmware-hostd" returns no process IDs , you know that the management agent is not running! We therefore need to find the files that reports a false PID.

As the root user, do:

cd \var\run\vmware

ls -l

If any of the following files exists, delete them:

vmware-hostd.PID

watchdog-hostd.PID

Once you are sure those files are deleted, try restarting the management agents again:

service mgmt-vmware restart

-or-

/etc/init.d/mgmt-vmware restart

Hope this helps a bit.

Regards

Rynardt Spies

VCP, vExpert

www.virtualvcp.com