VMware Communities > VMTN > VMware Infrastructure™ > VI: ESX 3.5 > Discussions

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
10 Replies Last post: Mar 12, 2009 7:55 AM by Rynardts
Reply

ESX Service Console Stop Working

Jan 28, 2009 6:46 PM

Click to view raytheman's profile Novice raytheman 11 posts since
Oct 12, 2007

I have a ESX 3.5, I can ping the service console IP but I can't connect to it, all the VMs are still up and running though. I have direct access to the server's console (Alt+F1) .

The vSwitch0 are used by the service console and VMs, the fact that all the VMs are still working makes me believe that all I need to do is just re-activating the service console.

Is there a way to re-activate without affecting the VMs? Thanks.

Reply Re: ESX Service Console Stop Working Jan 28, 2009 8:34 PM
Click to view weinstein5's profile Guru weinstein5 6,208 posts since
Nov 19, 2005
VMware
Since you have direct access to the service console run the esxcfg-nics -l, esxcfg-vswitch -l and esxcfg-vswif -l commands and post the output - to answer a question yes you will be able to reset the service console networking but first want to see how things are connected - to see more about these commands you can check this site out http://vmware-land.com/esxcfg-help.html

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

Reply Re: ESX Service Console Stop Working Jan 29, 2009 9:23 AM
in response to: weinstein5
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

If you can login to the service console and there is no OOPS on the screen (LInux Crash) then you most likely have a network issue. If it is a crash, you can only reboot the host to get back the SC. Which means cleanly shutting down the VMs usually....


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links
Reply Re: ESX Service Console Stop Working Jan 29, 2009 10:18 AM
Click to view raytheman's profile Novice raytheman 11 posts since
Oct 12, 2007
OK, now I managed to SSH into the ESX via the service console IP, so there shouldn't be any network issue. However, I still can't connect via VMWare Infrastructure Client, I have tried the following steps with no luck,

--root@esx1234 /--# service mgmt-vmware status
vmware-hostd is stopped

--root@esx1234 /--# service mgmt-vmware start
Starting VMware ESX Server Management services:
VMware ESX Server Host Agent is already running
Availability report startup (background) OK

--root@esx1234 /--# service mgmt-vmware status
vmware-hostd is stopped

All the VMs are still up and running, so rebooting will be the very last option that I want to take. Thanks.

Reply Re: ESX Service Console Stop Working Jan 29, 2009 10:43 AM
in response to: raytheman
Click to view Lightbulb's profile Virtuoso Lightbulb 1,389 posts since
Aug 15, 2008
Anything logged in /var/log/vmware/host.d

Try tail -25 /var/log/vmware/host.d from the cli
Reply Re: ESX Service Console Stop Working Jan 29, 2009 10:54 AM
in response to: Lightbulb
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

Run the command 'service mgmt-vmware restart' from the SC to fix the problem. Status of stopped means it could have crashed and therefore it will not start properly unless some files are cleaned up. Restart will do that.

Also look in /var/log/vmware/hostd.log for issues. As well as look in /var/core to see if that has files.

You may also need to do: service vmware-vpxa restart


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links
Reply Re: ESX Service Console Stop Working Jan 29, 2009 11:16 AM
Click to view raytheman's profile Novice raytheman 11 posts since
Oct 12, 2007
--root@esx1234 root--# service mgmt-vmware restart
Stopping VMware ESX Server Management services:
VMware ESX Server Host Agent Services OK
VMware ESX Server Host Agent Watchdog FAILED
VMware ESX Server Host Agent OK
Starting VMware ESX Server Management services:
VMware ESX Server Host Agent is already running
Availability report startup (background) OK
--root@esx1234 root--# service vmware-vpxa restart
Stopping vmware-vpxa: OK
Starting vmware-vpxa: OK

Still no luck. Meanwhile, I came across two old posts and appearently the easiest/fastest solution is reboot (even suggested by the support), or restart xinetd service.

Will restart xinetd service affect my VMs?

http://communities.vmware.com/thread/119675

http://communities.vmware.com/message/828264#828264

Reply Re: ESX Service Console Stop Working Jan 29, 2009 1:52 PM
in response to: raytheman
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

I noticed that the watchdog kill failed. Run the following commands, is there any output.... you are looking for defunct or zombie processes, if you have any, you will have to reboot the host to clear the issue.

ps ax | grep D | grep -v grep
ps ax | grep Z | grep -v grep


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links
Reply Re: ESX Service Console Stop Working Jan 29, 2009 1:57 PM
in response to: Texiwill
Click to view raytheman's profile Novice raytheman 11 posts since
Oct 12, 2007

Here are the result.

root@esx1234 root# ps ax | grep D | grep -v grep
PID TTY STAT TIME COMMAND
21604 ? S 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf /smhpd.conf
21620 ? S 0:00 /opt/hp/hpsmh/sbin/hpsmhd -DSSL -f /opt/hp/hpsmh/conf /smhpd.conf
root@esx1234 root# ps ax | grep Z | grep -v grep
root@esx1234 root#

Reply Re: ESX Service Console Stop Working Jan 29, 2009 4:28 PM
in response to: raytheman
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

That is not the problem then..... However, try 'service hpasm restart' to restart HPASM as it sometimes causes issues. Also, do 'service mgmt-vmware stop' then look for watchdog processs and kill it. Since the restart could not kill the watchdog process it may be an issue. Also look for a hostd process that may be hanging out and kill it. Then 'service mgmt-vmware start'


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs
Top Virtualization Security Links
Reply Re: ESX Service Console Stop Working Mar 12, 2009 7:55 AM
in response to: raytheman
Click to view Rynardts's profile Enthusiast Rynardts 97 posts since
May 10, 2006

I've had the same issue:

----root@esx1234 /--# service mgmt-vmware status
vmware-hostd is stopped

--root@esx1234 /--# service mgmt-vmware start
Starting VMware ESX Server Management services:
VMware ESX Server Host Agent is already running
Availability report startup (background) OK

--root@esx1234 /--# service mgmt-vmware status
vmware-hostd is stopped

I found that the PID files weren't cleaned up after the hostd agent was stopped the last time. Therefore when you try to restart the agent with service mgmt-vmware restart, the script finds what it thinks is a "life" PID for the vmware-hostd daemon, and therefore gives the "VMware ESX Server Host Agent is already running" line.

This is what I had to do to get past this:

If the commands "ps -auxwww |grep -i hostd" or "pidof vmware-hostd" returns no process IDs , you know that the management agent is not running! We therefore need to find the files that reports a false PID.

As the root user, do:

cd \var\run\vmware
ls -l

If any of the following files exists, delete them:
vmware-hostd.PID
watchdog-hostd.PID

Once you are sure those files are deleted, try restarting the management agents again:

service mgmt-vmware restart

-or-

/etc/init.d/mgmt-vmware restart

Hope this helps a bit.

Regards

Rynardt Spies
VCP, vExpert
www.virtualvcp.com


Actions