ESX host disconnected, web access fail.

[ckboon]

Hi,

One of my ESX host shows up as disconnected when viewed by the VCMS. But I can ping the host and all the VMs running under that host as well as ssh to that host. However, I cannot control the VMs or the host via VI client or browser. "Netstat -an" does not list port 80/8080/443, so that means the web server is dead.

I have tired to restart the following without success.

1) S85vmware-webAccess

2) S98mgmt-vmware

3) S99vmware-vpxa

Short of shutting down all the VMs and reboot the host, what else can I do to restart at least the web server?

Thanks for the help.

Regards,

[jdvcp]

I have run into similar scenarios, all related to too little service console memory (272 MB defaut). You can see in esxtop if physical mem is too low or if swap mem is too full. Also, look at swap in/out activity. check /var/log/messages for any processes which were killed. check /var/log/vmksummary - go to bottom, then all the way to the right. It will show you top 3 processes in use. If you see VMAP, this is buggy code and this is a runaway process eating up mem. Kill it, then restart the services as you did before.

If you have any agents running in the service console, consider turning them off (stop or kill service/processes). This could free up enough memory for you to turn other critical services back on.

Have vmware support walk you through correct service bounce/turn-up procedures to get this host back on VC.

[ckboon]

Not much via /var/log/messages & /var/log/vmksummary.

I tired to use vmware-cmd -l and it gave me this error:

"[root@esx04 /]# vmware-cmd -l

/usr/bin/vmware-cmd: Could not connect to server

(VMControl error -3: Invalid arguments: Login failed.)

#"

It looks like some major error. Anyway, I have to reboot the host in order to increase the service console memory to 384MB.

[weinstein5]

If you are resizing your service console memory - do not forget to resize he service console swap file to two times the service console to if you set the memory to 364 the service console will have to be at least 768

[ckboon]

Thanks for the heads-up.

I checked my swap size:

# free -m

Swap: 1027 4 1023

I have 1GB of swap space. That ought to be enough.

[ckboon]

I have rebooted my ESX host and I still can't connect to it via VI Client.

When I tired to use the VI Client connect wizard using IP / hostname, I get an error "Unable to access the specific host..."

However, I can ssh to the host though.

I have tired to restart "mgmt-vmware" but it keeps dying after a while.

1. /etc/rc3.d/S98mgmt-vmware status

vmware-hostd is stopped

I have also tired to add "AAMClient" to the firewall permit list, but it doesn't seem to take effect.

What can I do?

[ckboon]

Found out while the stupid hostd keeps dying.

I made a backup copy of the "/etc/vmware/firewall/services.xml" in the same directory. So hostd / firewall keeps trying to load both copies.

Removed the backup copy, restart "S98mgmt-vmware".

HA is now running, I can connect via VI Client, etc.

Duh!

[Brian_Wing]

How does one go about resizing the swap partition on this badboy? It looks like I didnt' allocate enough memory for the service console, and I've modified that (haven't rebooted yet tho)

Thanks

brian

[weinstein5]

in this thread there is discussion on how adjust the swap space - http://communities.vmware.com/message/484615

[Brian_Wing]

Thanks for this, I'm guessing though that by allowing the partitioning to be default at install I may have painted myself into a corner. I'm not super keen on the idea of trying a "file" swap space, I may just end up rebuilding an existing ESX server and manually partitioning.

Do you have any recommendations on partition layout? Unfortunatly we're using internal disk for VMs and won't be moving to SAN storage for a while. I do have ample disk space locally 1.5T or so.

[weinstein5]

yep painted inot a corner - I always setup my partions with a swap space of 2 GB so that if I need to increase service console memory I just reset the memory and reboot the host - the rest of of the default partition sizes are fine -