Connection failed: could not connect to host

[godzteeth]

ESX 3.5.0, 64607 - VIC 2.5.0, 84767

Hi. I hope someone can advise:

Our ESX server, running 7 VMs, recently started using up most of its physical memory, triggering CPU and memory usage alerts. It got to the point where VIC was not responding. I disconnected the host (which seemed like a good idea at the time) but now I cannot reconnect. The error I get when I try reconnect is: "Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem."

Fortunately the VMs are all still running.

I can connect to the VMs through web access. After enabling root access I can SSH to the ESX server successfully.

Using SSH, I restarted the physical server but it still refuses to connect. Furthermore, I get the message in SSH console "ALERT: Configuration changes were not saved successfully during previous shutdown. Please file a problem report including output of vm-support." (VIC -> Export diagnostic data reports ok for Virtual Center & VIC; ESX = log bundle hasn't been generated).

[MauroBonder]

try restart the service (service mgmt-vmware restart) and vpxa (service vmware-vpxa restart)

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[godzteeth]

Thanks - quick response!

I forgot to mention I had already tried that. It caused the 2 VMs that were using the most memory to instantly reboot!

I restarted the ESX server anyway in case there were other services that weren't responding but, as mentioned, no luck.

[COdlk]

Have you looked at any of the logs on the ESX host? Has anything changed on the VMs? you might try esxtop to help trouble shoot the problem.

david

[MauroBonder]

Once you have a session open, run the command vdf -h

It'll list out the partitions you have and will indicate if one of them is full.

http://communities.vmware.com/message/559640

**If you found this information useful, please consider awarding points for

"Correct" or "Helpful"**

[godzteeth]

Hi David,

i can find a mention of the error in the viclient logs. It doesn't tell me much though:

2009-06-15 17:52:30.738 An error occured running a task.

Vmomi.Fault.NoHost: Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem.

2009-06-15 17:52:40.520 Start Invoke 58 - Datacenter:datacenter-2.QueryConnectionInfo ...

2009-06-15 17:52:41.832 Finish Invoke 58 - Serial:0.001, Server:001.315 - Datacenter:datacenter-2.QueryConnectionInfo

Vmomi.Fault.NoHost: Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem.

vpxa.log shows

Creating temporary connect spec: localhost:443

Failed to discover namespace: Connection refused

Could not resolve namespace for authenticating to h$

messages log - I have no idea what to look for here!

the only oddity I can see is:

vmware-hostd[8182]: Accepted password for user root from 127.0.0.1

passwd(pam_unix)[8430]: password changed for vpxuser

ukha-esx01 vmware-authd(pam_unix)[8182]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= u$

vmware-hostd[8182]: Accepted password for user vpxuser from 127.0.0.1

last message repeated 2 times

On line 2 - password changed? This is repeated several time throughout the log.

Do you what else I can check? Incidentally, ping & nslookup do resolve the server name / ip.

[COdlk]

your DNS is working correct? Forward and reverse? Do you have anything in your /etc/hosts?

david

[godzteeth]

Thanks again. I can confirm the partitions all have plenty of space.

[godzteeth]

Hi David.

Yes, forward & reverse lookups are working- from the server that holds VIC: ping & nslookup resolve without any delays.

In hosts there is the loopback address & an internal IP: 192.168.x.y with the fqdn and the host name.

Just as a test I #'d out the internal IP. (Would I need to restart any services for that to take effect?) That didn't resolve the problem though.

[godzteeth]

Something I have noticed: /etc/vmware/hostd/datastores.xml seems to be missing.

At /, find . -name "datastores" returns no results.

Can I create this file manually? If so, what should it contain?

[sat1979hya]

It's not required to have datastore.xml in /etc/vmware/hostd/ folder. You may try creating additional user and try to login with the new user id created.