Error Connection, Connection Failed on connect to ESX host

[harrisandy]

Hi

I came in this morning and opened Virtual Center. I was unable to view anything in the console on any of the VM's hosted on this particular server, the screens were just black, couldn't try to RDP to them as they are all isolated from the main network. I rebooted the ESX box and i'm still unable to connect using Virtual Center or connecting direct to the ESX box using VC Client. I can ping the box OK.

I can connect via Putty fine but it does say the following after a reboot when i connect:

'Configuration changes were not svaed successfully during previous shutdown. Please file a problem report including output of vm-support'

The server is DL380 G4, 6GB RAM, using local storage. I have been using it to perform a migration from our existing environment (NT, Exchange 5.5, distributed architecture) to a new environment (Win2k3, Exchange 2003, Centralised and Virtualised) and to prove that VMWare is a viable option to use.

I'm a newbie to ESX and the whole Linux world so my troubleshooring skills are somewhat limited. I've searched the forum for answers but haven't come across any that fit my problem.

Has anybody got any ideas on how I can troubleshoot this?

Any help would be greatly appreciated.

Many Thanks

Julian

[Dave_Mishchenko]

Here are a couple of similar threads. The error may indicate that you've run out of space on one of the partitions. Are you able to connect to the server via SSH (you can use a tool called putty if you haven't done that before (http://www.chiark.greenend.org.uk/~sgtatham/putty/). Note that by default you won't be able to connect via SSH to the server with the root login as that is disabled. There is a work around for that if that is the case for you.

Once you have a session open, run the command vdf -h

It'll list out the partitions you have and will indicate if one of them is full.

http://www.vmware.com/community/thread.jspa?messageID=559640&#559640

http://www.vmware.com/community/thread.jspa?messageID=559640&#559640

[harrisandy]

Thanks for the fast response.

I have ran vdf -h via putty and got the following output:

\[root@stk-esx dev]# vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p3 4.9G 4.9G 0 100% /

/dev/cciss/c0d0p1 99M 29M 66M 31% /boot

none 132M 0 132M 0% /dev/shm

/dev/cciss/c0d0p6 2.0G 758M 1.1G 41% /var/log

/vmfs/devices 670G 0 670G 0% /vmfs/devices

/vmfs/volumes/45e450ff-421dad4f-963a-0013216b7241

123G 114G 9.3G 92% /vmfs/volumes/storage1

/vmfs/volumes/45e45199-fc5888a5-f7de-0013216b7241

410G 180G 229G 43% /vmfs/volumes/vmware_servers

I notice that c0d0p3 is 100% but I'm not sure how to clear this down as it looks like a file to me.

Cheers

[harrisandy]

OK - I have just checked the messages log and came acroos the following entry:

May 8 07:57:22 stk-esx esxcfg-boot: 2007-05-08 07:57:22 (31920) ERROR: Could not copy '/boot/initrd-2.4.21-37.0.2.ELvmnix.img' to '/tmp/vmware.0.tmp': No space left on device

May 8 07:57:22 stk-esx esxcfg-boot:

May 8 07:57:22 stk-esx esxcfg-boot: 2007-05-08 07:57:22 (31920) ERROR: Could not write out new initrds.

May 8 07:57:22 stk-esx esxcfg-boot: 2007-05-08 07:57:22 (31920) ERROR: /usr/sbin/esxcfg-boot failed, examine /var/log/vmware/esxcfg-boot.log for details.

May 8 07:57:22 stk-esx vmware: Saving VMware ESX Server configuration failed

May 8 07:57:22 stk-esx gpm: gpm shutdown succeeded

May 8 07:57:23 stk-esx sshd: sshd -TERM succeeded

May 8 07:57:23 stk-esx sshd\[927]: Received signal 15; terminating.

May 8 07:57:23 stk-esx snmpd\[918]: Received TERM or STOP signal... shutting down...

So it does seem to be space but I'm not sure what I can delete to create any more space?

Regards

Julian

[harrisandy]

I have managed to get rid off this error in the log file now by freeing up some addtional space on one of the volumes but i still can't log onto VI and i'm having the same problems. I get the following in the message log file now:

May 8 10:13:04 stk-esx hostd-support: Failed to create directory hostd-support-2413

May 8 10:13:04 stk-esx kernel: NET4: Linux TCP/IP 1.0 for NET4.0

May 8 10:13:04 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:04 stk-esx kernel: IP: routing cache hash table of 2048 buckets, 16Kbytes

May 8 10:13:04 stk-esx kernel: TCP: Hash tables configured (established 32768 bind 65536)

May 8 10:13:04 stk-esx VMware\[init]: Failed to write PID file /var/run/vmware/vmware-hostd.PID: No space left on device

May 8 10:13:04 stk-esx kernel: NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.

May 8 10:13:04 stk-esx watchdog-hostd: Executing cleanup command '/usr/sbin/hostd-support'

May 8 10:13:04 stk-esx VMware\[init]: + Segmentation fault (core dumped) setsid $CMD

May 8 10:13:04 stk-esx kernel: RAMDISK: Compressed image found at block 0

May 8 10:13:04 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:04 stk-esx sshd\[2978]: Accepted password for root from 10.8.1.51 port 1762 ssh2

May 8 10:13:05 stk-esx kernel: Freeing initrd memory: 5332k freed

May 8 10:13:05 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:05 stk-esx watchdog-hostd: Executing cleanup command '/usr/sbin/hostd-support'

May 8 10:13:05 stk-esx sshd(pam_unix)\[2978]: session opened for user root by (uid=0)

May 8 10:13:05 stk-esx kernel: VFS: Mounted root (ext2 filesystem).

May 8 10:13:05 stk-esx VMware\[init]: Failed to write PID file /var/run/vmware/vmware-hostd.PID: No space left on device

May 8 10:13:05 stk-esx kernel: Mod: 698: vmnix module init_module=0xd180f828 STACK_TOP_LA=0xfd402ff0

May 8 10:13:05 stk-esx watchdog-hostd: Executing cleanup command '/usr/sbin/hostd-support'

May 8 10:13:05 stk-esx VMware\[init]: + Segmentation fault (core dumped) setsid $CMD

May 8 10:13:05 stk-esx kernel: Mod: 3216: hb timer on.

May 8 10:13:05 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:05 stk-esx kernel: Mod: 3283: Started hb thread.

May 8 10:13:05 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:05 stk-esx kernel: IRQ: 291: COS is using IOAPIC

May 8 10:13:05 stk-esx watchdog-hostd: Executing cleanup command '/usr/sbin/hostd-support'

May 8 10:13:06 stk-esx VMware\[init]: Failed to write PID file /var/run/vmware/vmware-hostd.PID: No space left on device

May 8 10:13:06 stk-esx kernel: Mod: 2305: Setting 0xfce00000 to read-only

May 8 10:13:06 stk-esx VMware\[init]: + Segmentation fault (core dumped) setsid $CMD

May 8 10:13:06 stk-esx kernel: Mod: 2305: Setting 0xfd000000 to read-only

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: Mod: 2305: Setting 0xfd200000 to read-only

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: VMNIX: BSP APIC ID: 0x0

May 8 10:13:06 stk-esx watchdog-hostd: Executing cleanup command '/usr/sbin/hostd-support'

May 8 10:13:06 stk-esx VMware\[init]: Failed to write PID file /var/run/vmware/vmware-hostd.PID: No space left on device

May 8 10:13:06 stk-esx kernel: IRQ: 950: irq 0 is used

May 8 10:13:06 stk-esx VMware\[init]: + Segmentation fault (core dumped) setsid $CMD

May 8 10:13:06 stk-esx kernel: IRQ: 954: irq 0 is enabled

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: IRQ: 950: irq 1 is used

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: IRQ: 954: irq 1 is enabled

May 8 10:13:06 stk-esx VMware\[init]: Failed to write PID file /var/run/vmware/vmware-hostd.PID: No space left on device

May 8 10:13:06 stk-esx kernel: IRQ: 950: irq 2 is used

May 8 10:13:06 stk-esx VMware\[init]: + Segmentation fault (core dumped) setsid $CMD

May 8 10:13:06 stk-esx kernel: IRQ: 954: irq 2 is enabled

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: IRQ: 950: irq 14 is used

May 8 10:13:06 stk-esx VMware\[init]: connect: No such file or directory.

May 8 10:13:06 stk-esx kernel: IRQ: 954: irq 14 is enabled

I'm reallt at a loss as to how to troubleshoot this now.

Any hjelp would be great.

Cheers

Julian

[Dave_Mishchenko]

How much space does vdf -h show now. If you still have plenty of room, run these commands and then reconnect with the VI client. Are you running any 3rd party agents on the server? If the commands fail to help you may want to restart the host, but you'd want to make sure there is plenty of free space first on the root (/) drive.

service mgmt-vmware restart

service vmware-vpxa restart

[juchestyle]

In case anyone was wondering, (I was)

First one restart the service console

Second one restart the VC agent on the host.

service mgmt-vmware restart

service vmware-vpxa restart

Respectfully,