vSphere client could not connect to the vCenter server

[andrejnov123]

hi,

I take care for one ESXi 4.0.0-171294 (2009-06-22) box, that suddenly lost connectivity from vSpher client. I get: vSphere client could not connect to the vCenter server. Details: The server took too long to respond. (The operation has timed out).

I followed the

1. dns hostname resolvning is OK

2. ping OK

3. could not check because the commands are for ESX and do not work in ESXi

4. telnet 902 OK

5. the username is OK

6. we did not change any permissions

I also found the tread http://communities.vmware.com/thread/211725 but still cant solve it

any ideas how continue troubleshooting?

more info bellow:

list of storage space (it seems to be ok)

Filesystem Size Used Available Use% Mounted on

visorfs 215.9M 184.8M 31.1M 86% /

vmfs3 930.8G 502.6G 428.2G 54% /vmfs/volumes/4a853ad0-6b64a298-dd71-001517832738

vmfs3 274.0G 235.7G 38.3G 86% /vmfs/volumes/4a853a75-9fe462d1-d174-001517832738

nfs 1.8T 285.4G 1.5T 15% /vmfs/volumes/a595776e-6664f08e

nfs 1.8T 285.4G 1.5T 15% /vmfs/volumes/b2f833c4-32855697

nfs 1.8T 285.4G 1.5T 15% /vmfs/volumes/8259d688-dada9bf6

vfat 285.9M 230.2M 55.7M 81% /vmfs/volumes/c2a427e4-2d317086-fef9-b5750d88536c

vfat 249.7M 4.0k 249.7M 0% /vmfs/volumes/0f7cc67f-212a26d5-6da6-a6b5787df90c

vfat 249.7M 59.3M 190.4M 24% /vmfs/volumes/26653d79-219714bc-ad80-5f2413542f4b

vfat 4.0G 384.0k 4.0G 0% /vmfs/volumes/4a7adea5-e1b49bf8-f937-001517832739

list of CPU usage (hostd is constantly arround 100% - 1 core? - is that normal?)

ID GID NAME NWLD %USED %RUN %SYS %WAIT %RDY

1 1 idle 8 664.87 666.14 0.00 0.00 140.52

2 2 system 6 0.01 0.01 0.00 600.00 0.00

3 3 vim 1 0.00 0.00 0.00 100.00 0.00

6 6 helper 47 0.10 0.11 0.00 4700.00 0.01

7 7 drivers 9 0.01 0.01 0.00 900.00 0.00

8 8 vmotion 4 0.00 0.00 0.00 400.00 0.00

13 13 vmkapimod 13 0.00 0.00 0.00 1300.00 0.00

16 16 init.4181 1 0.00 0.00 0.00 100.00 0.00

167 167 busybox.4330 1 0.00 0.00 0.00 100.00 0.00

171 171 vmklogger.4334 1 0.00 0.00 0.00 100.00 0.00

631 631 FT 1 0.00 0.00 0.00 100.00 0.00

707 707 busybox.4888 1 0.00 0.00 0.00 100.00 0.00

747 747 ntpd.4927 1 0.00 0.01 0.00 100.00 0.00

763 763 sh.4942 1 0.00 0.00 0.00 100.00 0.00

773 773 hostd.4952 17 100.17 100.84 0.00 1613.25 0.00

785 785 sh.4964 1 0.00 0.00 0.00 100.00 0.00

795 795 vobd.4974 8 0.00 0.00 0.00 800.00 0.00

814 814 slpd.5002 1 0.00 0.00 0.00 100.00 0.00

876 876 sh.5064 1 0.00 0.00 0.00 100.00 0.00

886 886 openwsmand.5074 3 0.01 0.01 0.00 300.00 0.01

903 903 sh.5093 1 0.00 0.00 0.00 100.00 0.00

931 931 sh.5121 1 0.00 0.00 0.00 100.00 0.00

937 937 sh.5127 1 0.00 0.00 0.00 100.00 0.00

938 938 busybox.5128 1 0.00 0.00 0.00 100.00 0.00

947 947 net-cdp.5139 1 0.00 0.00 0.00 100.00 0.00

955 955 ash.5147 1 0.00 0.00 0.00 100.00 0.00

956 956 ash.5148 1 0.00 0.00 0.00 100.00 0.00

963 963 dcui.5155 1 0.00 0.00 0.00 100.00 0.00

964 964 ash.5156 1 0.00 0.00 0.00 100.00 0.00

968 968 busybox.5160 1 0.00 0.00 0.00 100.00 0.00

1079 1079 sfcbd.5272 1 0.00 0.00 0.00 100.00 0.00

1080 1080 cimslp.5273 1 0.00 0.00 0.00 100.00 0.00

1083 1083 sfcbd.5276 1 0.00 0.00 0.00 100.00 0.00

1084 1084 sfcbd.5277 1 0.00 0.00 0.00 100.00 0.00

1085 1085 sfcbd.5278 1 0.00 0.00 0.00 100.00 0.00

1129 1129 sfcbd.6036 4 0.00 0.00 0.00 400.00 0.00

1130 1130 sfcbd.6037 4 0.00 0.00 0.00 400.00 0.00

1502 1502 SBSERVER 4 5.36 5.61 0.04 397.49 0.16

1568 1568 ELROND 4 3.10 3.35 0.02 399.41 0.35

1640 1640 GIMLI 6 12.80 13.24 0.04 588.84 1.21

1777 1777 TERMINAL 4 9.51 9.95 0.05 393.03 0.28

47036 47036 sfcbd.80688 3 0.00 0.00 0.00 300.00 0.00

173186 173186 MGMT 4 2.97 3.33 0.00 399.72 0.28

349609 349609 SISTEM2 4 1.59 1.69 0.00 400.00 0.14

1476103 1476103 sfcbd.2400758 2 0.00 0.00 0.00 200.00 0.00

6479756 6479756 dropbearmulti.1 1 0.02 0.02 0.00 100.00 0.01

6479757 6479757 ash.10509465 1 0.00 0.00 0.00 100.00 0.00

6480125 6480125 esxtop.10509941 1 0.45 0.42 0.00 100.00 0.00

6482077 6482077 sh.10508407 1 0.00 0.00 0.00 100.00 0.00

list of proceses

WID CID World Name Command

4096 console

4097 idle1

4098 idle2

4099 idle3

4100 idle4

4101 idle5

4102 idle6

4103 idle7

4104 AsyncTimeout

4105 DeviceTaskmgmtWatchdog

4106 PathTaskmgmtWatchdog

4107 idle0

4108 helper0-0

4109 helper1-0

4110 helper1-1

4111 helper1-2

4112 helper2-0

4113 helper2-1

4114 helper2-2

4115 helper3-0

4116 helper4-0

4117 helper5-0

4118 helper6-0

4119 helper6-1

4120 helper6-2

4121 helper6-3

4122 helper7-0

4123 helper8-0

4124 helper9-0

4125 helper10-0

4126 helper11-0

4127 helper11-1

4128 helper11-2

4129 helper11-3

4130 helper12-0

4131 helper12-1

4132 helper12-2

4133 helper12-3

4134 helper13-0

4135 helper14-0

4136 helper15-0

4137 helper15-1

4138 helper15-2

4139 helper16-0

4152 helper16-13

4155 helper17-0

4163 OCFlush

4164 BCFlush

4165 reset-handler

4166 reset-watchdog

4167 memsched

4168 pshare-est

4169 vmscan

4170 lpage

4171 helper18-0

4179 PathTaskmgmtHandler

4180 DeviceTaskmgmtHandler

4181 4181 init /sbin/init

2379928 4952 hostd hostd

2384026 4952 hostd hostd

4291 helper19-0

4292 helper19-1

4293 helper19-2

4294 helper19-3

4295 helper19-4

4296 helper19-5

4297 helper19-6

4298 helper19-7

4320 usb

4330 4330 busybox syslogd

4334 4334 vmklogger vmklogger

8445 8445 vmx /bin/vmx

4350 NMP-failover

8447 vmm0:SBSERVER

8448 vmm1:SBSERVER

8449 8445 mks:SBSERVER /bin/vmx

8450 8445 vcpu-0:SBSERVER /bin/vmx

8452 8445 vcpu-1:SBSERVER /bin/vmx

8566 8566 vmx /bin/vmx

8567 vmm0:ELROND

8568 vmm1:ELROND

8569 8566 mks:ELROND /bin/vmx

8570 8566 vcpu-0:ELROND /bin/vmx

8581 8566 vcpu-1:ELROND /bin/vmx

4527 iscsi_trans_vmklink

4541 ata_piix

4542 ata_piix

4544 ahci

4545 ahci

4546 ahci

4547 ahci

4548 ahci

4549 ahci

4551 aacraid

4570 helper20-0

143844 4952 hostd hostd

2400758 2400758 sfcbd /sbin/sfcbd

2400759 2400758 sfcbd /sbin/sfcbd

8710 8710 vmx /bin/vmx

8711 vmm0:GIMLI

8712 vmm1:GIMLI

8713 vmm2:GIMLI

8714 vmm3:GIMLI

8721 8710 mks:GIMLI /bin/vmx

8722 8710 vcpu-0:GIMLI /bin/vmx

8724 8710 vcpu-1:GIMLI /bin/vmx

8725 8710 vcpu-2:GIMLI /bin/vmx

8726 8710 vcpu-3:GIMLI /bin/vmx

4640 DVSSync send world

4641 DVSSync ack world

4642 DVSSync recv world

4669 helper21-0

381540 4952 hostd hostd

4773 FS3ResMgr

4786 nfsRemountHandler

4787 nfsLockFileUpdate

4799 ipmi_si_drv

4807 helper22-0

4814 Accept

4821 vmotionServer

8933 8933 vmx /bin/vmx

8934 vmm0:TERMINAL

8935 vmm1:TERMINAL

8936 8933 mks:TERMINAL /bin/vmx

8937 8933 vcpu-0:TERMINAL /bin/vmx

8939 8933 vcpu-1:TERMINAL /bin/vmx

4888 4888 busybox inetd

4927 4927 ntpd /sbin/ntpd

4942 4942 sh /bin/sh

4952 4952 hostd hostd

4964 4964 sh /bin/sh

4974 4974 vobd vobd

4975 4974 vobd vobd

4976 4974 vobd vobd

4977 4974 vobd vobd

4978 4974 vobd vobd

4979 4974 vobd vobd

4980 4974 vobd vobd

4981 4974 vobd vobd

4982 4952 hostd hostd

4983 4952 hostd hostd

5002 5002 slpd /sbin/slpd

226201 4952 hostd hostd

9146 4952 hostd hostd

5064 5064 sh /bin/sh

5074 5074 openwsmand /sbin/openwsmand

5075 5074 openwsmand /sbin/openwsmand

5076 5074 openwsmand /sbin/openwsmand

5093 5093 sh /bin/sh

5121 5121 sh /bin/sh

5127 5127 sh /bin/sh

5128 5128 busybox crond

5139 5139 net-cdp /usr/sbin/net-cdp

5147 5147 ash /bin/ash

5148 5148 ash /bin/ash

5155 5155 dcui /sbin/dcui

5156 5156 ash /bin/ash

5160 5160 busybox getty

5162 4952 hostd hostd

5163 4952 hostd hostd

5164 4952 hostd hostd

5165 4952 hostd hostd

9745463 4952 hostd hostd

5245 4952 hostd hostd

5246 4952 hostd hostd

5272 5272 sfcbd /sbin/sfcbd

5273 5273 cimslp /bin/cimslp

5276 5276 sfcbd /sbin/sfcbd

5277 5277 sfcbd /sbin/sfcbd

5278 5278 sfcbd /sbin/sfcbd

10511775 helper18-3

6036 6036 sfcbd /sbin/sfcbd

6037 6037 sfcbd /sbin/sfcbd

6193 6036 sfcbd /sbin/sfcbd

6211 6037 sfcbd /sbin/sfcbd

10512504 helper18-2

6301 4952 hostd hostd

6318 6036 sfcbd /sbin/sfcbd

285135 285135 vmx /bin/vmx

285136 vmm0:MGMT

285137 vmm1:MGMT

285138 285135 mks:MGMT /bin/vmx

285139 285135 vcpu-0:MGMT /bin/vmx

285141 285135 vcpu-1:MGMT /bin/vmx

6756 6036 sfcbd /sbin/sfcbd

6805 6037 sfcbd /sbin/sfcbd

6808 6037 sfcbd /sbin/sfcbd

80688 80688 sfcbd /sbin/sfcbd

80692 80688 sfcbd /sbin/sfcbd

80693 80688 sfcbd /sbin/sfcbd

10513285 helper18-3

10509463 10509463 dropbearmulti dropbear

10509465 10509465 ash -ash

10513585 10513585 sh /bin/sh

10513586 10513586 ps ps

572725 572725 vmx /bin/vmx

572726 vmm0:SISTEM2

572727 vmm1:SISTEM2

572728 572725 mks:SISTEM2 /bin/vmx

572729 572725 vcpu-0:SISTEM2 /bin/vmx

572731 572725 vcpu-1:SISTEM2 /bin/vmx

[AntonVZhbankov]

Does ESXi reboot help you?

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[andrejnov123]

It is a production server and I have to wait till off hours. I hope that restart helps.

Andrej

[AntonVZhbankov · Accepted solution]

Try this:

1. Go to the ESXi console and press alt+F1

2. Type: unsupported

3. Enter the root password(No prompt, typing is blindly)

4. Restart the management service “/sbin/services.sh restart”

It won't affect VMs.

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[andrejnov123]

I will try your suggestions.

Andrej

[andrejnov123]

I did as you suggested:

~ # /sbin/services.sh restart

Running sfcbd-watchdog stop

sh: cannot kill pid 5093: No such process

Running wsman stop

Stopping openwsmand

Running slpd stop

Stopping slpd

Running vobd stop

watchdog-vobd: Terminating watchdog with PID 4964

Vobd stopped.

Running hostd stop

watchdog-hostd: Terminating watchdog with PID 4942

Running ntpd stop

Stopping ntpd

Running ntpd restart

Starting ntpd

Running hostd restart

mount: mounting visorfs on /var/lib/vmware/hostd/stats failed: File exists

Running vobd restart

Begin 'hostd ++min=0,swap,group=hostd /etc/vmware/hostd/config.xml', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000

Vobd started.

Running slpd restart

Starting slpd

Running wsman restart

Starting openwsmand

Running sfcbd-watchdog restart

vSphere client can now connect to the ESXi

Thanks for great help,

Andrej

[AntonVZhbankov]

If you cosider any comment as helpful please assign points

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[vmthunder]

Anton,

We are having the same issue. An ESXi reboot does help, but restarting the management agents either through the ESXi interface or the hidden console does not resolve the issue. We have rebuilt this machine once. Do you perhaps have any other suggestions?

[lshelton]

I realize that this is an old topic, so first, my apologies.

I am running 3 VMware 3.5 ESXi servers.  I was in the middle of configuring backups on them and suddenly lost connectivity from the backup server and the VI client to all of them at the same time.  None of the VM's were affected, I could ping and ssh to the VMware servers, and I ran /sbin/services.sh restart which fixed the problem.

The question I have is why would this happen on all the servers at the same time?

I also have a 4.1update1 machine and it was unaffected.

Thank you,

Lewis