Strange micro disconnections to vcenter via vSphere Infrastructure Client

[Satchelmouth]

Forgive me if this is in the wrong category, however it seems relevant as the issue involves vCenter to some extent.

I had a customer report an issue ( which I was subsequently able to reproduce) where they experience a very very short disconnection alert in the Infrastructure Client. The VI Client brings up the typical attempting reconnection to vcenter lsot window for a fraction of a second before it reconnects without issue. While it's a somewhat trivial issue it means that any open console windows lose focus / are moved to the background and perhaps is a indicator of larger issues.

Looking at the VI Client logs, I can see the following ( log file shows all logs between two seperate disconnection events with disconnection event itself bolded )

-

---

viclient:Critical] 2010-09-16 10:27:27.388 Plugin caused a cancel of the WFU loop, ignoring...

2010-09-16 10:27:27.404 OnServiceEvent ConnectionDropped for https://vcenter.X.X.X/sdk : No detail

2010-09-16 10:27:27.404 OnConnectionDropped:GetNumberOfConnectedServices = 1

2010-09-16 10:27:27.404 OnConnectionDropped:ShowNotConnectedWarning...

2010-09-16 10:27:27.419 ConnectionManagement.ShowNotConnectedWarning: Null Text

2010-09-16 10:27:27.419 (VIClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null) PropView -> VpxClient.GroupPropView

2010-09-16 10:27:27.419 Setting up VpxClient.GroupPropView for Folder:group-d1 http://vcenter.X.X.X

2010-09-16 10:27:27.419 Replacing old blank view by VpxClient.GroupPropView

2010-09-16 10:27:27.466 WaitForUpdates Version = 24

2010-09-16 10:27:27.466 Invoke 89 Start WaitForUpdates on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.419 PollerDispatcher for server: https://vcenter.X.X.X/sdk enabled

2010-09-16 10:27:27.466 Invoke 85 Finish Destroy on PropertyFilter:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]A883742E-F2EB-4DA3-8E8E-2BF31CF4F5DD http://vcenter.X.X.X - Serial:0.001, Server:000.050

2010-09-16 10:27:27.482 Invoke 90 Start Destroy on PropertyFilter:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]04A35C46-BF66-42EF-A6E1-AF24AE119A5B http://vcenter.X.X.X.

2010-09-16 10:27:27.482 Invoke 91 Start CreateFilter on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.498 Form1.InitMessageOfTheDayListener: filter created, service = https://vcenter.X.X.X/sdk

2010-09-16 10:27:27.498 (VIClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null) Toolbar -> VpxClient.Toolbars.DatacenterFolderToolbar

2010-09-16 10:27:27.529 Invoke 87 Finish Destroy on PropertyFilter:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]B067D01D-110F-4956-AECD-DE790928711F http://vcenter.X.X.X - Serial:0.000, Server:000.162

2010-09-16 10:27:27.529 Invoke 90 Finish Destroy on PropertyFilter:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]04A35C46-BF66-42EF-A6E1-AF24AE119A5B http://vcenter.X.X.X - Serial:0.000, Server:000.048

2010-09-16 10:27:27.529 Invoke 88 Finish CreateFilter on PropertyCollector:propertyCollector http://vcenter.X.X.X - Serial:0.003, Server:000.160 [session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]C499A032-32BB-40B7-A250-B5A6585BFA5A]

2010-09-16 10:27:27.576 (VIClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null) PropView -> VpxClient.VMPropView

2010-09-16 10:27:27.576 Setting up VpxClient.VMPropView for VirtualMachine:vm-23495 http://vcenter.X.X.X

2010-09-16 10:27:27.576 Replacing VpxClient.GroupPropView by VpxClient.VMPropView

2010-09-16 10:27:27.638 (VIClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null) Toolbar -> VpxClient.Toolbars.VirtualMachineToolbar

2010-09-16 10:27:27.654 secondary cache: retrieving a node for as requested by VpxClient.Common.Util.Helper.IsVMwareServerHost

2010-09-16 10:27:27.654 Invoke 92 Start RetrieveContents on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.685 Invoke 92 Finish RetrieveContents on PropertyCollector:propertyCollector http://vcenter.X.X.X - Serial:0.007, Server:000.022

2010-09-16 10:27:27.685 Invoke 93 Start Modify on ListView:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]09771C2A-BFA2-4AB8-9EF6-50B2AFBBBFD9 http://vcenter.X.X.X.

2010-09-16 10:27:27.701 Invoke 94 Start CreateFilter on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.779 Invoke 95 Start Destroy on PropertyFilter:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]3220BB4D-05B9-40CB-B8E9-A9C0FE0F591E http://vcenter.X.X.X.

2010-09-16 10:27:27.779 Invoke 96 Start CreateFilter on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.794 Invoke 93 Finish Modify on ListView:session[DFA9A785-E9FF-45EA-894B-E4E85B4C7C54]09771C2A-BFA2-4AB8-9EF6-50B2AFBBBFD9 http://vcenter.X.X.X - Serial:0.000, Server:000.108

2010-09-16 10:27:27.810 Invoke 97 Start CreateFilter on PropertyCollector:propertyCollector http://vcenter.X.X.X.

2010-09-16 10:27:27.841 Unable to add item.0 already added.

2010-09-16 10:27:27.841 Unable to add item.1 already added.

2010-09-16 10:27:27.857 OnServiceEvent ConnectionRestored for https://vcenter.X.X.X/sdk : No detail

2010-09-16 10:27:27.857 ClearNotConnectedWarning

-

---

When these disconnects happen I am also monitoring ( admittedly its probably too high-level to be completely effective ) packet loss between the workstation with VIClient and the vcenter server and am not seeing any packet loss at all.

I have monintored vcenter's CPU and RAM usage during the connection loss events and resource usage on the VM hosting vcenter is low / at normal levels.

I can only replicate this issue when connecting via another ISP's connection. Connecting the VIClient from a machine in the datacenter ( 100mbit link ) does not show the same issue or via this ISP's DSL connections ( which made me think its possibly DNS related, but that seems like an all or nothing problem, either you dont connect to vcenter or you do? )

I am also trying to reconcile any vcenter vmotion events to client disconnections which may account for millisecond stuns of the VM but cannot match any of these up and so am at a loss to explain. Does anyone have any experience with this? Would be grateful for any tips on troubleshooting this one.

Below are the relevant technical details

Hosts - ESX 4.0 (non update 1 )

Host Hardware - BL460c

VI Client version - 4.0.0 build 162856

VI Client Host's OS - Windows XP SP3 & Windows Server 2003 SP3.

vCenter version - 4.0.0 build 162856

[weinstein5]

Welcome to the Forums - to me it sounds like a physical network issue because if you are in the data center it does not happen - you can 'hide' the problem by adjusting the network time out settings on the vSphere c lient -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[chadwickking]

The user I assume connects by a VPN? if not then I would suppose via IP or DNS from his vsphere client on his desktop. If they are using a laptop or something mobile they could always bring it and have them try it on the company's network to see if they have any issue. If they are using WiFi you could have them try using a physical connection. Just some basic things to look at

I know in our environement the VIC doesnt like to work with High Latency or as some would say "high ping times". As Weinstein said if its not happening on your network then I would say it could be pretty obvious that its a physical network issue.

You can set a larger remote command timeout going to Edit/Client Settings/Remote Command TimeOut/Use Custom Value.

Cheers,

Chad King

VCP-410 | Server+

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[hicksj]

How are you testing for packet loss currently? If you're running a simple ping, that may not be entirely effective as you mention. I would suggest running a linux box, and running "sudo ping -f ". A flood ping is much better at capturing short periods of packet loss.

Or, using Wireshark on the client, analyze the packet captures for lots of retransmissions, other tcp errors, or perhaps just periods of high latency.

As others have mentioned, changes to the timeout value may address the client disconnect issue, but it won't help figuring out the root cause.