ESXi hosts and vCenter intermittent disconnects

LukasLundell · ‎02-07-2012

We've been seeing ESXi hosts disconnect from vCenter for approximately 2-12 minutes. After this period, the host reconnects back to the vCetner server. This happens about 2-3 times per 24 hours. DNS and general networking seem fine. From vpxd's perspective, we see timeouts due to components not responding during host sync requests.

During this time period, we see some interesting log messages in vpxa.log:

We see 2-5 of the following messages. This message appears at the begining of the event, and is also interspursped throughout the event:

"2012-02-02T18:36:18.466Z [42931B90 verbose 'Default' opID=HB-host-41@10260-92e04dfd-ab] [VpxaMoService::GetChangesInt] Vpxa restarted or stolen by other server. Start a full sync

2012-02-02T18:40:48.518Z [429E8B90 verbose 'Default' opID=HB-host-41@10332-a9b72171-ce] [VpxaMoService::GetChangesInt] Vpxa restarted or stolen by other server. Start a full sync"

We also see these messages in Vpxa during the disconnect event:

2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:21.355Z [FFC78780 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:21.355Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:21.356Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:37:41.350Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:41.358Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:41:01.358Z [428CEB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:41:01.358Z [428CEB90 verbose 'Default'] [PollCurrentStats] Failed to gather host quickStats from provided host metrics.

2012-02-02T18:41:01.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:41:01.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results

2012-02-02T18:41:01.359Z [428CEB90 trivia 'Default'] [PollCurrentStats] Stats polling took [10] ms

2012-02-02T18:41:01.359Z [428CEB90 trivia 'VpxProfiler'] Ctr: TotalTime = 10 ms

and these messages in syslog:

2012-02-02T18:36:20Z Unknown: Capability, config.powerSystemInfo, config.cacheConfigurationInfo, hardwareInfo, networkInfo, resourceInfo, configStatus, licenseInfo, licensableResource, vmConfigOptionDesc, vmConfigOption, vmConfigTarget, localizationMgr]

So far, we have found no suspicious messages in the hostd or vmkernel logs.

Reaching out to see if anyone had anymore information on these messages and if anyone had seen this behaviour before.

Regards,

Lukas

Virtualinfra · ‎02-08-2012

Welcome to the community.

See if this ports are opened are not between ESXi and vCenter if there is a firewall between them.

443

902

903

Also double check the DNS, do nslookup fromesxi to vcenter and viceversa and Also check the DNS entries for the ESXi host.

Award points for the helpful and correct answer by clicking the below tab

Thanks & Regards Dharshan S VCP 4.0,VTSP 5.0, VCP 5.0

firestartah · ‎02-08-2012

Hi

It sounds like you're having problems with the hosts agents talking to vCenter.

Have you tried restarting the agents of the hosts?

Have you tried migrating all the vm's off one host and removing it from the inventory and re-adding it? This should cause the agent to reconnect and possibly fix the problem?

Did these errors just start all of a sudden or was it after a specific change?

Gregg

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful". Gregg http://thesaffageek.co.uk

Myretyr · ‎02-08-2012

I haven't seen the messages you have in your log. But I have had some problem with ESXi hosts losing connection to vCenter.

In my case the problem disappeared when I changed IP adress for the ESXi host. Somehing in my lab environment used the IP assigned to the ESXi host. Not the whole time, but at times spread around the clock. And then the connection between vCenter and the host was lost frome time to time.

Maybe it helps to check it!

Regards!

--==[ VCP 5 ]==-- --==[ VCP 4 ]==--

LukasLundell · ‎02-08-2012

No firewall or DNS issues 902 UDP is open and working fine. I am actually able to ping the vCenter server from ESXi during the issue, and stay connected to both the ESXi server via SSH and to the VMware Server via RDP as well.

The issue duration is 2-12 minutes, after which everything reconnects just find and stays connected until the next occurence of the issue (2-3 times per day).

In the instance today, one host started as disconnected (and stayed disconnected for 5 minutes), then 2 host (for another 2 minutes), and then everything came back online.

Regards,

Lukas

Virtualinfra · ‎02-08-2012

Is this happening to particular only one host or to all the host during that time.

I bet if it happens to all the host during that time in frequently, then this is a network issue or DNS issue.I had the same issue, may your also like that.

During the issue can you try to do Nslookup to esxi host name and see if there is a time out.

Also in DNS you will have 2 address primary and secondary.. during this issue try nslookup <servername> <primarydnsip> if its not reachable.. then work with network team and DNS team to fix this.

Award points for the helpful and correct answer by clicking the below tab

Thanks & Regards Dharshan S VCP 4.0,VTSP 5.0, VCP 5.0

LukasLundell · ‎02-09-2012

Checked DNS and it was fine. I am also seeing these warnings about "No auth data" and mutex locks:

2012-02-09T14:07:58.165Z [56E85B90 warning 'Default'] [VpxaAccessChecker] No auth data found for privileged operation:method=fetchQuickStats
2012-02-09T14:08:42.258Z [56F09B90 warning 'Default'] [VpxaAccessChecker] No auth data found for privileged operation:method=waitForUpdates
2012-02-09T14:08:42.258Z [56F4BB90 warning 'Default'] [VpxaAccessChecker] No auth data found for privileged operation:method=retrieveChanges
2012-02-09T14:08:42.419Z [56F2AB90 verbose 'Default' opID=HB-host-466@135-44903733-5e] [VpxaMoService::GetChangesInt] Vpxa restarted or stolen by other server. Start a full sync

vpxa.log:2012-01-07T17:54:18.080Z [4CEC8B90 warning 'Default' opID=HB-host-12186@11523-ef1b1127-4d] [ProcessAlarmSpec] GetTranslators -- vpxd to host translation is empty. Dropping results

vpxa.log:2012-01-07T17:54:18.080Z [4CEC8B90 warning 'VpxMutex' opID=HB-host-12186@11523-ef1b1127-4d] Mutex InvtLock locked for 1280 milliseconds(counterName: VpxMutex/Name='InvtLock')
vpxa.log:2012-01-07T17:54:23.379Z [FFBBC780 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results

LukasLundell · ‎02-09-2012

One other interesting thing... I believe there are a few types of "Specs" that VPXA needs in order to process data... an "AlarmSpec" an "QuickStatsSpec" and "ResourcePoolSpec", excetera.

I believe during this issue VPXA is having trouble getting this Spec... right when the issue is over, we see something like

'2012-02-02T18:41:42.186Z [42994B90 verbose 'VpxaHalResourcePool' opID=HB-host-41@10332-b540c0cb-48] Starting to process resource notifications - new spec sync from vpxa
2012-02-02T18:41:52.340Z [42A6DB90 verbose 'Default' opID=HB-host-41@10339-c387505e-22] [VpxaHalStatsHostagent::ProcessQuickStatsSpec] Received spec sync for QuickStats collection'

Once VPXA gets the QuickStats spec sync, then its stops the "Dropping Results" messages for quickstats.

Does anyone know where VPXA gets these specs from?

Regards,

Lukas

LukasLundell · ‎02-09-2012

Also, what could cause this "spec sync for QuickStats" to take a lot longer than it should take?

2012-02-10T07:21:59.824Z [296B4B90 verbose 'Default' opID=HB-host-43@10717-6fb5e05d-3a] [VpxaHalStatsHostagent::ProcessQuickStatsSpec] Received spec sync for QuickStats collection

Regards,

Lukas

LukasLundell · ‎02-14-2012

Anyone have more information on the "Translation" code or these "specs" that vpxa is using to report data up to vpxd?

jintoa · ‎02-15-2012

If you are still experiencing the issue, please collect vm-support from the Host and log bundle from the VC when the host is disconnected and upload to vmware FTP.

ftp instruction:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100852...

Signature: Disclaimer: My postings are my own and don’t necessarily represent VMware’s positions, strategies or opinions. If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanx!

LukasLundell · ‎02-15-2012

2012-01-21T11:20:32.673Z [29BCDB90 error 'PropertyCache' opID=WFU-f3f71687] Failed to diff 11:resourcePool, had ManagedObjectReference, got ManagedObjectReference

2012-01-21T11:20:32.673Z [29BCDB90 error 'PropertyCache' opID=WFU-f3f71687] Failed to diff 16:resourcePool, had ManagedObjectReference, got ManagedObjectReference

2012-01-21T11:20:32.681Z [29BACB90 error 'PropertyCache' opID=WFU-c873f14e] Failed to diff 10:resourcePool, had ManagedObjectReference, got ManagedObjectReference

2012-01-21T11:20:32.681Z [29BACB90 error 'PropertyCache' opID=WFU-c873f14e] Failed to diff 17:resourcePool, had ManagedObjectReference, got ManagedObjectReference

2012-01-21T11:20:32.692Z [29C73B90 error 'PropertyCache' opID=WFU-35f63be0] Failed to diff 8:resourcePool, had ManagedObjectReference, got ManagedObjectReference

2012-01-21T11:20:33.438Z [29C73B90 error 'PropertyCache' opID=WFU-35f63be0] Failed to diff 14:resourcePool, had ManagedObjectReference, got ManagedObjectReference

These errors also normally coincide with the end of an disconnection event.

dbthree · ‎02-20-2012

LukasLundell,

Did you get a resolution to this issue? I have two servers in their own cluster with the exact same problem. Thank you!

Dan C. Barber // VCAP // NCIE // CCNP-DC Data Center Solution Architect Presidio www.presidio.com

paulz201110141 · ‎02-24-2012

We have absolutely the same problem.

It occurs not every day, but 1 time every 2-3 days.

We have 3 ESX hosts... and it happens for all of them, generally at the same moment.

Any help will be very appreciated.

LukasLundell · ‎02-25-2012

Hello Paul and dbthree,

We do not have resolution yet. I have a few questions for you that will help us narrow down the issue:

Are you using VMware View or Xen Desktop and Active Directory-based authentication in your environments?

How long do the disconnects normally last?

Are you seeing the error messages I indicated for the duration of the disconnect?

How many times does it occure per day?

Do hosts automatically reconnect after the issue?

Regards,

Lukas

Girishkulkarni · ‎02-26-2012

Hello lukas,

What hardware you are used to install esxi ? have you upgraded that firmware and tried ?

Regards

Girish

Regards Girish Kulkarni ( If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful )

dbthree · ‎02-26-2012

Lukas,

We have solved out problem. Thanks. First of all, we have a remote vCenter offsite at another location. Though almost all of our vCenters are like that. It did automatically reconnect.

However, the issue turned out to be a misconfiguration of our Cisco trunks, which was causing the switches to alternate ports for connection. Once we corrected the switch trunks, the issue completely disappeared.

Dan C. Barber // VCAP // NCIE // CCNP-DC Data Center Solution Architect Presidio www.presidio.com

paulz201110141 · ‎02-27-2012

Hi, Lukas!

THanks for reply.

We do not have resolution yet. I have a few questions for you that will help us narrow down the issue:
Are you using VMware View or Xen Desktop and Active Directory-based authentication in your environments?

No. No VMware View, no Xen Desktop, no Active Directory-based authentication.

How long do the disconnects normally last?

This night there was 2 alarms.

The first starts about 2 AM : exactly at 2AM for one host and at 2.15 for two others.

It ends at 2.50...

So it was about 35 minuts (50 for host that was alarmed at 2AM).

The second alarm was at 8.06 AM for one host and 8.13 for two others.

And it took about 10 minutes ( 17 for the first host).

Are you seeing the error messages I indicated for the duration of the disconnect?

Yes. The messages are almost the same. I say "almost" because I don't verify all of your messages, but I verified a lot of and we have absolutely the same.

How many times does it occure per day?

It depends. Sometimes it happens one time per day, but for exemple today there were already two... and the day is not yet over.

Do hosts automatically reconnect after the issue?

Yes... even more...

I have not an impression that the host disconnect. I'm not sure, but it seems that it works almost well... but the alarms raised.

In any case, yes... they are reconnected automatically. We do nothing but after 10-30 minutes alarms passe from Red to Green.

paulz201110141 · ‎02-27-2012

We have solved out problem. Thanks. First of all, we have a remote vCenter offsite at another location. Though almost all of our vCenters are like that. It did automatically reconnect.
However, the issue turned out to be a misconfiguration of our Cisco trunks, which was causing the switches to alternate ports for connection. Once we corrected the switch trunks, the issue completely disappeared.

Hi, dbthree.

This is a great news...

For me it seems also that it could be network problems.

But... could you give a little bit more details, because I don't understand well that you have done.

Thank you very much.

Paul Zakharov

dbthree · ‎02-28-2012

We have upstream Cisco switches that were connecting to HP switches. ESX plugged directly into HP, and then we ran a port-channel out of each HP switch to the Cisco core. the problem i think was related to the fact that the HP switches were dropping one connection on the trunk, and then after a timeout were resuming on the other connection, almost like spanning tree, though that was not even turned on.

Once we reconfigured some settings on the Cisco (namely, turning off LACP, since it was off on the HP as well), then the problem went away.

Dan

Dan C. Barber // VCAP // NCIE // CCNP-DC Data Center Solution Architect Presidio www.presidio.com

All

ESXi hosts and vCenter intermittent disconnects