Rapid heartbeatStatus changes

[SunF1re]

We have a IBM BladeCenter H with HS21 blades in it runnin ESX4.0 U1. Linux VM's running without any issues. However on our syslog host we see this:

Hostd: \[2010-04-22 12:56:26.812 12CDCB90 verbose 'vm:/vmfs/volumes/4899871a-0df01e67-294d-001a645c9a06/labxsp1/labxsp1.vmx'\] Updating current heartbeatStatus: yellow

Hostd: \[2010-04-22 12:57:06.813 12CDCB90 verbose 'vm:/vmfs/volumes/4899871a-0df01e67-294d-001a645c9a06/labxsp1/labxsp1.vmx'\] Updating current heartbeatStatus: green

Hostd: \[2010-04-22 12:57:26.814 683FAB90 verbose 'vm:/vmfs/volumes/4899871a-0df01e67-294d-001a645c9a06/labxsp1/labxsp1.vmx'\] Updating current heartbeatStatus: yellow

Hostd: \[2010-04-22 12:58:06.815 12B58B90 verbose 'vm:/vmfs/volumes/4899871a-0df01e67-294d-001a645c9a06/labxsp1/labxsp1.vmx'\] Updating current heartbeatStatus: green

It keeps on cyceling like that forever. But the VM has the most recent VMware Tools installed and in the VM Summary is shows "Vmware Tools: OK".

Any idea why it is doing that and if we need to be worried?

Thanks

[golddiggie]

I would start by checking to see where the alarm was triggered. Look in your vSphere Client to see where it happened and any conditions surrounding the event. Also check the status of all hardware in the VMware environment, including connections to storage, hosts, etc...

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[SunF1re]

I tried to isolate the issue and used a plain vanilla HS21 XM blade server for testing. First of all I ran several memory tests and other hardware diagnostics. No hardware errors at all.

I installed ESX 4.0 U1 IBM image. Updated with host update utility. Then installed a Ubuntu 8.04.4 LTS as a guest. Installed VMWare Tools. Restarted VM.

So the entire installation is a plain as is can be but the log is full of weired messages incl. the heartbeat status changes:

Hostd: \[2010-04-26 09:45:14.827 38ED2B90 verbose 'Cimsvc'\] Ticket issued for CIMOM version 1.0, user root

Hostd: \[2010-04-26 09:45:21.379 38E50B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Updating current heartbeatStatus: yellow

Hostd: \[2010-04-26 09:45:24.733 38D8DB90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 294281216 bytes

Hostd: \[2010-04-26 09:45:24.734 38D8DB90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Hostd: \[2010-04-26 09:45:43.293 38D8DB90 verbose 'ResourcePool ha-root-pool'\] Root pool capacity changed from 17775MHz/28648MB to 17775MHz/28647MB

Hostd: \[2010-04-26 09:45:44.735 38E91B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 293228544 bytes

Hostd: \[2010-04-26 09:45:44.736 38E91B90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Hostd: \[2010-04-26 09:46:01.383 38ED2B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Updating current heartbeatStatus: green

Hostd: \[2010-04-26 09:46:04.737 38D8DB90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 293752832 bytes

Hostd: \[2010-04-26 09:46:04.738 38D8DB90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

vmkernel: 2:16:15:02.524 cpu3:763973)ScsiScan: 843: Path 'vmhba2:C0:T0:L0': Vendor: 'IBM ' Model: '1726-2xx FAStT ' Rev: '0617'

vmkernel: 2:16:15:02.524 cpu3:763973)ScsiScan: 846: Path 'vmhba2:C0:T0:L0': Type: 0x1f, ANSI rev: 5, TPGS: 0 (none)

vmkernel: 2:16:15:02.524 cpu3:763973)ScsiScan: 105: Path 'vmhba2:C0:T0:L0': Peripheral qualifier 0x3 not supported

Hostd: \[2010-04-26 09:46:21.383 38D8DB90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Updating current heartbeatStatus: yellow

Hostd: \[2010-04-26 09:46:24.739 38E91B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 294277120 bytes

Hostd: \[2010-04-26 09:46:24.739 38E91B90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Hostd: \[2010-04-26 09:46:24.742 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 name

Hostd: \[2010-04-26 09:46:24.742 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 triggeredAlarmState

Hostd: \[2010-04-26 09:46:24.752 38E91B90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 datastore

Hostd: \[2010-04-26 09:46:44.741 38DCEB90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 293220352 bytes

Hostd: \[2010-04-26 09:46:44.741 38DCEB90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Hostd: \[2010-04-26 09:46:44.744 38ED2B90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 name

Hostd: \[2010-04-26 09:46:44.744 38ED2B90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 parent

Hostd: \[2010-04-26 09:46:44.744 38ED2B90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 resourcePool

Hostd: \[2010-04-26 09:46:45.085 38E50B90 verbose 'Cimsvc'\] Ticket issued for CIMOM version 1.0, user root

Hostd: \[2010-04-26 09:47:01.385 38E50B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Updating current heartbeatStatus: green

Hostd: \[2010-04-26 09:47:01.385 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 config.defaultPowerOps

Hostd: \[2010-04-26 09:47:01.385 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 config.hardware.device

Hostd: \[2010-04-26 09:47:01.385 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 environmentBrowser

Hostd: \[2010-04-26 09:47:01.385 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 runtime.recordReplayState

Hostd: \[2010-04-26 09:47:01.388 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 network

Hostd: \[2010-04-26 09:47:01.388 38E0FB90 verbose 'PropertyJournal'\] ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow 16 config.extraConfig

Hostd: \[2010-04-26 09:47:04.743 38E0FB90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 293744640 bytes

Hostd: \[2010-04-26 09:47:04.743 38E0FB90 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Hostd: \[2010-04-26 09:47:21.386 38E50B90 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Updating current heartbeatStatus: yellow

Hostd: \[2010-04-26 09:47:24.745 38BCCDC0 verbose 'vm:/vmfs/volumes/4ab1fd57-018b1972-fc32-00215e249b08/Testdummy/Testdummy.vmx'\] Actual VM overhead: 293212160 bytes

Hostd: \[2010-04-26 09:47:24.745 38BCCDC0 verbose 'Vmsvc'\] RefreshVms updated overhead for 1 VM

Any ideas what is going wrong?

[bulletp31]

What Hardware version is the VM?

and have you enabled Paravirtualization? maybe worth checking to see if it has enabld itself..... seen this once or twice with Linux based VMs

[SunF1re]

ESXi 4.0 Version: 244038

VMware Tools: 4.0.0 build-236512 for Linux

VM Hardware Version: 7

Guest OS: Ubuntu Linux (64-bit)

Paravirtualization: disabled

[bulletp31]

thats my idea out the windows then............. I have been seeing strage errors with some Windows servers and this was due to the VM hardware being at Version 4

[HannaL · Accepted solution]

There is a kb that matches your symptoms:

vCenter server alarm for Virtual Machine heartbeats: The heartbeat of guest keeps changing between yellow and green

http://kb.vmware.com/kb/1017091

Hanna

[SunF1re]

Thank you so much Hanna, that did it! :smileygrin: No more status changing.

I wonder why this KB article didn't come up when I was running a search....

Thanks again!