Unexpected vib installation on hosts

[a_p_]

Today I deployed vSphere Replication 5.8 on a vSphere 5.5 U1 cluster (single site installation). Installation and configuration went fine, and I was able to configure replications as expected. However, shortly after recovering a replicated VM, an installation task - initiated by "com.vmware.vcHms" - started on all ESXi hosts. After this, all host's had event entries like "Installation started" and "... 0 vibs installed, 0 vibs removed ...", and all hosts are flagged with "Reboot required" now. I didn't find anything related to this in the documentation, the Release Notes nor online.

Did someone else see this too, or have an explanation for this behavior? I didn't open a support request yet, but may do this later.

André

[mvalkanov]

Hi,

VR 5.6 (available to subscribers of vCHS DR service), VR 5.8 (publicly available) and next VR releases, do deploy a VIB package with firewall rules to allow outgoing traffic from source ESXi to the VR appliance at ports 10000-10010 for tunneling replication traffic to the cloud site.

This is done on each host in the vCenter inventory, even before replication is configured, to allow vMotion of the replicated VM to another source ESXi.

The 'reboot required' is not intended behavior.

Please file an SR and if possible post the SR number here.

Regards,

Martin

[a_p_]

Thanks for the quick feedback. Tomorrow, I will be at the customer's site again and will try to collect the log files.

André

[tonyjwood]

Hi

The same has happened to me.  I see a reboot required on a lot, but not all of the hosts in the site.

There are also a lot of repeat VIB install events: Successfully installed 0 VIBS(s), removed 0 VIBS(s)

I thought I would see if this was resolved by you before I open an SR.

Tony

[a_p_]

You remind me of something...

I didn't open an SR yet, but I looked at the log files and found some entries that I'm not sure how to interpret. Anyway - out of curiosity - how did you install the ESXi hosts? In my case these hosts were all reinstalled using the "VMware-VMvisor-Installer-5.5.0.update01-1746018.x86_64-Dell_Customized-A00.iso" and then patched using Update Manager.

André

[tonyjwood]

These were installed with the Customised HP Image - upgraded from 5.1 via Update Manager. The install task occurs a lot and also after a host is rebooted.  I will raise an SR with VMware to find out why this keeps occurring.

[crosdorff]

Has anyone found a solution for this problem yet?

I see the install task on every host hundreds off times per day. It started after installing vsphere replication 5.8.

[tonyjwood]

I have an SR open with VMWare so will hopefully have an update soon.  Perhaps you could raise one too if possible as I don't think they have seen many occurrences of this problem.

[vNEX]

perhaps manual installation of the firewall VIB could stop this from occuring  ...:/

VMware KB: Replications to cloud remain in Not Active status

[crosdorff]

The package you mentioned is installed perfect:

# esxcli software vib list | grep vr2c

vr2c-firewall                  5.8.0.1-2170514                        VMware           VMwareCertified   2014-11-11

[vNEX]

have you tried to uninstall vr2 VIB from the host delete all the VIBs (or just vr2 VIB) from the VUM patch repository redownload all the VIBs/VIB and initiate stage and remediation process again?

[crosdorff]

I've removed the vib's from all servers:

~ # esxcli software vib remove -n vr2c-firewall

Removal Result

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed: VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

   VIBs Skipped:

But as soon as a new task runs (for example creating or removing a snapshot) it's installed again.

And again, and again, and again.

2015-02-10T13:44:32Z esxupdate: downloader: DEBUG: Downloading from https://10.1.137.31:8043/vib/vr2c-firewall.vib...

2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Skipping installed VIBs VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

Actions for the last few hours:

~ # grep "Final list of VIBs being installed" /var/log/esxupdate.log

2015-02-10T11:23:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:24:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:25:49Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:52:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:53:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:54:51Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:55:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T11:56:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T12:01:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T12:40:47Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T13:12:31Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T13:13:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T13:40:42Z esxupdate: Transaction: INFO: Final list of VIBs being installed: VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

2015-02-10T13:45:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

All the install tasks are obfuscating 'normal' tasks in the task history.

[mvalkanov]

Hi,

Do you know what changes are done at ESXi configuration each and every minute? VRMS will only attempt to re-install the VIB, if it detects the ESXi coming out of maintenance mode, or network configuration change or something similar.

Please open SR and mention that this issue might be related to SR 15583212901. Engineering team is already looking into the issue with unnecessary VIB install attempt, if the VIB is already present.

Regards,

Martin

[crosdorff]

I've created a Service Request with number 15602397302

[Dipak]

I have the SR with VMware support for more than a month now, yet to get anything concrete. In my case we get the VIB install messages and immediately the hosts will get disconnected and reconnected automatically. I have around 150+ hosts including remote site hosts. All remote site hosts are ESXi 5.0 hosts and local site hosts (around 50+) are ESXi 5.5 Build 2302651. It happens only between 1:00 am and 2:00 am) every night.

This vCenter is part of a SRM site and the VIB install is initiated by VRMS server. Last week I shut down the VRMS form 12:30 am to 2:30 and didn't have any host disconnects at all. I passed the info to VMware support waiting on their feedback. Shall post once i hear anything from them.

I have 3 other vCenters with SRM but not seeing this behavior on them.

[hyperbart]

Hi guys

+1 from a Belgian engineer here.

I've got the same symptoms: Dell PowerEdge's installed with the 5.1 Dell customized image and the install events tasks keep piling up...

Rebooting the hosts now to see if that changes anything.

I must say this is an "inherited" environment with absolutely no patching done in the last years...

Environment:

vCenter 5.1 build 880146

Cluster with

ESXi 5.1.0 build 799733

ESXi 5.1.0 build 799733

Stand alone host

ESXi 5.1.0 build 799733

In the meantime I already logged a SR: 15621359303

[tonyjwood]

Hi

I was informed by VMware this is a bug and that it would be fixed in the new release.  I've not installed 6.0 yet so cannot confirm if the problem goes away.

Tony

[crosdorff]

VMware vSphere Replication 6.0 is compatible with VMware vCenter Server 5.5 / 5.5U1 / 5.5U2 according to the VMware Product Interoperability Matrixes

So it's worth a try.

[tonyjwood]

Do you know version 6 is compatible with Site Recovery Manager 5.8?  It wasn't clear on the matrix.  I would upgrade to SRM 6.0 but it requires vSphere 6.

[mvalkanov]

Hi Tony,

VRMS attempts to re-deploy its VIB at certain configuration/state changes at ESXi. This might result in unnecessary download and log messages.

VRMS does not first attempt to check the metadata at ESXi before attempting a (re-)deploy. This is not fixed in VR 6.0, but fix is expected for a 5.8.x and 6.0.x maintenance release.

Regards,

Martin