could not reach isolation address

[Ken_Mc]

So, I just upgraded my vCentre server to 5 and now all my 4.1 ESX servers are responding with:

vSphere HA agent on this host could not reach isolation address: 192.168.0.1

Any idea why this message occurs now, and not before the upgrade?  How do I turn off this message?

[a_p_]

Just a quick question. Is this the correct isolation address (usually the management network's gateway address)?

André

[Ken_Mc]

Yes.

[a_p_]

Are you able to ping this address from the ESXi hosts?

André

[Ken_Mc]

I think so.  It's our main firewall supporting 1000 users.  I can ping from all VM's, but I havn't tried from the host.  Why would that change with the upgrade, did the firewall settings change?

[iw123]

Have you tried 'reconfiguring for HA' on the hosts? It maybe that the HA agent is playing up for some reason following the upgrade. Can you confirm whether you can ping the gateway address from one/all of your hosts.

Regards.

[Ken_Mc]

Yes, I've tried the reconfiguration.  Yes, I can ping from all the hosts.

I've since opened a ticket with VMware, and so far they have been unable to resolve my issue.

[admin]

And there are no host config issues complaining of lack of access to other hosts' IP addresses?  How many hosts in the cluster?  Any other hosts have the same issue?

Each HA agent (fdm) slave performs periodic pings of each other HA agent slave every 5 minutes by default.  It also pings the default gateway for the management interfaces every 5 minutes.  Those pings are initiated through the physical adapter associated with the management vSwitch, so it is possible there is no route between the management vSwitch's adapter and the default gateway, even though you can initiate a successful ICMP ping from the host itself, since it may be using a different vSwitch's physical adapter which does have a route to the default gateway.  Please take a look at that first by examining the host's configuration Networking.

If that fails to yeild an answer, can you check the file /var/run/log/fdm.log (or one of its gzipped backups if you reconfigured a while ago) or your host's syslog if redirected to see if there are any errors such as "Failed to create ipv[4|6] socket", or see if there are any other errors or warnings.  search for "ClusterPing" in these log files.  Turning on "trivia" level logging may be better for diagnosing this.  to do that, set the HA cluster advanced option das.config.log.level to trivia.  The reconfigure HA on that host, wait for the config issue, then examine the FDM log on that host.

Let us know if this is of help.

[admin]

Can you supply us with the SR number so we (in engineering) can monitor and take a look at the logs?

[Ken_Mc]

Well I only have one subnet, no routing on my network.  Our firewall is getting spammed by all the hosts actually - reported to be one ping per second per host.

I've just turned on that logging option and sent you my SR number.

[Ken_Mc]

I have 2 hosts in one cluster and 3 hosts in another cluster.  All hosts are complaining - "could not reach...".  This only happened after the upgrade of vCentre server.  I have since upgraded all the hosts from 4.1 to 5.0 and no change.

[admin]

Once a host cannot ping its isolation address, the frequency of pings rises to every second for 5 seconds until the address becomes pingable again.  This is to remove the configuration issue as quickly after recovery as possible.

[NYSteve]

Hi,

     All of our hosts are receiving the same error after having been updated to vSphere 5.  Our vMotion is on a separate VLAN/subnet since staying on the same subnet as the Maintenance Network defeats the network vmnic assignments for Virtual Standard Switches (thanks, VMware), and forces VMotions over the Management Network no matter how you've configured it.  That being the case, is there a workaround for this problem?  As it stands now, I am unable to ping the gateway for the VMotion network from any of our upgraded hosts, and not for any network related issue.

     Thanks,

          NYSteve

[Slymsoft]

I have the exact same issue. Did you find a way to make it work ?

[rickardnobel]

Slymsoft wrote:

I have the exact same issue. Did you find a way to make it work ?

Could you describe your setup? If possible, a screenshot of the networking view on a host reporting this.

[Slymsoft]

I found the issue : it looks like vSphere HA is pinging using the maximum size of packet (1472 for MTU 1500) and this was blocked on our network equipment. Plus, there was a rule on the maximum number of ping allowed in a time frame.

I hope it helps.

[rickardnobel]

Slymsoft wrote:

I found the issue : it looks like vSphere HA is pinging using the maximum size of packet (1472 for MTU 1500) and this was blocked on our network equipment.

How could default packet sizes be blocked? Sounds like a difficult setup to maintain. Nice to hear that the problem was solved anyway!

[Slymsoft]

only ICMP requests with maximum size were blocked.

[Ken_Mc]

I had fixed this last year by chaning the IP to something that can be pinged.  That was the solution tech support gave me at the time, although I don't know if anythings changed.

[eXpat_SE]

My Solution

In my case my default gateway is my firewall which has ping disabled by default. I created a firewall rule that allowed the ip adresses of my hosts to ping the firewall. Instant success.