topic Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster" in VMware vSAN Discussions

Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Thu, 20 Nov 2014 17:32:55 GMT

Hi community,

we have a problem (?).

We have a VSAN enabled cluster with four hosts. Everything looks fine,

- the configuration is good,

- the VSAN status page displays "Network status: (green arrow) Normal",

- the Disk Management page displays "Status: Healthy" for all of our disk groups.

- Even a "esxcli vsan cluster get" on every host returns a "HEALTHY"

But we have a littly yellow exclamation mark on every host "Host cannot communicate with all other nodes in the VSAN enabeld cluster".

Anyone with the same problem? Anyone with an idea or just a hint?

Thank you!

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

larstr — Thu, 20 Nov 2014 18:30:22 GMT

Hi,

This is a physical network switch configuration problem. Make sure you have enabled igmp snooping on your vSAN vlan.

Lars

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

CHogan — Fri, 21 Nov 2014 13:37:08 GMT

Its not simply an alarm left over from when you were configuring the cluster?

What I mean, is that was the VSAN cluster formed before the network was configured? This might explain why all the hosts have this alarm.

Now you have resolved the network issue, but the alarm still needs to be acknowledge.

If that is the case, then all you need to do is acknowledge the alarms and reset them to green.

HTH

Cormac

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Mon, 24 Nov 2014 17:02:52 GMT

Thank you for your reply.

@larstr

IGMP snooping is disabled on the vsan vlan because on this vlan is only vsan traffic.

@CHogan

Unfortunately not that easy.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

ezequielcarson — Mon, 24 Nov 2014 17:14:26 GMT

Do you have all servers in the cluster with a VMK tagged with VSAN traffic?

Can you ping from all servers to all server in the cluster?

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Mon, 24 Nov 2014 17:25:57 GMT

Every host has one vmk with vsan traffic and every host can ping all other hosts in the cluster via the vsan network.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

ezequielcarson — Mon, 24 Nov 2014 17:44:26 GMT

Have you tried to reapply vsan configuration from RVC?

How many nodes do you see in the esxcli vsan cluster get?

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

larstr — Mon, 24 Nov 2014 19:23:30 GMT

Mrhaseins,

What type of switches do you use? IGMP is required for vSAN to work.

I saw the exact same from esxcli as you (everything reported ok), but error message in GUI until we got the switch config correct.

I wrote a bit about my experience on HP switches here: http://www.core-four.info/2014/11/vsan-and-hp-5400-switches.html

Lars

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

ezequielcarson — Mon, 24 Nov 2014 19:34:26 GMT

I have Vsan working with no igmp snopping enabled at all.

Do you refeer to igmp as a protocol or igmp snooping?

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

larstr — Mon, 24 Nov 2014 19:39:02 GMT

P er http://www.yellow-bricks.com/2014/03/31/vsan-misconfiguration-detected-2/ :

Here are your two options:

Enable IGMP Snooping for your VSAN network (VLAN) and define an IGMP Snooping Querier. Default setting on most Cisco switches is IGMP Snooping enabled but without an IGMP Snooping Querier. In this configuration VSAN will not be able to configure correctly!
Disable IGMP Snooping for your VSAN network (VLAN). Please note that you can typically disable IGMP Snooping globally and per VLAN, in this case if you want to disable it… disable it on your VLAN!

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Mon, 24 Nov 2014 21:14:26 GMT

We use Juniper EX4500 for our storage network. And of course only IGMP snooping is disabled on the vsan vlan.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

larstr — Mon, 24 Nov 2014 22:24:20 GMT

Mrhaseins,

Sound good. As vSAN communicates by using multicast you should then atleast make sure igmp is enabled. By using "show igmp group" on your vsan vlan you should two multicast groups.

Lars

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

ezequielcarson — Mon, 24 Nov 2014 22:34:26 GMT

Lars,

I dont have IGMP enabled on my EX4550 and the VSAN cluster is working fine.

ezequiel@COBOGWBP4500CLOUDx1> show igmp group

ezequiel@COBOGWBP4500CLOUDx1>

of course all ports are in access and the entire virtual chassis is dedicated for VSAN

ezeq.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Tue, 25 Nov 2014 11:04:53 GMT

Updating vcenter to the latest build and the errors disappears. Problem solved! The "old" vcenter version was from September 2014, strange.

Thank you for your help!

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

rom3010 — Fri, 05 Dec 2014 12:11:45 GMT

Hi Mrhaseins,

Did you remember which version were you running and what you have now?

I'm experiencing the same problem.

C:\Windows\system32>"C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe" -v

VMware VirtualCenter 5.5.0 build-2001466

Control Panel > Program and Features

VMware vCenter Server 5.5.0.42389

The installation was updated the last time with "VMware-VIMSetup-all-5.5.0-2105955-20140901-update02.iso".

Thanks!

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

Mrhaseins — Fri, 05 Dec 2014 13:11:02 GMT

I upgraded our vcenter last time in September 2014 so the "old" version could be the "vCenter Server 5.5 Update 1c for Windows | 22 JUL 2014 | Build 1945274". Now we are on the latest version "vCenter Server 5.5 Update 2b | 09 OCT 2014 | Build 2183111". But I think the reconnect of the hosts after upgrading der vcenter solved the problem not just the upgrade.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

rom3010 — Fri, 05 Dec 2014 13:20:51 GMT

Thanks for your fast reply.

I think then in my case it's not related to the version.

The problem started to appear after changed the Management vmkernel IP address of the vSAN hosts members.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

sureshsiwach — Sat, 04 Jul 2015 10:53:21 GMT

Hi All,

I have seen the same issue in my environment while I was configuring VSAN at new site. I have worked for two days to resolving this issue. Now i can advise you how to troubleshoot it.

You can follow the below point. Its up to you if you want to follow step by step or any individual which suits to your infrastructure.

1. Restart management services.

2. Restart ESXI host.

3. Login on ESXI and try to ping the IPs which you are using for handle VSAN traffic.

4. Recreation of virtual adapter, use for VSAN traffic.

5. Recreation of Vlans(Port Group)use for VSAN.

6. Recreation of DVswitch use for VSAN.

7. Enable\disable VSAN,

8. Recreation and deletion of disk group.

If you are following step 3 and you will be successful to ping it. Then approach to your network Team\person to make changes on physical switch. The ports used for Virtual SAN have multicast enabled. Enabling multicast can be done in one of two way on your physical switches:

• Disabling IGMP snooping.
• Configure IGMP snooping for selective traffic.

I am pasting a link which can help you more in resolving above issue or other related to VSAN. https://www.vmware.com/files/pdf/products/vsan/VMware_Virtual_SAN_Quick_Monitoring_Reference_Guide.pdf

I was able to perform all step because my infrastructure was in implementation phase. Please consider you infrastructure and then follow the step carefully and at your own risk.

Thanks

Suresh Siwach

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

ChristianDickma — Mon, 06 Jul 2015 06:54:00 GMT

Just a quick note: A lot of the checking (ping tests, deducing multicast issues) are now part of the VSAN Health check. All 6.0 deployments should install the plugin. It makes life much easier.

Re: Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

pari2k3 — Sun, 13 Dec 2015 12:20:55 GMT

Recently, I have noticed the same issue and found I missed to select virtual san traffic in one of the host. The following link gives step by step guide to configure vSAN https://www.virtuallyboring.com/vmware-virtual-san-6-setup-and-configuration-part-2/