Error "Host cannot communicate with all other nodes in the VSAN enabeld cluster"

[Mrhaseins]

Hi community,

we have a problem (?).

We have a VSAN enabled cluster with four hosts. Everything looks fine,

- the configuration is good,

- the VSAN status page displays "Network status: (green arrow) Normal",

- the Disk Management page displays "Status: Healthy" for all of our disk groups.

- Even a "esxcli vsan cluster get" on every host returns a "HEALTHY"

But we have a littly yellow exclamation mark on every host "Host cannot communicate with all other nodes in the VSAN enabeld cluster".

Anyone with the same problem? Anyone with an idea or just a hint?

Thank you!

[larstr]

Hi,

This is a physical network switch configuration problem. Make sure you have enabled igmp snooping  on your vSAN vlan.

Lars

[CHogan]

Its not simply an alarm left over from when you were configuring the cluster?

What I mean, is that was the VSAN cluster formed before the network was configured? This might explain why all the hosts have this alarm.

Now you have resolved the network issue, but the alarm still needs to be acknowledge.

If that is the case, then all you need to do is acknowledge the alarms and reset them to green.

HTH

Cormac

[Mrhaseins]

Thank you for your reply.

@larstr

IGMP snooping is disabled on the vsan vlan because on this vlan is only vsan traffic.

@CHogan

Unfortunately not that easy. 😞

[ezequielcarson]

Do you have all servers in the cluster with a VMK tagged with VSAN traffic?

Can you ping from all servers to all server in the cluster?

[Mrhaseins]

Every host has one vmk with vsan traffic and every host can ping all other hosts in the cluster via the vsan network.

[ezequielcarson]

Have you tried to reapply vsan configuration from RVC?

How many nodes do you see in the esxcli vsan cluster get?

[larstr]

Mrhaseins,

What type of switches do you use? IGMP is required for vSAN to work.

I saw the exact same from esxcli as you (everything reported ok), but error message in GUI until we got the switch config correct.

I wrote a bit about my experience on HP switches here: http://www.core-four.info/2014/11/vsan-and-hp-5400-switches.html

Lars

[ezequielcarson]

I have Vsan working with no igmp snopping enabled at all.

Do you refeer to igmp as a protocol or igmp snooping?

[larstr]

P     er http://www.yellow-bricks.com/2014/03/31/vsan-misconfiguration-detected-2/     :

Here are your two options:

• Enable IGMP Snooping for your VSAN network (VLAN) and define an IGMP Snooping Querier. Default setting on most Cisco switches is IGMP Snooping enabled but without an IGMP Snooping Querier. In this configuration VSAN will not be able to configure correctly!
• Disable IGMP Snooping for your VSAN network (VLAN). Please note that you can typically disable IGMP Snooping globally and per VLAN, in this case if you want to disable it… disable it on your VLAN!

[Mrhaseins]

We use Juniper EX4500 for our storage network. And of course only IGMP snooping is disabled on the vsan vlan.

[larstr]

Mrhaseins,

Sound good. As vSAN communicates by using multicast you should then atleast make sure igmp is enabled. By using "show igmp group" on your vsan vlan you should  two multicast groups.

Lars

[ezequielcarson]

Lars,

I dont have IGMP enabled on my EX4550 and the VSAN cluster is working fine.

ezequiel@COBOGWBP4500CLOUDx1> show igmp group

ezequiel@COBOGWBP4500CLOUDx1>

of course all ports are in access and the entire virtual chassis is dedicated for VSAN

ezeq.

[Mrhaseins · Accepted solution]

Updating vcenter to the latest build and the errors disappears. Problem solved! The "old" vcenter version was from September 2014, strange.

Thank you for your help!

[rom3010]

Hi Mrhaseins,

Did you remember which version were you running and what you have now?

I'm experiencing the same problem.

C:\Windows\system32>"C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe" -v

VMware VirtualCenter 5.5.0 build-2001466

Control Panel > Program and Features

VMware vCenter Server 5.5.0.42389

The installation was updated the last time with "VMware-VIMSetup-all-5.5.0-2105955-20140901-update02.iso".

Thanks!

[Mrhaseins]

I upgraded our vcenter last time in September 2014 so the "old" version could be the "vCenter Server 5.5 Update 1c for Windows | 22 JUL 2014 | Build 1945274". Now we are on the latest version "vCenter Server 5.5 Update 2b | 09 OCT 2014 | Build 2183111". But I think the reconnect of the hosts after upgrading der vcenter solved the problem not just the upgrade.

[rom3010]

Thanks for your fast reply.

I think then in my case it's not related to the version.

The problem started to appear after changed the Management vmkernel IP address of the vSAN hosts members.

[sureshsiwach]

Hi All,

I have seen the same issue in my environment while I was configuring VSAN at new site. I have worked for two days to resolving this issue.  Now i can advise you how to troubleshoot it.

You can follow the below point. Its up to you if you want to follow step by step or any individual which suits to your infrastructure.

1. Restart management services.

2. Restart ESXI host.

3. Login on ESXI and try to ping the IPs which you are using for handle VSAN traffic.

4. Recreation of virtual adapter, use for VSAN traffic.

5. Recreation of Vlans(Port Group)use for VSAN.

6. Recreation of DVswitch use for VSAN.

7. Enable\disable VSAN,

8. Recreation and deletion of disk group.

If you are following step 3 and you will be successful to ping  it. Then approach to your network Team\person to make changes on physical switch. The ports used for Virtual SAN have multicast enabled. Enabling multicast can be done in one of two way on your physical switches:

• Disabling IGMP snooping.
• Configure IGMP snooping for selective traffic.

I am pasting a link which can help you more in resolving above issue or other related to VSAN.  https://www.vmware.com/files/pdf/products/vsan/VMware_Virtual_SAN_Quick_Monitoring_Reference_Guide.p...

I was able to perform all step because my infrastructure was in implementation phase. Please consider you infrastructure and then follow the step carefully and at your own risk.

Thanks

Suresh Siwach

[ChristianDickma]

Just a quick note: A lot of the checking (ping tests, deducing multicast issues) are now part of the VSAN Health check. All 6.0 deployments should install the plugin. It makes life much easier.

[pari2k3]

Recently, I have noticed the same issue and found I missed to select virtual san traffic in one of the host. The following link gives step by step guide to configure vSAN https://www.virtuallyboring.com/vmware-virtual-san-6-setup-and-configuration-part-2/