VMware Cloud Community
caydenlu00
Contributor
Contributor

Use different vmkernel number in vSAN cluster

Experts,

Can I use different vmkernel number in vSan cluster?

For example vmk0 for vsan in host1 while vmk2 for vsan in host2. Will cause any problems?

Reply
0 Kudos
5 Replies
TheBobkin
Champion
Champion

Hello caydenlu00​,

Welcome (back) to Communities.

Yes, you can do this and it shouldn't cause any issues, that being said we do overall advise consistency be maintained to avoid confusion/mistakes (e.g. someone running a script against all hosts against vmkX assuming they are consistent).

Bob

Reply
0 Kudos
caydenlu00
Contributor
Contributor

Hi TheBobkin,

Thanks for your reply.

IHAC has the same configuration, and there is a warning in one ESXi host's monitoring: "Host cannot communicate with one or more other nodes in the vSAN enabled cluster".

The customer suspected that the vmkernel numbering is the root cause of this warning.

Have some design guide or KB references to this part? Thanks.

Reply
0 Kudos
TheBobkin
Champion
Champion

Are you sure they don't just have it misconfigured?

e.g. Let's say we have 4 nodes and all of them can communicate over vmk0 but not over vmk2, if you have 3 configured with vSAN on vmk0 they will communicate with one another but if one has vmk2 configured for this it will be isolated from the cluster.

I would advise starting with basic network troubleshooting here e.g. is the cluster fully-formed, which vmk there is route between and at what MTU (and compared to what is configured), whether the unicastagent lists on the nodes state vmk in them (they don't need to but this can be specified and if specified it will use this only), which vmk is configured for vsan/witness traffic and/or whether multiple are tagged for this, whether they are on a VLAN or not etc. .

To further clarify on my last comment - you mentioned 'vmk0' this is by default (unless changed) used for Management traffic - it is not supported to multihome traffic types on the same vmk or subnet (e.g. Management, vMotion, vSAN should all be in their own subnet and on their own vmk):

VMware Knowledge Base

Bob

Reply
0 Kudos
caydenlu00
Contributor
Contributor

Customer has 5 nodes vSAN cluster.

All nodes have 4 vmkernels:

Host 1-5

vmk0-MGMT

vmk1-vMotion

Host1-3

vmk2-vSAN    #vlan tagged, MTU9000, no unicast

vmk3-iSCSI

Host4-5

vmk2-iSCSI

vmk3-vSAN # #vlan tagged, MTU9000, no unicast

Now the vSAN cluster running well and everything ok, but have one host with the warning.

Reply
0 Kudos
TheBobkin
Champion
Champion

Thanks for clarifying the configuration - bearing this in mind, if mixed vmk number was the problem then you would have either 2 or 3 hosts triggering the Health alert, I have seen configurations like this a ton of times and never seen it cause issues.

I would advise checking the vsanmgmtd and vmkernel logs of the host triggering the alert - maybe it has some other issue or vsanmgmtd is hitting memory pressure etc. .

Bob

Reply
0 Kudos