vsan network adapter question

[jayharmer]

I need a little help guys...:)   BTW, this is on unsupported  hardware but shouldn't affect this scenario.

I have vsan enabled on three Dell servers. They each have a 4-port nic. I was hoping to have one physical port used for management and one for vmotion, and one for vsan.

It appears they are all attached to the first physical port. This physical port is untagged at the switch. The vsan ping tests work and connectivity is in  place.

vmk0 - mgmt-pg

vmk1 - vsan-pg

vmk2 - vmotion-pg

Two questions:

1) How do i make the vsan traffic use physical port 2 on the 4-port nic? I assume this is a good idea.

2) If  I plug the cable into the physical ports 2 or 3, then communication  instantly breaks until I remove the cable. Why does this happen?

Thank you,

Jay

[TheBobkin]

Hello Jay,

Do you see these Quad-link devices as 4 Physical uplinks at the host Networking level?

If so then the likely cause of plugging cables into port 2-4 is that these are not assigned to the vSwitch/vDistributed Switch (vDS) on the host or they are not configured for use on the Port Groups (PG).

You can check this and/or add uplinks to the vSwitch/vDS via the Web Client:

Networking > click the vDS > Manage/Configure > Topology > click the button that looks like a host+spanner (Add host Networking) then go through the Wizard to add host physical networking.

You can assign uplinks to different uses for each PG:

Via Web Client on a vDS:

Networking > Click the vDS > Click Distributed Port Group > Edit Settings > Teaming and Failover > move uplinks up/down into Active/Standby/Unused

Most set-ups that I have seen tend to use Active for one uplink and Standby for another uplink on a PG, then the opposite on another PG (e.g. PG1 uses uplink1 as Active and uplink2 as Standby, PG2 uses uplink2 as Active and uplink1 as Standby) - this allows for failover but would only cause contention when one link is down, this is good as lower QoS beats no Networking for some PGs in this scenario!

The above may be more intuitive in the C# Client (though this doesn't work with 6.5), these similar steps are also relatively simpler for standard switches but need to be configured individually on each host.

Bob

[jayharmer]

Do you see these Quad-link devices as 4 Physical uplinks at the host Networking level?

Still new here. I'm unsure. It looks like phy nic #1 is tied to management and everything works with that one cable plugged in but...

Phy nic #2 is tied to vsan and nic #3 is tied to vmotion. Please see attached images.

"IF" I wanted to attach phy port #1 to mgmt, #2 to vmotion and #2 to vsan, would I need to create separate single uplink port groups?

Thank you,

Jay

[TheBobkin]

Hello Jay,

To use more than one uplink per host you will have to connect more than one uplink from the hosts to the switch (‽) - All links other than vmnic0 are showing as down.

It is kind of hard to tell from your 3rd image whether these are assigned as intended (but still in down state), is that 6.5 vDS view or some NSX configuration (going by the name at top)?

Bob

[jayharmer]

VCSA 6.5 so yes, 6.5 vDS view. The name of the switch has NSX in it but not relevant  yet.

If i plug in another adapter and both nic ports (ex: 1 and 2) are in the same untagged vlan on the switch, then yes it works.  If not, it breaks, of course.

My original goal was to understand how to assign the vsan traffic to it's own nic port. Actually, want to know how to tie any vmkernel port to  a specific adapter.

[TheBobkin · Accepted solution]

Hello Jay,

What direction you likely want to go then is configuring VLANs correctly for traffic segmentation. Unfortunately I am not a Network design/implementation expert (though I know enough to fix it when it breaks vSAN!), so please have a read of the relevant sections of this document and inquire here or in Networking sub-forum if anything is not clear:

https://storagehub.vmware.com/export_to_pdf/vmware-r-vsan-tm-network-design

Bob