VMware Cloud Community
andvm
Hot Shot
Hot Shot

ESXi Network design questions and best practice

Hi,

Am currently revising how best to assign network traffic for Management/vMotion/vSAN/VM Traffic when having 6x NIC scenario with same 10GB/s uplink speed

By default the ESXi Management gets connected to a standard vSwitch, is it best practice to leave it like this with a load balancing of Route based on originating virtual port and both network cards as Active?

For vMotion I have been using the vSAN dvSwitch (1 Active and 1 Standby uplink with Route based on originating virtual port) and specifying the other uplink as the primary with NIOC preferring the vSAN traffic in case of NIC failure. Potentially if the ESXi Management uses a dvSwitch I could place vMotion on their uplinks instead while using the same concept of setting the other uplink as primary and NIOC.

As with regards to VM traffic, is it best practice to set LACP active on Physical Switch side and a LAG on vCenter or to not set LACP on physical Switch side and use Route based on physical NIC load? Does this change which Load balance to use as default is Route based on originating virtual port?

Thanks

Reply
0 Kudos
6 Replies
T180985
Expert
Expert

Regarding LAG Ports, unless you have a real specific reason to use it, I wouldnt bother. Just adds extra complexity that you may not see much benefit from.


In terms of vMotion networking etc, there is some really good documentation on this. See: Networking Best Practices for vSphere vMotion

You dont need to set the uplinks to Active/Standby though as each vmkernel adapter only uses a single uplink. You can leave these set to Active/Active however i would move vMotion off the vSAN switch and onto your management switch

Or see https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi...  page 38 onwards

Please mark helpful or correct if my answer resolved your issue. How to post effectively on VMTN https://communities.vmware.com/people/daphnissov/blog/2018/12/05/how-to-ask-for-help-on-tech-forums
scott28tt
VMware Employee
VMware Employee

Moderator: Moved to vSphere vNetwork


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
MikeStoica
Expert
Expert

I'd leave the vSAN traffic alone, nothing else. Create a dvSwitch only for vSAN traffic and you can create another one for Management where you can have the vMotion traffic also.

andvm
Hot Shot
Hot Shot

The vSAN traffic would use only one uplink, the other uplink is there for failover so doing nothing really hence why I assigned vMotion to it.

The other option as you mention is to create a new dvSwitch to be used for Management and vMotion using the same above principle.

Do you see any disadvantages of having the Management vmkernel migrated to a dvSwitch?

Is there any documented process for migrating the Management from a Standard to Distributed Switch?

I am thinking the following, would this cause any downtime?

Create Distributed Switch

Assign Hosts

Assign both Physical Adapters

Create Port Group

Assign Management vmk to Port Group

Migrate any VM's Networking to the newly created Port Group

Thanks

Reply
0 Kudos
andvm
Hot Shot
Hot Shot

ok so based on the responses the ideal setup should be:

1x dvs (2x Uplinks) for both Management and vMotion in separate VLAN's with 1 Active and 1 Standby uplink (Alternate active uplink)

1x dvs (2x Uplinks) for vSAN. Does this dvs need to be 1x per cluster since each cluster vSAN traffic will be in a different VLAN? (Planning to use LACP/LAG so it diversifies uplink usage)

1x dvs (2x Uplinks) for VM Traffic. This should be just one dvs for all clusters for ease of management (Again planning to use LACP/LAG so it diversifies uplink usage)

The above will result in at least 3x dvs, do you agree that this the best practice/easier for troubleshooting?

If not please suggest alternate (such as by less # of dvs if possible and why preferred)

Thanks

Reply
0 Kudos
NathanosBlightc
Commander
Commander

It's not needed to create isolated distributed switches for each traffic inside the virtual network, you can separate them (mgmt, vMotion, vSAN & VMs) via changing the failover order for each of dvPortGroup inside single VDS.

However management can be easier while the troubleshooting maybe more complex

Please mark my comment as the Correct Answer if this solution resolved your problem
Reply
0 Kudos