Management vmk dropping packets when vmnics are teamed

[michaelcprice]

Looking for help with the following issue:

Issue: When I team two or more physical nics on my ESXi 5.5 U1 host on the vswitch containing the management vmk and ping the host management vmk IP address from other devices it will drop packets.

Here is some information that I believe may be helpful to others in diagnosing my issue:

- The packet drop rate seems to be between 5-10% when pinging the mgmt vmk from another device on the network.

- When I ping from the ESXi host out to other network devices I get NO packet loss in my outbound pings.

- As soon as I remove the teamed NIC from the vSwitch and go back to using just one physical NIC the problem disappears.

- I am using "Route based on the originating virtual port ID" for my load balancing algorithm on the vSwitch and vmk port itself. No specific configuration settings have been done at the switch level in relationship to this.

- All the host's physical NICs are connected to a stack of 6 Dell PowerConnect M8024k switches. All 6 physical switches are cabled together and configured as one logical switch stack. M8024k switches are all I/O modules in the back of blade chassis.

- All physical NICs are dual-port Broadcom 57810 CNAs. There are total of 6 ports for each host spread across 3 CNAs.

- Al physical NICs are running NPAR to split each physical NIC port out into 4 virtual NICs. This results in a total of 24 vmnics for the host. (6 ports * 4 NPAR functions per port = 24)

- All NIC ports have a native untagged VLAN that the mgmt vmk uses and also carry additional tagged VLANs. Connecting ports on the M8024k switches run in General (hybrid) mode with the mgmt vmk vlan as the untagged vlan.

- Server hardware is Dell PowerEdge M620 Blade server.

Any help would be greatly appreciated. Have seen this issue on multiple hosts running off this particular blade chassis. On other hosts the problem seemed to magically disappear at some point but I cant seem to pin down what configuration change I made on them that resolved the issue for the other hosts.

Thanks,

Mike

[MFGCORP]

• Please enter replies above this line *** You are receiving this message because a ticket was created for your company. Replies to this email will be added as a note onto this ticket. Ticket #T20150312.0003: New message: "Management vmk dropping packets when vmnics are teamed" , Thank you for contacting us. A service ticket (#T20150312.0003) has been created for MFG UK Limited. We will attend to your ticket as soon as possible. The details of the ticket are listed below. When replying to this ticket, please ensure that the ticket number is included in the email subject line. Ticket #: T20150312.0003 Created on 12-03-15 01:38 (GMT) by Autotask Administrator Title: New message: "Management vmk dropping packets when vmnics are teamed" Description: You can access your service ticket via our client portal by clicking the following link: . If you do not have access to the client portal and would like to use it, please let us know. Sincerely, Your Support Team If you are a staff member, you can access this ticket by clicking the following link: .

[michaelcprice]

Thanks for the procedure. I will give this a shot and see what kind of results I get.

Mike

[MFGCORP]

• Please enter replies above this line *** You are receiving this message because a ticket was created for your company. Replies to this email will be added as a note onto this ticket. Ticket #T20150312.0041: New message: "Management vmk dropping packets when vmnics are teamed" , Thank you for contacting us. A service ticket (#T20150312.0041) has been created for MFG UK Limited. We will attend to your ticket as soon as possible. The details of the ticket are listed below. When replying to this ticket, please ensure that the ticket number is included in the email subject line. Ticket #: T20150312.0041 Created on 12-03-15 13:44 (GMT) by Autotask Administrator Title: New message: "Management vmk dropping packets when vmnics are teamed" Description: You can access your service ticket via our client portal by clicking the following link: . If you do not have access to the client portal and would like to use it, please let us know. Sincerely, Your Support Team If you are a staff member, you can access this ticket by clicking the following link: .

[cfizz34vmware]

what I am seeing in my environment is if you are in one chassis, then only one of the two vmnics will ping, and the opposite on the other chassis.  I am seeing this with brand new ESXi builds using 6.5.  I could have sworn I tested this in the past by putting one or either of the two vmnics in the mgmt networking (even before adding it to vcenter) and I could ping on both paths with no issues.  I too am using a dell chassis and nexus 5K's.

[schn]

Creating new DVswitch and portgroup has solved this issue.

[gparker]

I have just come across the same issue. I'm building a new vSphere 7.0U2 environment with HPE ProLiant servers. Some servers came with Broadcom 1Gb NICs and others came with Intel 1Gb NICs. The problem is ONLY happening on the hosts with Broadcom NICS! I've checked Broadcom firmware and drivers etc. When vmnic0 and vmnic1 are active, I get packet drops. When vmnic0 is active & vmnic1 is standby, no packet drops. When vmnic1 is active & vmnic0 is standby = packet drops. If I unplug vmnic0's ethernet cable and vmnic1 is active = no packet drops! Vmnic1 doesn't like the presence of vmnic0 on the management PG/vmk. Once again, this is only happening with the Broadcom adapters, the Intel's are rock solid.

I am going to try re-creating the management vSwitch/vmk0 like in the suggestions and see what happens.  

[waukeeit]

Did you try rebuilding the management interface yet? I've been having similar issues with one of my esxi host out of the three we have in production. Seems to be random high pings/latency going to the next router hop from the host and from the router back to the host's management IP.  I've tried placing one of the NIC into standby/unused but the random packet loss/high ping still occurs. It does it on both NIC when they're swapped to be in the active mode. 

I have HPE ProLiant Gen10 with HPE Ethernet 10Gb 2-port 562SFP+ Adapter having the issue. The other two hosts have the same hardware/configurations but i'm not seeing the same losses. 

Curious to know if your scenario was fixed with redoing the management network.

[gparker]

Hi, yes I did test it...I created a new management vSwitch/VMK assigning the Broadcom NICs to it and set the VMK's MAC address to "00:50:56...." - i.e., a VMware MAC address as opposed to the MAC address of one of the Broadcom NICs and that worked. But it's a bit of trouble/time consuming to do across all of my hosts, so I stuck with leaving VMNIC0 and active, VMNIC1 standby on all the hosts with Broadcom NICs.

Regards, George.