vSphere vNetwork

Hi All,

We recently found the dropped packets in one of the portgroup in the dvs.

Find the details below :

~ # esxcli network vm list

World ID  Name          Num Ports  Networks

--------  ------------  ---------  --------------------------------------------------------------------------------

   11431  ABCD-LI005v          3  dvportgroup-33, dvportgroup-64, dvportgroup-41

   11474  ABCD-LI002v          3  dvportgroup-64, dvportgroup-37, dvportgroup-33

   11535  ABCD-WS005v          3  dvportgroup-64, dvportgroup-41, dvportgroup-33

   12183  ABCD-LI004v          3  dvportgroup-33, dvportgroup-64, dvportgroup-39

   12458  ABCD-CS005v          3  dvportgroup-64, dvportgroup-40, dvportgroup-33

8133815  ABCD-WS004v          5  dvportgroup-241, dvportgroup-122, dvportgroup-33, dvportgroup-64, dvportgroup-39

8778004  ABCD-CS004v          5  dvportgroup-241, dvportgroup-121, dvportgroup-64, dvportgroup-38, dvportgroup-33

9607952  ABCD-CS002v          5  dvportgroup-241, dvportgroup-123, dvportgroup-33, dvportgroup-64, dvportgroup-36

9608033  ABCD-WS002v          5  dvportgroup-241, dvportgroup-124, dvportgroup-64, dvportgroup-33, dvportgroup-37

~ # esxcli network vm port list -w 9608033

   Port ID: 50331899

   vSwitch: dvSwitch0

   Portgroup: dvportgroup-241

   DVPort ID: 2360

   MAC Address: 00:50:56:ba:66:94

   IP Address: 0.0.0.0

   Team Uplink: vmnic5

   Uplink Port ID: 50331654

   Active Filters:

   Port ID: 50331902

   vSwitch: dvSwitch0

   Portgroup: dvportgroup-33

   DVPort ID: 157

   MAC Address: 00:50:56:ba:48:ce

   IP Address: 0.0.0.0

   Team Uplink: vmnic1

   Uplink Port ID: 50331650

   Active Filters:

~ # esxcli network vm port list -w 8133815

   Port ID: 50331874

   vSwitch: dvSwitch0

   Portgroup: dvportgroup-241

   DVPort ID: 2356

   MAC Address: 00:50:56:ba:6b:64

   IP Address: 0.0.0.0

   Team Uplink: vmnic5

   Uplink Port ID: 50331654

   Active Filters:

   Port ID: 50331876

   vSwitch: dvSwitch0

   Portgroup: dvportgroup-33

   DVPort ID: 151

   MAC Address: 00:50:56:ba:01:f0

   IP Address: 0.0.0.0

   Team Uplink: vmnic1

   Uplink Port ID: 50331650

   Active Filters:

~ # esxcli network port stats get -p 50331654

Packet statistics for port 50331654

   Packets received: 319985529

   Packets sent: 80426648

   Bytes received: 9160281096

   Bytes sent: 2967801542

   Broadcast packets received: 4284048

   Broadcast packets sent: 1174020

   Multicast packets received: 229618310

   Multicast packets sent: 13192

   Unicast packets received: 86083171

   Unicast packets sent: 79239436

   Receive packets dropped: 1623007

   Transmit packets dropped: 0

~ # esxcli network port stats get -p 50331650

Packet statistics for port 50331650

   Packets received: 1492943750

   Packets sent: 2246222030

   Bytes received: 37198486200

   Bytes sent: 2609396458292

   Broadcast packets received: 22352047

   Broadcast packets sent: 152967

   Multicast packets received: 39589668

   Multicast packets sent: 178492

   Unicast packets received: 1431002035

   Unicast packets sent: 2245890571

   Receive packets dropped: 30

   Transmit packets dropped: 208

Please assist as to how do I go about tracing as to what is happening?

Regards

Niri

This is a tough one. I'd start by throwing up a Wireshark VM and using the vDS port mirroring feature to see if you can capture and analyze some of those packets.  I'd also set up a SPAN port on the physical switch, pointed to a different Wireshark client to capture the traffic you want to analyze upstream, before it hits your ESXi host.

Now that I think about it a bit, that seems like quite a bit of work right off the bat.  You could have a physical layer issue.  Perhaps you can try swapping out cables or disabling or disconnecting ports that are configured for redundancy to see if removing a certain path eliminates the dropped packets.  This, then, might lead you to cables or NICs or switches or switchports.  Take into consideration the entire path from source to destination, although, when you do this, you might find that it's between your ESXi host and its first upstream switch.  I say this because malformed packets would likely be dropped way upstream if they were bad from the start.  Perhaps packets are only getting mangled on the last "leg" of their journey.

This might also be a security issue - perhaps you've set your vSwitches to drop forged transmits or MAC address changes.

You're looking at rolling your sleeves up and getting dirty on this one. Good luck!

-----------------------------------------

Please consider marking this answer "correct" or "helpful" if you found it useful.

Mike Brown

VMware, Cisco Data Center, and NetApp dude

Consulting Engineer

michael.b.brown3@gmail.com

Twitter: @VirtuallyMikeB

Blog: http://VirtuallyMikeBrown.com

LinkedIn: http://LinkedIn.com/in/michaelbbrown