Dropped output packets

[esnmb]

We’re seeing dropped output packets on about half of the virtual Ethernet interfaces.  We were getting complaints from some departments of sporadic network performance.  Also getting alarms from our monitoring devices saying some systems are offline randomly, but of course when we check they are up.

Last weekend we put in the Nexus 1000V and we are getting much more visibility to actually point the finger at VMware.  Before we were pointing at the third party vendors or the monitoring equipment....  Fun.

Here is an example of what we see in the Nexus.

Vethernet22 is up

    Port description is SRVR-MARKET, Network Adapter 1

    Hardware is Virtual, address is 0050.56ae.07ee

    Owner is VM "SRVR-MARKET", adapter is Network Adapter 1

    Active on module 6

    VMware DVS port 546

    Port-Profile is N1KV-Network0

    Port mode is access

   5 minute input rate 37 bytes/second, 0 packets/second

    5 minute output rate 26267 bytes/second, 33 packets/second

    Rx

    6934 Input Packets 6624 Unicast Packets

    0 Multicast Packets 310 Broadcast Packets

    1479868 Bytes

    Tx

    1631896 Output Packets 1021706 Unicast Packets

    120860 Multicast Packets 489330 Broadcast Packets 899046 Flood Packets

    1091877424 Bytes

    1 Input Packet Drops 3275 Output Packet Drops

[RBurns-WIS]

That's the great thing about the 1000v.  Network visibility into your VMs traffic just as you have in the physical world.

vEth output drops are usually indicative of a guest VM OS issue.   I've seen the e1000 driver cause such issues.  Might want to try the VMXNet3 driver and see if that makes any difference.  Either way you determine why the traffic is being dropped.

If you're curious you can do a vempkt capture on the VEM hosting the VM experiencing the output drops.  You can even filter by "dropped" packets.

If you're interested in trying this let me know and I'll provide detailed instructions.  This will allow you to examine the packets before the 1000v drops them.

Regards,

Robert

[esnmb]

Cisco and VMware suggested using the vmxnet3 hardware.  I added it to several guests and that seems to be correcting the issue so far.  I'll update and let you know!

[depping]

I've seen this issue in the past when the e1000 was used and I believe it had to do with  buffer size, but it has been a while. for me upgrading to vmxnet3 also did the trick.

By the way, you can also see dropped packets by looking at esxtop!

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive