Dropped RX packets in Windows VM only

[AllBlack]

Hi,

I have had an SR open with VMware support for a month now but we are no closer to a fix. I was hoping someone has experienced a similar problems and may have the solutions.

I am seeing dropped received packets for all my Windows VM but none of the RHEL VM under ESXtop. Netstat indicates issues too

The issue appears for Windows VM regardless whether it is 2003,2008 or 2012. Most of NIC are using vmxnet3 but I am seeing the issue regardless of NIC driver in use. I am seeing this at physically separated sites but only in those that are used for replication.

The VM are connected to a VDS which makes use of 10 Gbps connectivity and are located on two different clusters which make use of same storage backend (IBM N-Series 6240).
I have come to the following conclussions:

Site A:

-It is not vDS related as I moved VM to a vSS and packets still drop

-It is not pNIC related as my two clusters ( One HP DL380 and other DL560 )use different hardware and different type of pNIC. Therefore it is not a driver issue and I tried upgrading the driver

-Although no sign of contention, I evacuated hosts of all VM except one to rule out it out. I still noticed the dropped packets.

-I moved VM to local storage just in case there is something storage related but I still saw dropped packets

Site B:

-Same behaviour and these hosts have completely different hardware (IBM hosts) so not hardware or driver related.

Site C:

-I have a couple of DL380 hosts that make up a cluster and this one does not have any issues. It uses same make of storage backend but different model. All networking is configured the same and VM's are rolled from same template.

The thing that Site A and B have in common is that they use same type of backend storage. These are used for replication so we can use SRM.

Some things done in an attempt to fix it:

-Ensure that ESXi was patched to 5.1 U2. This did not make any difference

-Tested adjusting the buffers for the guest OS nic as per VMware's advice. This did not make a difference

-VMware asked to create vanilla windows VM. No software or joined to domain. Problem still exist.

One thing that made the problem disappear is altering the guest OS MTU to 9000. I just tried this out of desperation and VMware has not given me an explanation as to why this would be.

We never had issues with MTU 1500 in guest OS and prefer not to change this. We also ran a packet trace on one of the VM and noticed that the VM actually creates packages that are bigger than 1500 bytes even though the MTU is set to 1500.

[AllBlack]

I decided to have a look in a development lab I am building. This lab is vSphere 5.5 and makes use of separate storage and separate 10 Gbps switch

I noticed that a Windows 2012 VM shows packet loss in ESXtop as well. This lab has a cluster with 5 hosts but only 1 of the host is currently connected to a vDS. The VM happens to sit this host. I moved the VM to a vSS on another host which is currently not connected to vDS and the packet drop disappears.

I then created a vSS on the original host and migrated VM and connected it to vSS. The packet drop re-appears. This says to me it may be related to the fact that a host is connected to a vDS regardless of whether VM is connected to vDS. A few more things to note:

-The dropped packets show in the performance chart and ESXtop but only for VM not the vSwitch and the vmnic.

-vDS Health Check has been reported as a possible culprit but this is not enabled in this lab and I did disable it in the production environment as a test.

[ramkrishna1]

Hi

Welcome to communities.

If still have problem and loosing packet use some packet monitoring tool ,

For more please follow this -http://www.vmrage.com/using-vsphere-5-5s-enhanced-host-level-packet-capture/

[Pratyush_21]

Hello,

I also have a similar problem, but the drop is somewhat desired, because I am trying to pass malformed packet for some sort negative testing. But the thing is the VMXNET3 driver is dropping such packets on RX.

It seems that the intentionally corrupted packets are dropped on RX at the receiver port. The packets which are dropped are corrupted such that the IP header length is more than the Ethernet payload length.

I am using raw socket.

The linux is CentOS release 6.5 (Final).

The VM Network driver is vmxnet3, version: 1.1.30.0-k-NAPI

The drops are shown in the interface stat as below:

pkts rx err: 37800

drv dropped rx total: 37800

err: 37800

fcs: 0

rx buf alloc fail: 0

Can someone please help to find if there is any configurable by that I can receive the malformed packets?

Regards,

Pratyush Khan