I finally solved this for my environment.
The solution was two fold...
First thing I needed to do was edit the VMWare Adapter Polcy on my Cisco UCS to increase the ring 1 buffer size from 256 to 4096 (the maximum value) and enabled RSS (Receive Side Scaling)
The immediately stopped all of the discards happening on the vmnic interfaces of the ESX server and drastically reduced the overall drops I was seeng in my entire environment. It seems that Cisco's default adapter policy has a very small buffer and so if the traffic is bursty this can be overloaded and will cause drops. Changing this value along with enabling RSS to allow more cores to handle the workload seemed to remove that bottleneck and push the packets as quickly as possible to the actual VMs.
After a little while I did notice the side effect of this change - I was still seeing some periods of dropped packet spikes still but now these drops were showing as Tx drops on our Nexus 1000v ports and Rx drops on the OS adapter interface. What this meant to me was that the OS was not processing the incoming traffic quickly enough.
To make this problem better I logged into the OS and edited the Adapter configurations advanced settings for the VMXnet3 adapter to the following:
RSS - Enabled
RX Ring #1 Size - 4096
Small Rx Buffers - 8192
After tweaking these settings all discards and dropped packets in my environment have almost entirely ceased. Where I was seeing sometimes tens of thousands of drops each day I am now seeing less than 50.
These are some of the key takeaways I learned during this process:
- If you are seeing drops on the vmnic interfaces it is most likely due to some buffering issue somewhere in the path. In my case it was at the hardware level. Working with VMWare support we found a value of rx_no_buffs on the VMNic interface which perfectly correlated with the drops we were seeing. We traced this to a value of rq_drops on the Cisco Palo interface and this allowed us to see that it was indeed the hardware adapter that was dropping the packets
- RSS allows for more than one core in a system to process the TCP stack. In a VM that has a lot of traffic to handle, this can really help to clear the buffers more quickly. Even if you have huge buffers - if the OS is not clearing them quickly enough they will still fill up. This is also something to be concious of when you are using CPUs that have many cores with low clock rates since even less CPU is dedicated to process the network traffic without RSS.
- Packet drops of this amount are NOT normal. Keep fighting until you figure out what is causing it if you see this in your environment. Several issues in our network have improved since this discovery. VMware should be able to tell you what is causing these drops if you keep on them long enough to make them examine the adapters.
