Are these numbers unusually high?

[wstoffel1]

I was looking at Inventory>Networking in the vSphere client when I noticed these numbers for a problem client.  Is this a cause for concern?  And how would I go about tracking down what's happening?  Thank you.

[lenzker]

there are a lot of dropped packets. Which network adapter have you build in your virtual machine?

What kind of applications are running in this virtual machines?

[MKguy]

Yes, these numbers seem way too high. Are you running anything special layer 2 related (like NLB) in these VMs?

Also check the (r)esxtop network view for dropped packets on these vNICs.

[wstoffel1]

Interestingly enough, i'm not a vm guy.  I'm a network guy.   And i've been working on this bandwidth issue for several weeks, see http://communities.vmware.com/message/2190976#2190976

When i run iperf i'll get the full 20Mbps, so the circuit is ok and there's very little if any dropped packets.  Everything "looks" ok, but when i sniff regular user traffic at random times during the day i'm seeing dropped packets, but couldn't place where.  That picture above, last night i was just poking around in vSphere for the first time and nearly fell out of my chair!

This virtual network is accessed from 2 locations.  One is Hartford over a dedicated point to point, the other is Stamford over their internet connection...Stamford has no issues.  Hartford does.  These are high end machines running CAD and Revit.  Most are with dual monitors on the local side. There's one or two zero clients, most are laptops and workstations accessing the VDI with View 5.01.  There are also about 5 physical machines on this virtual network that have PCoIP host cards in them.  Those are shared among all the users in both offices.

The thing that gets me, in testing you can access a Hartford VDI from Stamford and there's no problems.  In Hartford you can access a Stamford VDI and have a lot of problems.  Which almost assuredly points to a circuit bottleneck, but hartford is about 20 maybe 25% utilized.

It was great finding this, but now I'm bothered by just a couple of things.  One, the machines above are mostly Hartford machines but if this is actually the issue I've been looking for:

1. Why would a Stamford machine exhibit this behavior and it not be a problem?

2. It's only a couple of Hartford machines, why is the entire office affected?

I have a couple of logical reasons for both, but I pose the questions to see what you VM Gurus come back with

Thank you very much!

[wstoffel1]

I gotta hope the VMadmins are going to be familiar with Esxtop and Resxtop.  Quick search gave me the nuts and bolts of what they are, but I'm not about to go mucking around.

Would it make sense that this issue might only affect Layer2 circuits?

We have many customers on this infrastructure, and only recent additions, with point to point circuits are having issues.

And i just heard back from one of the admins, saying not to put too much faith in those numbers...?

[wstoffel1]

Does vCenter being buggy sound right?

SSH'd into one of the hosts that holds 2 of those VM's and ran esxtop and on the 6 physical nics there are zero dropped packets.

The search continues I guess.

[MKguy]

Are these ports VM vNICs or physical dvUplink ports?

You can also check dropped packets statistics for VMs in the GUI like in this screenshot.

The high number of packet exceptions is also jolting:

http://kb.vmware.com/kb/1029844

Would it make sense that this issue might only affect Layer2 circuits?

We have many customers on this infrastructure, and only recent additions, with point to point circuits are having issues.

It sure is pointing into that direction if you can isolate the issues on that particular property.

And i just heard back from one of the admins, saying not to put too much faith in those numbers...?

I'd rather say to not put too much faith into this opinion unless he can explain it the numbers properly or disprove them with (r)esxtop or contradicting statistics.

Does vCenter being buggy sound right?

SSH'd into one of the hosts that holds 2 of those VM's and ran esxtop and on the 6 physical nics there are zero dropped packets.

And were you experiencing the issues during that time? Did the dropped packet counter on those dvSwitch ports increase during that time? What if you vMotion the VM to another physical host, does the issue persist?

On another note, what hardware/NIC/driver are those boxes running on? There have been a couple of issues recently with older out of the box drivers that needed updating to work properly (e.g. HP DL gen8 server onboard NICs). Check and post the output:

# ethtool -i vmnic0
driver: bnx2
version: 2.2.1l.v50.1
firmware-version: bc 1.9.6
bus-info: 0000:03:00.0

# ethtool -i vmnic2
driver: e1000e
version: 1.1.2-NAPI
firmware-version: 5.11-2
bus-info: 0000:0b:00.0

# esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:03:00.00 bnx2        Up   1000Mbps  Full   00:1b:78:72:bb:16 1500   Broadcom Corporation NC373i Integrated Multifunction Gigabit Server Adapter
vmnic1  0000:05:00.00 bnx2        Up   1000Mbps  Full   00:1b:78:72:bb:0a 1500   Broadcom Corporation NC373i Integrated Multifunction Gigabit Server Adapter
vmnic2  0000:0b:00.00 e1000e      Up   1000Mbps  Full   00:26:55:df:7f:7c 1500   Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic3  0000:0b:00.01 e1000e      Up   1000Mbps  Full   00:26:55:df:7f:7d 1500   Intel Corporation 82571EB Gigabit Ethernet Controller


# ethtool --show-pause vmnic0
Pause parameters for vmnic0:
Autonegotiate:  on
RX:             off
TX:             off