Packet Loss under Virtual Machine

[mjiang]

Hi there,

One of our customer is experiencing serious drop-outs to a number of web sites that their applications exchange data with. The application is running on over a dozen Windows 2003 virtual machines under 3 x 3.0.1 ESX servers (HP DL585) in HA/DRS cluster. The virtual environment has been running happily since 04/2007 until last Sunday.

During investigation, pingplotter is used to troubleshoot potential packet loss. It has been found that almost all virtual machines suffer packet loss to remote web sites, while physical Windows box (either a HP server or a laptop) does not drop a single packet, when plugged into the same switch.

\* 'ifconfig -a' output for vmnics in failover-only. Both hard-coded to 100full

vmnic4 Link encap:Ethernet HWaddr 00:17:08:7D:AB:2A

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:2458984635 errors:0 dropped:0 overruns:0 frame:0

TX packets:2305693456 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:646256027 (616.3 Mb) TX bytes:922113379 (879.3 Mb)

Base address:0x7000 Memory:fdee0000-fdf00000

vmnic5 Link encap:Ethernet HWaddr 00:17:08:7D:AB:2B

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:111102594 errors:0 dropped:0 overruns:0 frame:0

TX packets:1772 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:2689378738 (2564.7 Mb) TX bytes:297097 (290.1 Kb)

Base address:0x7020 Memory:fdea0000-fdec0000

\* 'sh int counters errors' output of the C2950 switch which both esx servers and physical windows box plugged in after 'clear counters' and wait for a while

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize

Fa0/1 0 0 0 0 0

Fa0/2 0 0 0 0 0

Fa0/3 0 0 0 0 0

Fa0/4 0 0 0 0 0

Fa0/5 0 0 0 0 0

Fa0/6 0 0 0 0 0

Fa0/7 0 0 0 0 0

Fa0/8 0 0 0 0 0

Fa0/9 0 0 0 0 0

Fa0/10 0 0 0 0 0

Fa0/11 0 0 0 0 0

Fa0/12 0 0 0 0 0

Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants

Fa0/1 0 0 0 0 0 0 0

Fa0/2 0 0 0 0 0 0 0

Fa0/3 0 0 0 0 0 0 0

Fa0/4 0 0 0 0 0 0 0

Fa0/5 0 0 0 0 0 0 0

Fa0/6 0 0 0 0 0 0 0

Fa0/7 0 0 0 0 0 0 0

Fa0/8 0 0 0 0 0 0 0

Fa0/9 0 0 0 0 0 0 0

Fa0/10 0 0 0 0 0 0 0

Fa0/11 0 0 0 0 0 0 0

Fa0/12 0 0 0 0 0 0 0

\* vmxnet driver is installed & used automatically in all virtual machines

\* as usual no change has been made on the esx servers for a while

We are basically stuck with no idea what to look/try next. Any suggestion/hint will be greatly appreciated.

Regards,

mjiang@mct