VMware ESXi 4.0 RARP Traffic

[SteveFuller2011]

Hi,

Can anyone enlighten me as to the reasons an ESXi host sends Reverse ARP (RARP) traffic?

We have approximately 30 ESXi 4.0 (build 256968) hosts supporting a Windows 7 based VDI environment, and looking at the traffic hittnig the external (Cisco) switches, we're seeing approximately 40% of the total packets sent to the network is RARP. This level of broadcast is much higher than I'd expect and so I'm trying to figure out if this is normal for ESX.

I've read various discussions and articales that highlight a number of reasons for ESX to send RARP packets, including VMotion, VM power on and NIC failover event.

I'm fine with those, but what we're seeing is RARP traffic sent periodically twice at 60-second intervals, with an approximate 15-second interval between the 1st and 2nd transmission (see attached traffic graphs). This pattern would suggest the aforementioned RARP causes are not the cause in this case as I'd expect those events to produce a random pattern.

Are there other reasons for RARP transmissions, and in particular, regular periodic RARP transmissions?

Thanks in advance.
Steve

[SteveFuller2011]

I've done a little more investigation of this issue, but we've still not found the answer as yet.

The RARP level is slightly higher than originally seen with a network capture showing RARP traffic by packet volume accounts for ~47% of total trafifc. Obviously not ideal.

When I dug a little more into the packet capture I see 1244 VMs sent RARP traffic during the capture period, with large numbers of VMs sending an identical number of packets and bytes e.g., 896 packets and 53,760-bytes during the capture period. This is seen in the attached rarp_sources.png.

To check whether the RARP was either originated by or as a result of some event on the VM, I took a Wireshark capture on one of the VMs and captured traffic from that same VM in the network at the same time.

I can't include the packet captures themselves, but what I can see is that the RARP traffic appears in the network capture, but not in the capture on the VM itself. In the attachment wireshark_comparison.png we observed RARP packets (frames 4-5, 14-71 and 76-77 highlighted in yellow) are only observed in the SPAN capture taken from the external Cisco switches. The implication of this is that the RARP frames are produced by the ESX host itself, and not by the VMs.

The duplicate IMGP Membership Report and Leave Group messages (frames 74-75) seen only in the SPAN capture is due to the replication of the IGMP messages within the Cisco switches to which the ESX hosts are connected so is not an issue here.

We've got a ticket open with VMware now to see if they can shed any light and I'll feed that back as and when it's forthcoming.

Regards