Mirror quality in VDS with ESX 5.1 (Entreprise+)

[hervelemoal]

Hi everybody,

I'm asking some questions about the quality and transparency of mirroring in VDS.

To begin, the goal of this mirror is to make some performance measurement of the network and applications hosted in the ESX.

In order to be sure about the mirror quality we've done some tests.

Here's the lab. description :

• 2 virtual machines hosting windows server 2008

• network card config (default) on the two hosts:

∘ IPv4 checksum offload : rx&tx enabled

∘ Jumbo packet : disabled

∘ Large send offload (ipv4) : enabled

∘ Coalesce buffer : 128

∘ Receive Buffers : 256

∘ TCP Checksum Offload : Rx & Tx Enabled

∘ Transmit Buffers : 512

∘ UDP Chesksum Offload : Rx & Tx Enabled

• 1 Probe receiving mirrored traffic

• VDS:
• A Server : port 1
• B server : port 0
• Mirror out : port 11
  • Mirror config :
    • Source Port 0, direction Both
    • Allow normal IO : Yes
    • Encap Vlan : none
    • Preserve Vlan : Yes
    • Mirror packet length : 1500

The test is done with iperf with the following parameters:

• A Server 192.168.80.229 iperf server:
  • iperf -s -p 50000
  • wireshark capture during test
• B Server 192.168.80.223 iperf client :
  • iperf -c 192.168.80.229 -p 50000 -t 1
  • wireshark capture during test
• Probe 192.168.80.231
  • tcpdump -ni any -s0 host 192.168.80.223 and port 50000

Results :

• some packets coming from B (under 60 bytes size) are seen at 60 bytes in A and Probe
  • after some searches and tweaking we've found that a parameter : advanced parameters/net/Net.MinEtherLen = 60 by default
  • Test done with /net/Net.MinEtherLen = 0, result is now correct, packets under 60 bytes are now seen well
• Apart from that, the most critical thing is packet disorder :

    as you can see in this wireshark screenshot

    nearly all packets are not in order, ack is coming before data packet.

    as you can easily imagine, it complicates in a huge amount the probe behaviour.

>> Have you got other experience return ? Is there others parameters to tweak in advanced parameters ?

You'll find pcap file attached (Source, Destination and mirror)

[grasshopper]

Interesting post. I am curious if your results are more stable if you set the desired transmission size to 1472 (i.e. on iperf).  The reason I ask is I see some packets coming in with a size of 1514 resulting in the "TCP Window Full" message. With that said, I will humbly defer this to the real networking folks here (I'm just a server guy!).

[hervelemoal]

Hi Grasshopper, thanks for feedback

About 1514 Bytes packets seen :

• 1514 is the packet length seen in Wireshark splitted as below :
• 1460 Bytes of Useful Data :

• 20 Bytes of IP Header :

• 20 Bytes of TCP Header :

{ Please Note that MTU seats here : 1460 bytes of data + 20 Bytes IP header + 20 Bytes TCP Header = 1500 Bytes carried by Ethernet Layer}

• 14 Bytes of Ethernet Header :

Total packet size : 1460 + 20 + 20 +14 = 1514 Bytes

Hope it's more clear for you now ? Sometimes, MTU and packet size are confusing !

Regards