VMWare ESXi 5.1 promiscuous mode.

[d2fg4]

Hi,

I have installed VMWare ESX5.1 and I created serveral virtual machines v.7 over it.

All the virtual machines are in the same vSwitch and Port Group which are configured to reject promiscuous mode.

The problem is if I dump the traffic from the vMachines I cat see ALL the traffic originated and destinated to other VMs.

I used tcpdump in order to dump the traffic like this:

tcpdump -i eth1 not host <my_laptop_ip>

And I see stuff like this:

16:03:45.386981 IP 192.168.19.108.http > 2.194.11.124.51972: P 40724:41157(433) ack 1189 win 175

192.168.19.108 is the IP address of another machine in the same ESX.

Is that normal?

Thank you in advance

[marcelo_soares]

all vms on the same network? this happens with any VM (from any you can see the same content)?

[rickardnobel]

Very strange. Do you see all kinds of traffic between all kinds of nodes?

If you run inside one VM: tcpdump -i eth1 -n icmp

and then from another VM ping the default gateway, do you see it inside the first VM?

[d2fg4]

Yes, that's exact.

If I run "tcpdump -i eth1 -n icmp" inside one VM and then, I  try to ping enything from another VM, I can see the traffic.

This happens on every VM on the same ESX server.

I tried the same on another ESX server and I had the same result.

[MKguy]

That seems very strange, unicast traffic should never be visible to other VMs like that. Are you sure promiscuous mode is disabled on the vSwitch AND port group level and one doesn't override the other?

Also post the output of the following commands from the host:

# esxcli network vswitch standard policy security get -v [vSwitchName]

# esxcli network vswitch standard portgroup policy security get -p [PortGroupName]

When ESXi rejects promiscuous mode on a port it also logs this to the vmkernel log file. Check your vmkernel logs if there are any messages similar to this one:

2013-05-25T12:52:58.596+02:00 vmkernel cpu3:7201556)etherswitch: L2Sec_EnforcePortCompliance:226: client VM-NAME.eth1 requested promiscuous mode on port 0x5000153, disallowed by vswitch policy

Also take a look at the vmware.log file in the VM's directory.

[d2fg4]

Strange:

~ # esxcli network vswitch standard policy security get -v vSwitch0

   Allow Promiscuous: false

   Allow MAC Address Change: true

   Allow Forged Transmits: true

~ # esxcli network vswitch standard portgroup policy security get -p "VM Network"

   Allow Promiscuous: false

   Allow MAC Address Change: true

   Allow Forged Transmits: true

   Override Vswitch Allow Promiscuous: true

   Override Vswitch Allow MAC Address Change: false

   Override Vswitch Allow Forged Transmits: false

I can't understand the directive "Override Vswitch Allow Promiscuous: true", both vswitch and portgroup doesn't allow Promiscuous mode. This is the Portgroup security configuration:

Thank you again

[MKguy]

I don't really think this will make a difference, but try disabling the explicit override of the port group to reject promiscuous mode. It should inherit the reject from the vSwitch level already.

Did you take a look at the vmkernel log as I mentioned too?

And you're not using VLAN ID 4095 on this port group by chance, right?

Also, you're not doing anything fancy networking wise in those VMs, right? In tcpdump, check that the actual destination MAC (-e for ethernet header) is not it's own or some broadcast/multicast bit set MAC too.

[rickardnobel]

Any update on this problem? Since it really should not work like that it would be interesting to know the reason.

[d2fg4]

Hi,

my apologies for the delay.

The destination MAC address (01:00:5e:28:13:01) of all the traffic is the multicast MAC address of the default gateway (which is a firewall cluster):

Esample:

15:05:35.724101 00:50:56:91:74:91 (oui Unknown) > 01:00:5e:28:13:01 (oui Unknown), ethertype IPv4 (0x0800), length 1454: 192.168.1.7.www > 151.65.13.131.63688: Flags [.], seq 25259:26659, ack 2262, win 105, length 1400

[MKguy · Accepted solution]

The destination being a layer 2 Multicast MAC, fully explains why other VMs in that VLAN see any outgoing traffic that is routed over that router. Note that you should not see any incoming frames originating from router, as the destination MAC of these frames would be the unicast MAC of the respective VMs.

Also, physical hosts on your network would see all this traffic just like VMs unless your firewalls send IGMP membership reports and you have IGMP snooping enabled on your layer 2 switches.

So the behavior you're seeing is basically "perfectly normal", from the vSwitch/layer 2 side.

That being said, mind telling us what kind of firewall or clustering are you using? Is this some active-active firewall cluster that requires multicasting? In any case, the vendor ID 01-00-5e corresponds to IPv4 multicast addresses. Are you seriously using a multicast IP (e.g. 224.x.x.x) as your default gateway in this subnet?! I'm fairly sure this is not how things are meant to work in the IPv4 world.

[rickardnobel]

Agree with MKguy above. Using a multicast IP and ethernet address as default gateway will typically mean that all traffic to the default gateway will be flooded at ALL ports in the LAN, unless the switches has support (and enabled) for IGMP snooping.

[d2fg4]

We are behind a Watchguard A/A Cluster and Cisco Catalyst switches.

The Watchguard cluster is reached by a unicast IPv4 IP (192.168.X.Y) and the multicast MAC address 01:00:5e:28:13:01.

Is it normal that the "problem" is only inside the ESXi hosts.

From the ESXi host "A" we don't see the traffic generated by the VMs on the ESXi host "B" (which sends to the multicast MAC address too).

This is becouse (I think) the Catalyst is instructed to route the traffic directed to "01:00:5e:28:13:01" to a specific port. (Am I right?)

Is there a way to do the same on the VMWare vSwitch?

Thank you.

[MKguy]

From the ESXi host "A" we don't see the traffic generated by the VMs on the ESXi host "B" (which sends to the multicast MAC address too).

This is becouse (I think) the Catalyst is instructed to route the traffic directed to "01:00:5e:28:13:01" to a specific port. (Am I right?)

Yes, in this case it seems like your physical switches forward the traffic directly to the ports the firewalls are attached to instead of flooding it to all ports in the VLAN. They would do this because the FW is most likely sending IGMP membership reports and the switches have IGMP-snooping enabled, allowing them to track where which multicast addresses are actually present and thus prevent the default flooding mechanism.

Unfortunately, the vSwitch (or distributed vSwitch too) has no such feature and will always flood frames originating from a VM with a multicast MAC as destination address to all ports in the VLAN besides the physical uplink. This is also explained very well here:

http://blog.ioshints.info/2010/11/vmware-virtual-switch-no-need-for-stp.html

Limited flooding of broadcasts/multicasts

Broadcast or multicast packets originated by a virtual machine are sent to all other virtual machines in the same port group (VMware terminology for a VLAN).

[d2fg4]

Thank you very much for you support.

This topic can be considered "closed".

[MKguy]

Sure, I'm glad if I was able to help.

Just one last thing to add:

The full-blown Nexus 1000V virtual Switch should be able to handle IGMP-snooping or maybe some other advanced feature to limit the flooding of these frames.