VMware Communities: Message List - Network | Promiscuous Mode

Re: Network | Promiscuous Mode

JMills — Sun, 21 May 2006 10:34:01 GMT

Actually, a very good request, since the instructions you quote date back quite a ways now.

Maybe a weekend project for me once the current time-consumer switches to maintenance mode. ;-d

Re: Network | Promiscuous Mode

VMadmin — Thu, 27 Apr 2006 20:19:32 GMT

Thank you for the excellent write up Brandon! I found this extremely helpful.

One bit I'd like to add, I recently got instructions from VMware support to set up sniffing on the virtual switch. Here are the instructions I received:

--------------------
For sniffing all traffic on the vmnic0 (you can substitute this for a bond0, etc.)

# echo "PromiscuousAllowed yes" > /proc/vmware/net/vmnic0/config
#insmod vmxnet_console devName="vmnic0" nameMask="test"

Then bring the device up:
#ifconfig test up

Start the tcpdump
#./tcpdump -i test -w <path&filename_of_capture_file>

To read the file:
#./tcpdump -r <path&filename_of_capture_file>
--------------------
Note: Vmware e-mailed tcpdump to me but I believe it is on the ESX CD.

It would also be great if VMware created a technical document that provides an authoritative set of instructions and published it in their technical documents. My fear is that over time, Brandon's post will get buried and this question will be posted again!

Re: Network | Promiscuous Mode

jeffbrooks — Wed, 05 Apr 2006 05:17:37 GMT

Does setting ethernet[n].noPromisc = TRUE in the VMX actually work for anyone? I've tried multiple times to disallow a vNIC from being in promiscuous mode when the vSwitch is promiscuous. Echoing into the vNIC seems turn off promiscuity, but I can't seem to get it to work from the VMX.

Re: Network | Promiscuous Mode

ahoogerhuis — Tue, 21 Mar 2006 05:23:26 GMT

Similar, I have that setup, as a routed setup, wiht outer and inner vSwitches, and a iptables firewall sitting on both nets. That works like a charm. What i needed to add was that the same host also had a few tapX interfaces and ran openvpn, and then tried to bridge tapX onto eth1 (the inside vSwitch).

I can't seem to get much of anything non-broadcast to get across the bridge. I can get ARP to resolve, but the ethX interface on the host running the bridge will not get packets destined to the tapX side of the bridge.

When I run "brctl showmacs br0" on the bridge host lists correctly the MACs to be seen on bridge port 1 (eth1) and 2 (tap1). When i run arping from a host located on the eth1 side of the bridge to resolve the IP/MAC of a host on the tap1 sid eof the bridge it works. If i ping between the same hosts it doesn't work, and tcpdump of both the bridge interface itself or the p.ort interfaces of the bridge show's no trace.

(also, to add, I'm running the same setup on a physical machine, that works ust as intended. If there are differences in the setup then I have not spotted them after pouring on this for days)

Message was edited by:
ahoogerhuis

Re: Network | Promiscuous Mode

JMills — Tue, 21 Mar 2006 03:48:12 GMT

So you have something like this, right?

Physical network bound to vSwitch "1"

"bridge" Host "A" connected to vSwitch "1" and vSwitch "2"

"remote" Hosts {B..F} connected to vSwitch "2"

Re: Network | Promiscuous Mode

ahoogerhuis — Tue, 21 Mar 2006 01:29:56 GMT

Hi, thanks for an excellent writeup. I have one thing I cannot make work properly for the life of it, and was hoping this would solve for me: ethernet bridging in Linux.

What I am trying to do is have one server running openvpn and bridge tapX into ethX (connected to a virtual switch inside ESX). This seems to work partially, but not fully.

For the not so Linux-minded, basically what I am trying to archive is to have a linux virtual network adapter and the esx vlance ethernet adapter bridgted at the OS level insid eLinux so taht clients connecting over a VPN would appear to be bridged into the inside LAN that exists in a vSwitch.

I get broadcast related traffic across the bridge, but anything unicast seems forever lost and will not appear on the ethX interface of the ESX host doing this.

If anyone has tried and succeded in similar travails I'd love to know what Im doing wrong. I've had a similar setup work fine on a GSX server (yes, I know resources are different there).

Re: Network | Promiscuous Mode

JMills — Tue, 21 Feb 2006 13:41:57 GMT

In my efforts to find information about promiscuous
mode settings, I've found the Admin Guide to be
limited and some of the threads pretty scattered, so
I've written this for anyone looking for help with
their promiscuous mode settings.

GSTK (Good Stuff To Know) | Promiscuous Mode

Definition: In regards to VMware ESX Server;
promiscuous mode is used to allow a host to see all
traffic passing on a vswitch.

Uses: Firewalls, Port Sniffers, IDS, etc.

Requirements: When using a vswitch, both the virtual
adapter and vswitch must have promiscuous mode
enabled. When not using a vswitch (e.g. if you had a
dedicated NIC for sniffing and used port mirroring on
your physical switch) it just needs to be enabled on
the virtual adapter. (The latter is a good config if
you dont want to incur the processing overhead
associated with promiscuous mode for a sniffer
application).

"Enabled" would be a bit of a misnomer for the virtual switch layer... what you're doing is flipping a permissions bit which allows a vNIC it actually see traffic when it is put in promiscuous mode.

Step 1 | Enable Promiscuous Mode for the VM

Add a line to the configuration file, this probably
goes without saying, but substitute n for the adapter
(0-4) and remove the brackets.

ethernet[n].noPromisc = FALSE

Youll want to apply this to all VMs you need
promiscuous mode enabled on.

Technically, this is 'FALSE' by default, but good to discuss in the context of more locked-down environments where it may be explicitly set to 'TRUE' for all vNICs.

Step 2 | Enabling the Virtual Adapter and/or vswitch
to Use Promiscuous Mode

For those of you familiar with Ciscos IOS, enabling
the virtual adapter for promiscuous mode
automatically turns its port on the vswitch into a
mirror port of sorts, but it doesnt mirror only
selected ports, it mirrors traffic from all ports on
the vswitch. You must enable promiscuous mode for the
vswitch, in addition to the virtual adapters for this
to work. Mirroring can be selectively disabled for
specific MAC addresses if needed, but otherwise its
a global setting for the vswitch. (Refer to the Admin
Guide, a hyperlink is provided in the references
section at the bottom, if you want to know about the
MAC address piece).

Yes, but it will only "mirror" traffic within the same broadcast domain.

Re: Network | Promiscuous Mode

Brandon_Sanders — Fri, 17 Feb 2006 18:43:30 GMT

Thanks. I'm glad it's helping people out.

I'm on my way to Virtuoso; only 5001 points short you know.

Re: Network | Promiscuous Mode

grasshopper — Fri, 17 Feb 2006 18:44:51 GMT

you'll be there before you know it

Re: Network | Promiscuous Mode

DerekA — Fri, 17 Feb 2006 18:42:17 GMT

Yes, thanks for the write-up. We are looking at implementing a bit of IDS infrastructure in vm's so this will be of great value.

Network | Promiscuous Mode

Brandon_Sanders — Fri, 17 Feb 2006 17:43:00 GMT

In my efforts to find information about promiscuous mode settings, I've found the Admin Guide to be limited and some of the threads pretty scattered, so I've written this for anyone looking for help with their promiscuous mode settings.

GSTK (Good Stuff To Know) | Promiscuous Mode

Definition: In regards to VMware ESX Server; promiscuous mode is used to allow a host to see all traffic passing on a vswitch.

Uses: Firewalls, Port Sniffers, IDS, etc.

Requirements: When using a vswitch, both the virtual adapter and vswitch must have promiscuous mode enabled. When not using a vswitch (e.g. if you had a dedicated NIC for sniffing and used port mirroring on your physical switch) it just needs to be enabled on the virtual adapter. (The latter is a good config if you dont want to incur the processing overhead associated with promiscuous mode for a sniffer application).

Need to know: vmnet = internal only switch, vmnic = vswitch with only 1 dedicated physical adapter, bond = vswitch with 2 or more dedicated adapters. Familiarity with startup script syntax and precedence. COS file system.

Step 1 | Enable Promiscuous Mode for the VM

Add a line to the configuration file, this probably goes without saying, but substitute n for the adapter (0-4) and remove the brackets.

ethernet[n].noPromisc = FALSE

Youll want to apply this to all VMs you need promiscuous mode enabled on.

Step 2 | Enabling the Virtual Adapter and/or vswitch to Use Promiscuous Mode

For those of you familiar with Ciscos IOS, enabling the virtual adapter for promiscuous mode automatically turns its port on the vswitch into a mirror port of sorts, but it doesnt mirror only selected ports, it mirrors traffic from all ports on the vswitch. You must enable promiscuous mode for the vswitch, in addition to the virtual adapters for this to work. Mirroring can be selectively disabled for specific MAC addresses if needed, but otherwise its a global setting for the vswitch. (Refer to the Admin Guide, a hyperlink is provided in the references section at the bottom, if you want to know about the MAC address piece).

At the service console, logged in as root, type:

echo "PromiscuousAllowed yes" > /proc/vmware/net/[vmnic0]/ config

This allows the guest operating systems in all virtual machines using [vmnic0] to enable promiscuous mode. If the adapter is using a different network, such as vmnet_0, make the appropriate substitution in the command. For a bond, apply this to the individual vmnics that are members of the bond as well as the bond itself. (e.g. vmnic0, vmnic1, and bond0 if bond0 was comprised of members vmnic0 and vmnic1. Look at the config file of the bond to find out which vmnics are members (a.k.a. teaming slaves) of the bond).

*Important Note: This change is applied to the /proc filesystem, which means this setting is applied to the running config only (per say). You must perform Step 3 below to make this setting non-volatile (survive a reboot of the ESX server).

Step 3 | Making the Promiscuous Mode Setting Non-Volatile

In case of an ESX server reboot, this setting needs to be applied to the /proc filesystem vswitch configs before the virtual machines startup sequence begins. This ensures all your sniffer apps or whatever come up clean in the event your ESX server is rebooted.

I would steer away from creating a new script and setting its startup order. Instead I would add a simple entry to the end of the /etc/rc.local script. This applies the change before VMs start powering on. In my opinion, this is the cleanest way to make sure these commands get executed on each system startup and before VMs power on.

Add the following command to the end of the /etc/rc.local script. Repeat for each vmnet, vmnic, or bond you modified in Step 2.

echo "PromiscuousAllowed yes" > /proc/vmware/ net/[vmnic0]/ config

-End of Instructional Steps-

Why Written: The Admin Guide for ESX leaves something to be desired when speaking about Promiscuous Mode settings and all the associated configuration dependencies for it to work.

References: http://www.vmware.com/support/esx21/doc/esx21admin_virtualadapter.html or page 360 in the ESX Server 2.5 Administration Guide PDF and VMware Knowledge Base Answer ID 1557.

All Due Respect: I quoted some pieces from the VMware Admin Guide. Also, the user posts from JMills, lambeth, and ken.cline@hp on the VMTN forums helped me out a lot while I was learning this for one of my environments.

Brandon Sanders, VCP
bsanders@data-strategy.com
Systems Engineer | Data-Strategy

Re: Network | Promiscuous Mode

grasshopper — Fri, 17 Feb 2006 18:25:57 GMT

Nice write up Brandon.