Stacked / Nested / QinQ 802.1Q VLAN tags to physical interface: How to?

[screefchap]

Hi Guys,

We've bumped into the following issue, were hoping that you could enlighten us.

We have several VMs, each of which is connected via a dedicated portgroup that applies a unique (i.e. per-VM) VLAN tag. All of these portgroups are connected to the same single external interface. The net effect is that the traffic from each VM is multiplexed across a single physical interface.

Everything works as advertised, provided that traffic from the VMs contain only untagged traffic.

If, however, we send tagged traffic from a VM (regardless of the specific tag we apply), we observe that no traffic egresses the physical interface, and we suspect that it's being dropped by the portgroup.

What we would like is that the portgroup applies its tag to each frame from the VM regardless whether the frame is tagged or not. So for instance, in the specific case of singly-tagged frame, we would expect to see a doubly-tagged frame leaving the portgroup and physical interface, where the inner tag is the one applied by the VM, and the outer tag is the one applied by the portgroup.

Any idea how we can make this happen?

Thanks,

-Bruno

PS: If you're wondering why we would even attempt such a thing, it's because what's described above is a QA environment for networking products.

[mike_laspina]

Hello,

Yes this architecture works, however I would not use it myself under normal conditions.

You will need to assign VLAN 4095 to the port group as this will then carry the tagged packet to the pSwitch for handling.

Here is some addtional info on it.

[screefchap]

Hi Mike,

Thanks for the links.

We're aware of the 4095 "trick", but what we want to do isn't simply pass tagged traffic. We wish to tag traffic whether it already has a tag or not.

So there are actually two cases:

1. An untagged frame leaves the VM. In this case, everything works as expected, and as described in the VLAN whitepaper.

2. A tagged frame leaves the VM. In this case, what we want is that a second tag is added to the frame. What we see happening is that the portgroup is dropping the tagged frame. If we were to set the tag to 4095, the tag that we want to have added wouldn't be -- both in this case, and in the first case too.

-Bruno

[mike_laspina]

Assigning the VLAN port ID to 4095 instructs the vSwitch to forward a VLAN tagged packet, it is not used an a VLAN data plane but more like a backbone.

You cannot assign an additional tag to a 802.1q packet it will not work.

In this case you would assign VLAN tag ID's to the 802.1q VM interface and it would communicate on those VLAN ID planes over the vSwitch backplane.

[Texiwill]

Hello,

The vSwitch is seeing the double encapsulated packet as an attack and dropping it. Note you are using VGT (Virtual Guest Tagging) in combination with VST (Virtual Switch Tagging) which means you end up with a double tagged/encapsulated packet. This is seen as a VLAN attack bu the vSwitch and therefore dropped.

You need to use portgroup 4095 if you will be using VGT.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[screefchap]

Edward,

Thanks for the reply.

We do wish to stack VLAN tags, which is a perfectly legal (and commonly done) in the carrier space. I do realize that this may appear to be quite unusual as viewed from the perspective of an enterprise or server admin, as echoed by some of the reactions in this thread.

I'm not sure I agree with your assesment that it's the vswitch that's dropping the traffic. I say this because when setting the VLAN to 4095 on the portgroup, we're able to send doubly-tagged frames from the VM out the physical interace just fine. My guess is that the portgroup is where the problem lies.

If I were to imagine how this would be handled if there were a GUI option for it, it would be a tickbox near the VLAN box that would say "allow vlan stacking".

Is there, perhaps such an option that can be added elsewhere?

Thanks,

-Bruno

[Texiwill]

Hello,

I'm not sure I agree with your assesment that it's the vswitch that's dropping the traffic. I say this because when setting the VLAN to 4095 on the portgroup, we're able to send doubly-tagged frames from the VM out the physical interace just fine. My guess is that the portgroup is where the problem lies.

vSwitch or porgroup it makes no difference. The portgroup is defined within the vSwitch. I imagine it is just an array or sorts. The vSwitch does all the heavy lifting. The double encapsulation protection is within the vSwitch. Portgroup -> vSwitch -> pNIC. Since there is no way to monitor either the portgroup or vSwitch, it would be a hard thing to tell exactly where this protection is within the code. It is not ignored within VLAN 4095, I just found out, but VGT lets the guest handle everything. I do not think it will even allow double encapsulation with VGT, but I sent a note to the security/networking people I know within VMware.

Is there, perhaps such an option that can be added elsewhere?

There is no support for this at the moment. I would open up a feature request with either your VMware Sales or Support Representative. It sounds like it could be very useful.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

Message was edited by: Texiwill

[admin]

If I understand this situation correctly, the issue is that we will not apply a second tag to the packet.

That is absolutely true and it is notsomething that we currently support. From the vSwitch perspective, it is not necessarily looked at as an attack, but the fact is that we will just not provide the second level of encapsulation, nor will we support any frames that are terminated at the vSwitch that are double encapsulated.

By setting the portgroup VLAN tag to 4095 we will just pass the traffic directly through to the VM, because we are not terminating the VLAN and are therefore not the one that are stripping the tags off in VGT mode. In VGT mode it is up to the OS in the VM to parse the VLAN tags. So, it is not that it removes any other protections, it actually just takes the vSwitch out of the VLAN picture because the user has decided that they want the OS to manage the VLANs. That said, the protections like Forged Transmits, MAC Changes, and Promiscuous Mode can still be applied.

[screefchap]

Hi Randall,

Yes, your understanding is correct.

We would heartily applaud this functionality being added.

While the environment described isn't a typical enterprise-style setup, adding VLAN stacking would enable us to virtualize literally hundreds of existing physical PCs.

My suspicion is that adding this wouldn't be a huge deal, since you already know how to tag and untag frames.

Thanks,

-Bruno

[Texiwill]

Hello,

I would still open a feature request with your VMware Sales or Support Rep. Rob can transmit that as well to the proper folks, but it looks MUCH better coming direct from the customer following the proper procedures.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[screefchap]

Edward,

We're working with Gerald Camacho to get a formal feature request assembled and submitted.

While waiting for a real fix to this problem, we came up with the following creative workaround:

Instead of doing tagging on the port group, insert a very small VM (as in bootable floppy linux firewall that supports VLAN stacking) between each of the VMs and the virtual switches (which are all set to 4095).

Thanks to everyone for your help and insight on this problem.

-Bruno

[Texiwill]

Hello,

May I use this as an networking example in my next book? It is an intriguing requirement and solution.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[screefchap]

Absolutely.

Bruno

[screefchap]

An update on this issue.

Cisco plans to release what appears to be a virtual implementation of a richer-featured switch. They're calling it the "Nexus 1000V Virtual Switch".

According the the link above, the 1000V supports 802.1q, but it doesn't mention anything about QinQ.

Access to the beta of the 1000V is being restricted to people who are on the ESX 4.0 Beta, and are on 64-bit hardware. We're attempting to get on the ESX 4.0 beta for this reason; we have plenty of 64-bit hardware.

In the meantime, can anyone confirm whether the 1000V supports QinQ or not?

Thanks,

Bruno

[Texiwill]

Hello,

Unfortunately the details of the Nexus 1000V and ESX VI4 betas can not be shared on a public forum. The NDAs signed are pretty strict. In your position I would work with Cisco to get further information under NDA.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Fastlane]

Hello,

I have the same problem, but did't get it work with the workaround VLAN 4095

If I use VLAN 0 QinQ packets are forwarded to the Linux Vmware, but all outgoing packets are droped on the VSwitch as described here.

With VLAN 4095 a single trunk goes through, but a QinQ packed is already droped incomming. Any ideas what to do here?

We found one solution : We build a Layer 2 tunnel from an outside Linux Server to the Vmware Linux and tunnel the QinQ packet. This works, but we would prefere a solution without an additional tunnel.

I also hope that the Nexus 1000V will support QinQ, but I think the chance is not high as the Nexus 5000 did't support it. On other Cisco switches like Catalyst 3750 or 6500 you can configure dotq tunneling.

Thanks

Olaf

[Fastlane]

Hello Bruno,

Maybe I understand your workaroud not correct. As I understand the plan is to decapsulate the QinQ packet on Linux Vmwares running on the ESX. But how do you get the QinQ packet to them? Has it not to go also through a VSwitch where it is droped even with VLAN 4095?

Thanks

Olaf

[screefchap]

Hi Olaf,

We have observed that when set to 4095, that traffic is moved in and out of the VM and/or the physical NIC without being touched -- whether the frame has a tag or not. It sounds as though this is not what you have experienced, which surprises me.

In other words, a vswitch with a tag of 4095 behaves like a true switch in that it simply forwards frames, regardless what they contain (untagged, single tagged, multiply tagged).

It's only when we attempt to tag or untag using a vswitch (i.e. tag != 4095) that we experience the inability to stack tags.

-Bruno

[RBurns-WIS]

I don't believe QinQ will be supported at initial release. Subsequent releases are another story.

[iwagner]

Just to be clear here: do the virtual switches support QinQ packets or do they drop them? In my experience the QinQ packets are dropped, but it seems that some people have seen the packets come through. Do different builds of ESXi have different behavior with QinQ packets? Is the official VMware stance that QinQ packets are dropped?

Thanks,

Isaac