Im hoping to find some clarification and a better understanding of using VLAN tagging on vSwitches with ESX. I recently installed ESX 3.0.2 on 2 HP bl480c servers with 4 pNICs each in a c-Class chasis.. Two of the NICs are for the service console and vMotion in a failover configuration. The other two are for VMs. All four are internally connected to a Cisco 3020 switch on the back of the blade chassis. The Cisco 3020 is then trunked back to the core switch we have in the room, Cisco 3750.
My understanding is that the general consensus is to trunk the physical ports where the blade servers are connected to on the internal Cisco 3020 switch and then set VLAN tagging on the vSwitches inside ESX. However, the network switch team here at work is telling me that they dont want to trunk those ports. Instead they want to assign the VLAN id to the physical ports themselves. From an ESX standpoint, i would then have to create multiple vSwitches with dedicated pNICs assigned to each.
My question is why? When I asked them why, i didnt get an answer -- only that it was bad network design. This doesnt sit well with me and I would like to hear from what the community says about this. Other then the cliche "It depends ...", what is the real answer to this?
Not using tagging is very in-efficient and is almost a must when using blade servers that have a limited number of NIC's. That is not bad network design at all and is a very common practice. Sounds like your network people need to get themselves educated and get out of the stone age. The below guides talk about this...
VMware ESX Server 3 802.1Q VLAN Solutions - http://www.vmware.com/pdf/esx3_vlan_wp.pdf
ESX Server, NIC Teaming and VLAN Trunking - http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/
Integrating Virtual Machines into the Cisco Data Center Architecture - http://www.cisco.com/univercd/cc/td/doc/solution/vmware.pdf
VMware Virtual Networking Concepts - http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf
Networking Virtual Machines - http://download3.vmware.com/vmworld/2006/TAC9689-A.pdf
Networking Scenarios & Troubleshooting - http://download3.vmware.com/vmworld/2006/tac9689-b.pdf
ESX3 Networking Internals - http://www.vmware-tsx.com/download.php?asset_id=41
High Performance ESX Networking - http://www.vmware-tsx.com/download.php?asset_id=43
Network Throughput in a Virtual Infrastructure - http://www.vmware.com/pdf/esx_network_planning.pdf
From the above list, I would focus them on the following whitepaper:
Virtual Switch Tagging (VST) really is the only viable option in networks with more than one VLAN. Otherwise you would need two physical attachments for each network that the ESX box will service. This gets bad very quickly.
In many organizations the network team experiences trepidation because trunking to a server has been non-standard in the past. The important thing to emphasize to your network team is that they are not really trunking to a server. The are trunking to a Virtual Switch that is certified compatible with Cisco hardware. In essence, the ESX Server host replaces your access layer of switches and cuts down on physical port count, both good things from a network perspective.
Also my post above has a white paper direct from Cisco that talks about tagging. Here's my opinion on network guys when it comes to virtualization. There are alot of unusual concepts in a VMware environment that many network guys are not used to. It's always a challenge trying to get them to change there ways. It really helps to sit down with them and make them understand the unique networking features of Vmware (the concept of Vswitches, Vnics, Vmotion, Vlan tagging, Port groups, etc.). Once they understand more about it they tend to put up less resistance when you come to them with requests.
I am trying to setup a VST type of trunking with 2 virtual servers. I have a trunk coming in on a physical interface, which is fed into a virtual switch with proper vlan groups set. Each of the guests gets its own vlan from the vswitch. I dont know if it is a bug, but the two guests can not seem to receive decapsulated traffic at the same time. It seems that whichever guest comes up first grabs its vlan and the second server does not receive any traffic at all, tcpdump shows zip on the last one to come up.
Would anyone have any suggestions as to why this can be happening?
... Instead they want to assign the VLAN id to the physical ports themselves.
This is a fairly common response from "the network team". In a "traditional" network environment, VLAN tagging is used only between switches. They are accustomed to dealing with "regular" physical servers where you may have one or two VLANs connected. They don't understand that when you connect to an ESX server, you are actually connecting to another switch. The pNICs on an ESX server are effectively the uplink ports on a layer 2 switch (and before the network folks get excited on another topic...the vSwitch does not participate in spanning tree ). It is common - in fact it is best practice - to use 802.1Q VLAN trunking to an ESX server.
Technical Director, Virtualization
VMware Communities User Moderator