VLANs and vSwitches on non-Cisco gear (SMC 8024)

[DavidYonRFD]

I've got a set of ESXi4 nodes managed by vSphere Essentials.  Each one has a number of NIC's, all connected with an SMC 8024 managed switch.

I've been able to use statically-configured VLANs defined in the SMC 8024 to segment off my iSCSI traffic from the other traffic.  Works great.

But what I want to do now is start configuring some vSwitches that are tied together using VLAN tagging.  I.e., have two of the ESX nodes with several vSwitches, each with corresponding VLAN id's, and have the VLANs tunnel through the SMC using a single NIC in the ESX host.  Right now I've gotten by with partitioning things based on vSwitches tied to individual physical NIC's, but I'd like to have multiple VLANs tied to a single NIC and have It All Just Work(tm).

I know this is possible---I have an account up on a VMWare-based hosting provider, and they do things this way.

The examples I've seen in the past all assume Cisco backbones and therefore use Cisco-speak when discussing how to set this up.  But this is a smaller development/test rig that can't justify the expense of big-iron switching gear.  The SMC works just fine, is much more affordable, and from what I can tell should be able to handle this type of setup.

Any help out there?

[Josh26]

With cheaper switches you will generally be shooting yourself in the foot indefinitely trying to make things work.

If you cannot justify Cisco, look into HP Procurve, or "Linksys by Cisco", which is becoming the Cisco Small Business range.

[DavidYonRFD]

Well, "Just Buy Cisco" doesn't really get me any closer to understanding the disconnect I'm having when I attempt to tie multiple vSwitches on multiple ESX hosts using VLAN tagging.  There are three levels where things can go wrong, and it's often difficult with Layer-2 switching to understand which part of the chain is the brick wall:

• The vSwitches and their VLAN configuration
• The physical NIC, and whether it supports VLAN tagging in this manner
• The physical switch's configuration at the attached ports

The SMC seems to do everything I would need.  It has "Trunking" (not sure what the Cisco/HP terminology is, which is part of the problem in deciphering other posts).  You can restrict traffic on each port to one or more VLAN tags.  You can configure the port to strip off VLAN tags in inbound and/or outbound packets.  You can configure the port to leave the VLAN tags unchanged.  You can configure the port to add a VLAN tag if and only if an inbound packet is untagged.

At the moment I have two SMC's tied together using port trunking, a total of 2GBps of pipe between the two of them.  That was relatively straightforward.  For now I would be happy to get VLAN-based vSwitches working on a single switch, ideally I should be able to have them span the two switches.

What I'm having trouble with is when I tell the SMC to block/relay based on VLAN tagged packets, but to rely on the external device (in this case the ESX Hosts's NIC) to tag the traffic.  I've beat on this several times to no avail.  But the trunking of the two switches seems to accomplish much the same thing, so I know that works.

My first guess is that there may be some NIC chipsets that will transparently pass along tagged packets, and others that won't.  The ESX whiteboxes have built-in Broadcom NIC's as well as a number of Intel-based NIC's.

At this point I'd be happy to hear someone suggest a methodology by which I can narrow down the problem to a single area of misconfiguration.

[malaysiavm]

I think the terms you use here is quite confusing. When we say VLAN trunking, it meant you configure and allow multiple VLAN access to a single physical network port on the switch. When you combining more than 1 physical port on a switch for aggregation, it should define as Link Aggregation, or LACP, LAG or Cisco call it etherchannel. In normal scenerio, if you would like to perform the link aggregation cross different switch, you need to ensure this functionality is supported by the switch you are using. As my understanding, not every switch will support this feature, as most of them only support LACP or aggregation within the same switch to provide bandwidth more then 1G in gigabits environment. In VMWare, you can shoose the right setting while you configure differently on the NIC teaming policy. Choose the right policy which reflect to the configuration you have is very important.

[DavidYonRFD]

Well, I'm not too concerned about inter-switch propagation of the VLAN setup, doing it in a single switch would be sufficient.  Would be nice to have eventually, but baby steps for now.

As for what SMC means by "Trunking", they don't really say or refer to a standard.  It's probably a red herring for what I want to do.

The two switches are tied together with two ports on each side, aggregated using LACP.  Probably orthoganol to what I'm trying to accomplish with extending VLANs from multiple vSwitches through a single SMC 8024 switch.  But it does work.

As for VLAN on each port, the standard referenced is 802.1Q.  The SMC help page for the VLAN Ports page is below.  Nominally I have everything set to VLAN Aware, a Packet Type of "All" and a PVID of whichever VLAN I wish the port to be a member.  That works fine and statically defines VLANs on a per-port basis, with each port participating in a single VLAN.  Hardware connected to the port is blissfully unaware of the VLAN it's on.

The missing secret sauce is how I can:

1. Get a group of ports on the SMC Switch to only pass traffic between each other
2. Get the ESX vSwitch to add VLAN tags.
3. Convince the SMC Switch to preserve any VLAN tagging that the ESX vSwitch may have added
4. Convince the physical NIC on the ESX vSwitch to pass along the VLAN tagging to the SMC Switch

That shouldn't be too hard, right? :smileyconfused:

For your reading pleasure, below is the SMC description of the VLAN port configuration.   Apologies for the length.  Even if someone can map the terminology used by SMC to what HP/Cisco does might help me to decipher what others have successfully done in more expensive deployments.

Thanks much!

This page allows you to config the VLAN settings per port. Each row of the table corresponds to one port or trunk;  trunked ports cannot be configured individually. The columns of the table are used as follows:  Port/Trunk - The front-panel port-number of the port or the ID of a trunk. This cannot be changed. VLAN Aware - Set and show the VLAN awareness mode for port. QinQ - Set if multiple vlan tags can be accepted. See QinQ application example Packet Type - Set a port's handling of tagged and untagged packets. PVID - Set the Port VLAN ID.

VLAN Awareness

VLAN aware ports will strip the VLAN tag from received frames and insert the tag in transmitted frames (except PVID). VLAN unaware ports will not strip the tag from received frames or insert the tag in transmitted frames. For QinQ application, customer port should be VLAN unaware and network port (trunk port) should be VLAN aware.

QinQ

QinQ enabled port will accept packets up to 1526 bytes in length, which means double tag header frames can be accepted. For QinQ application, the QinQ should be enabled for provider port but not for customer port. Note: For QinQ application, customer ports indicate those ports which are connected to normal VLAN aware switches at  the customer network and the network ports are those which are connected to the service provider network. To tunnel the packets  through MAN, QinQ needs to be enabled on network ports.

Packet Type

PCs should be connected to ports with Packet Type set to All. PCs cannot, in general, send or receive tagged packets. Switches should be connected to each other with Packet Type set to Tagged Only. If the Packet Type is set to All, the port can accept incoming tagged and untagged packets. Untagged packets will be associated with the VLAN identified by the PVID. Tagged packets will be dropped unless the port is a member of the VLAN identified by the VLAN tag in the packet. Outgoing packets will be tagged unless the packet's VLAN ID is the same as the     PVID. If the Packet Type is set to Tagged Only, the port will drop untagged packets and will only send and receive tagged packets. Tagged packets will be dropped unless the port is a member of the VLAN identified by the VLAN tag in the packet. The PVID has no effect in this case.

Port VLAN ID

The PVID is (Port VLAN ID) is the VLAN ID that is associated with untagged, ingress packets. It is not possible to remove a port from VLAN 1 unless its PVID has been changed to something other than 1. The PVID has no effect on ports that have Packet Type set to Tagged Only. When the PVID is set to None, all outgoing frames are tagged Untagged frames received on the port will be classified to this VLAN ID. Frames classified to this VLAN ID will be sent untagged on the port. None is used for trunk port. For QinQ application, the PVID for each customer port must be unique.

QinQ Application Example

This example shows how to configure one network port and three customer ports with the following setup on this switch. Port 1 as network port and port 2-4 are customer ports.  Click off VLAN aware enabled checkboxes for port 2-4 and click VLAN aware enabled checkbox for port 1. Clear the QinQ enabled checkboxes for port 2-4 and click QinQ enabled checkbox for port 1. Type PVID 2,3,4 individually for port 2-4. The PVID for each customer port must be unique. In VLAN membership page, add VLAN group 2 with port 1 and port 2 as members.  Add VLAN group 3 with port 1 and port 3 as members. Add VLAN group 4 with port 1 and port 4 as members.