VLAN Tagging and vSwitches

[techcontact]

I have a blade server with 6 NICS. 2 of the NICS are used for management and the other 4 are going to be used for networking traffic. I have two vSwitches. 0 and 1. 0 has the service console on it. On the other switch I am trying to get multiple VLANs configured. It doesnt seem to be working

I have 3 machines on vSwitch one. Each one is on a differnet network. We will call them 1,2,3.

The server on 1 network is fine. Default vlan no big deal. The server on network 2 or 3 cannot get on the network. If I create new vSwtich for each network and assign a NIC to each one they can get on.

What am I missing?

[cmacmillan]

Please issue and post the following if your Cisco-HP interfaces are in question:

Cisco:

1. show interface Port-channel 3 etherchannel

It should show all members in LACP bound trunk. You could ALSO configure the ports in the trunk group to use STATIC trunking (IMHO, better) if it is configured on each side:

Cisco (cli):

(config) # interface GiX/Y

(config-if) # no channel-protocol lacp

(config-if) # channel-group 3 mode on

(config-if) #

HP (web GUI):

Configure each port for the same trunk group

Configure the trunk group as "Static" not LACP

This will need to be the case with the trunks and trunk ports connected to the ESX server as well, as ESX does not support any form of automatic trunking (unless using v1000). Seems consistent to do so end-to-end.... These trunk groups must be configured on the chassis switch as well, one group per ESX host/blade (again, static trunking). This assumes "IP Hash" balancing enabled on the trunked vSwitch(es)...

That said, using 802.3ad aggregated trunks for ESX is not ALWAYS the best solution, as it could affect how fail-over ports work. The simplest form of trunking is to let ESX use "Port ID" (default) balancing - only 802.1q encapsulation and VLAN tagging designations are needed.

Note TRUNKING and PORT AGGREGATION are often thrown around as the same term but they are not: trunking implies multiple VLAN (tagging) across the interface while port aggregation defines the use of 2 or more ports as a single logical trunk. ESX either supports STATIC 802.3ad aggregated trunks or multiple trunks (1 or more VLANs) logically combined in round-robin allocated, pseudo-static port groups (i.e. VM is designated to use only a single port in the aggregate, changing only on power-on or port failure).

It is much simpler to create the "Port ID" based trunk groups and vSwitch(es) and, confirm proper operation and fail-over/take-over; then convert to "IP Hash" and static 802.3ad trunk groups. Likewise, on the switch-to-switch interface, the most manageable protocol should be used. If there is ANY errata suggesting problems with LACP between your two switch types, USE STATIC. Otherwise, LACP should be working if BOTH switches show active LACP groups with the configured ports enabled.

Also, the comment about "allowed vlan 1,3000" is correct - only traffic on VLAN 1 and 3000 will be allowed on that trunk group (from the Cisco). Likewise allowances need to be configured on the HP switch and blade switch.

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

VMware vExpert 2010

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

I think I follow.

Please bear with my questions.

First we have made some changes since the inital post. We are currently using LACP, but we dont have too. This was done because this is how I read somewhere how to do it.

Here is my current Cisco config.

We have 4 Port Channels with Gig ports in each channel. One going to each HP Switch. I have included only one Port Channel and 1 Interface for example

interface Port-channel3

description LACP_HP-C7000-InterConnect3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

interface GigabitEthernet2/33

description HP-C7000-InterConnect3_LINK1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

channel-protocol lacp

channel-group 3 mode active

Here is the output as you requested. .

HQ-SW-4506-01#show interface Port-channel 3 etherchannel

Port-channel3 (Primary aggregator)

Age of the Port-channel = 00d:04h:39m:33s

Logical slot/port = 11/3 Number of ports = 2

Port state = Port-channel Ag-Inuse

Protocol = LACP

Ports in the Port-channel:

Index Load Port EC state No of bits

-

---

----------+

---

+--

---

0 00 Gi2/33 Active 0

1 00 Gi3/39 Active 0

Time since last port bundled: 00d:04h:36m:12s Gi3/39

So i am going to follow your advice and use Static.

We have changed it from allwoing vlans 1,3000 to allow a range of 1-100. Once we get this working we will be more selective.

I will attempt to make this changes and post back

[techcontact]

cmacmillan, One question for you.

Should I change both the Gig ports and the Port Channel to no LACP. Or only the Gig Ports?

Cancel that. request. LACP is no setup ad the channel level. My mistake

[cmacmillan]

cmacmillan, One question for you.

Should I change both the Gig ports and the Port Channel to no LACP. Or only the Gig Ports?

Cancel that. request. LACP is no setup ad the channel level. My mistake

Right (not needed at the Po config level, switchport and vlan configs ARE). You'll see a problem building the Port-Channel (show log) if your interface configuration does not match the port-channel config. It will NOT warn you at the config prompt...

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

Having some problems with the commands

(config) # interface 2/33

(config-if) # no channel-protocol lacp

(config-if) # channel-group 3 mode on

(config-if) #

If then do a show run and look at the interface no change has been made.

So I tried

(config) # interface 2/33

(config-if) #no channel-group 3

(config-if) # channel-group 3 mode on

(config-if) #

this removed the channel group but did not put it back. Can you have a channel group WITHOUT a protocol?

[cmacmillan]

(config) # interface 2/33

(config-if) #no channel-group 3

(config-if) # channel-group 3 mode on

(config-if) #

this removed the channel group but did not put it back. Can you have a channel group WITHOUT a protocol?

This is required to change the group mode of the interface. The protocol speaks to the auto/negotiated trunking applied by the "mode" of the interface group. Since mode is "on" there is no negotiation on the trunk - the "no" version of your LACP could have been "default channel-protocol" as well... either should result in PAgP as the default.

(config) # interface X/Y

(config-if) #no channel-group 3

(config-if) # default channel-protocol

(config-if) # channel-group 3 mode on

This MUST be done on all channel interfaces on both sides of the link to work properly, however, the Cisco will show the static trunk as up as soon as all interfaces are set to "mode on".

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

I think we are going to have to use LACP. I set it to PAGP and it doesnt work. I did some quick checking and found this

"The channel-group mode desirable will not work between a Cisco and GBE2c switch, as desirable mode uses PAgP and the GBE2c switch does not respond to PAgP?

It appears the Cisco Blade switch doesnt support PAGP.

Just to be clear we have a Cisco 4506 connecting to a clade chasses with 4 switches in it. Each port channel on the Cisco connects to 1 blade switch on the chassis.

[cmacmillan]

It "should not" matter which AUTO trunk protocol is configured if "mode on" is selected - that ignores auto trunking (LACP, PAgP, etc.) Set the auto type to whichever you'd most likely use... However, the STATIC or "mode on" configuration must be applied on BOTH sides of the trunk and on ALL trunk members for the trunks to properly operate.

Considering the VMware ESX vSwitch side, it is either:

1. the equivalent of STATIC (with "IP Hash" on the vSwitch) connected to a STATIC/"mode on" trunk group on the physical switch side, or

2. it is "Port ID" on the vSwitch and NO port-group on the physical switch side

For switch-to-switch, if LACP is working, great, but if you're having difficulty with STATIC on switch-to-switch, that could spell a problem on switch-to-vswitch down the road...

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

I agree on the switch to switch to vswitch problem.

default channel-protocol wants a protocol

(config-if)# default channel-protocol

% Incomplete command.

If I choose pagp the trunk wont come up on the Cisco side. It shows up on the HP Side, but I dont think its really up. I think once you put ports into a trunk group they show as up.

So if I use LACP for Cisco to HP

and I use static from the HP to the ESX that is what is causing my problem?

[techcontact]

I have the static working from the Cisco side. I apologize for the confusion.

Testing now

[cmacmillan]

OK. So, as I understand, your switch topology looks something like this:

Cisco4000(core)<->HPSwitch(es)<-->HPBladeSwitch(GbE2c)

If so, the blade switch ports connected to the compute blade need to be configured for the vSwitch type (or vice versa). if it's HP GbE2c then you're configuring the STATIC ports just like the Cisco trunks with a second trunk group. Do this for each ESX blade with a separate STATIC trunk group for each aggregated vSwitch on each ESX blade.

If you want to use port aggregation (static, just like the physical switches) then your matching vSwitch Load Balancing policy is "Route based on IP Hash." If you want to use the default load balancing policy, it is "Route based on the originating Port ID."


-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

Close there are no HP switches in the middle

Cisco4000(core)<-->HPBladeSwitch(GbE2c)

Our goal is to build a vSwitch and allow x number of VLANs to connect using port groups. Each HPBladeSwitch(GbE2c) has four ports that we are "uplinking" (not sure if thats the right term to use) to the Cisco.

There are 4 HPBladeSwitch(GbE2c) so we have 16 cables running from the HP switches to the Cisco. I have 4 port channels on the Cisco each containg 4 Gigabit connection.

I currently am trying to use Static (non lacp) channel groups, but it doesnt appear to want to come up on the Cisco side.

interface Port-channel3

description LACP_HP-C7000-InterConnect3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

interface GigabitEthernet2/33

description HP-C7000-InterConnect3_LINK1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

channel-group 3 mode active

Port-channel3 is down, line protocol is down (notconnect)

On the HP Side the trunks are manually configured not using LACP

Trunk Group 2 is my connection to an ESX Host. Trunk Group 3 are the "uplinks" to the Cisco

Trunk Group 2

Protocol - Static

Current settings: enabled

ports: 4

Current IP trunk hash settings:

sip dip

Trunk Group 3

Protocol - Static

Current settings: enabled

ports: 21, 22, 23, 24

Current IP trunk hash settings:

sip dip

If I use LACP I can get the trunk to come up to the HP. I dont know how to verify if the trunk is up between HP and ESX, but it doesnt sound like LACP is the way to go, so I am going to move forward trying to avoid it.

Message was edited by: techcontact

[cmacmillan]

interface GigabitEthernet2/33

description HP-C7000-InterConnect3_LINK1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

channel-group 3 mode active *****

Port-channel3 is down, line protocol is down (notconnect)

The line marked "*****" is the problem, it should read:

channel-group 4 mode on

If that is NOT an available option, you're forced to use LACP. It would be a weird IOS not to have it...

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

Ok we have success with the Cisco>HP connection. Now onto the HP to ESX connection. Do the VLANs have to be configured in the HP Switches? I wouldnt think so because I dont want to assign a vlan to a specific port in the HP. Or should the VLANs be setup, just not assigned to a port so it can see the vlanid and pass it through

[cmacmillan]

Ok we have success with the Cisco>HP connection. Now onto the HP to ESX connection. Do the VLANs have to be configured in the HP Switches? I wouldnt think so because I dont want to assign a vlan to a specific port in the HP. Or should the VLANs be setup, just not assigned to a port so it can see the vlanid and pass it through

The VLANs must be set as tagged on each trunk group that will pass VLANs to the vSwitch. For the HP switches I've configured, I've had to explicitly define the tagged VLANs on each trunk (not port). The PVID of the trunk is equivalent to the "default VLAN" in the Cisco world and is usually VLAN 1. This corresponds to the "default" VLAN in the vSwitch (when VLAN ID is "none").

For instance, if your CSO does not want VLAN1 ever to be seen by your VMware stack, you'd use a PVID on the HP switch of something other than VLAN1 and the "default" VLAN for the VMware vSwitch would correspond to that PVID VLAN.

The HP switch normally does not allow trunk assigned ports to be modified with respect to VLAN membership...

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[techcontact]

I certainly appreciate all the help. Ill see if i can find some help on the HP Side. It appears you can only set up tagging at the port level. Currently both vlan tagging and port tagging are enabled. the PVID of each port is set to 1.

Thanks again for your assitance.

[cmacmillan]

If you found the information that any of your peers - including me - provided help in resolving your issue, please circle around and award points to them,

Thanks!

-- Collin C. MacMillan, VCP4

Cisco CCNA/CCNP, Nexenta CNE

VMware vExpert 2010

SOLORI - Solution Oriented, LLC

http://blog.solori.net

If you find this information useful, please award points for "correct" or "helpful".

[rogard]

That is correct, once you create a Trunk on the HP switches you need to and the (for example) Trk1 interface to the correct VLANs

[techcontact]

I finally got it working. For anyone else needing to connect a Cisco switch to an HP C7000 or 3000 with blade servers this is how you do it.

We are using VST mode

Setup Channel Ports on the Cisco side (repeat this for each switch in your chassis.)

interface Port-channel3

description Connection to gbe2c switch1

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100 (this could be a handful of vlans such as 1,5,9 or a range) These are the vlan that your guest vms are going to be on

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

Set up the ethernet ports and repeat this for each port you plan on using on the chassis switches. We uised all 4

interface GigabitEthernet2/33

description Connection to gbe2c switch1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-100 (this could be a handful of vlans such as 1,5,9 or a range) These are the vlan that your guest vms are going to be on

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

channel-group 3 mode on

HP Switch Config

Create one trunk group and put ports 21-24 in it (these are the ports that connect back to the CIsco)dont use LACP

On the ports as the previous step (Most likley ports 21-24)

Make sure you have VLAN Tagging and PVID Tagging enabled

We left the PVID at 1 (This needs to be set correctly for your environment, many times 1 will work fine)

On the switch ports that are connecting to your blade servers

VLAN Tagging Enabled

PVID Tagging Disabled

Create all the VLANs for the guest machines and make sure the correct switch ports are in the VLAN. It will be ports 21-24 along with the ports that your servers are in. So if your server is in slot 5, make sure you include port 5

vSphere

Create a vSwitch with a port group per VLAN and add your machines as needed.

It should work from end to end.

Good Luck