VMware Cloud Community
atbnet
Expert
Expert
Jump to solution

ESX 3.5 Cisco Trunking 802.1q and Link Aggregation 802.3ad - Problem

Hi I'm hoping someone here can help us, maybe that has done this before.

We currently have two ESX 3.5 (U2) servers both have 4 gigabit network interfaces each. There are two virtual switches with 2 nics each, each nic links to a different cisco core switch (4506) for failover:

vSwitch0 = Virtual Machine Port Group (VLAN 70) and Service Console1(VLAN none)

vSwitch1 = Service Console2(VLAN 1) and VMkernel (VMotion) (VLAN 170)

1. We want trunk ports to all ESX network interfaces so we can seperate various traffic to different VLANs (802.1q).

2. We want to load balance using the ip hash to improve bandwidth usage and increase througput (802.3ad).

What we have tried so far is setting all the ports to trunk ports and assigning the correct vlan and gatway for the service console2, but I cannot reach the IP of the service console2 (all vlan routing is correctly configured). Any ideas?

I have confirmed the correct gateway for the new service console VLAN 1 is set on the ESX servers with esxcfg-route and esxcfg-vswitch -l

I have the original Service Console1 still set with no VLAN so we can remove the trunk ports and get back to a working state.

We tried setting up the link aggregation incase that was why it was not working but no luck there.

Here is an example of the cisco config for these one of the ports:

interface port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

!

exit

!

interface GigabitEthernet5/1

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

channel-group 2 mode on

!

?:|

Thanks,

Andy, VCP,

[

|http://www.vmadmin.co.uk/]

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos
1 Solution

Accepted Solutions
Dean_Holland
Enthusiast
Enthusiast
Jump to solution

Silly question(s) - are the VLAN's in the Cisco's VLAN database, and have you tried specifying the VLANs to trunk with "switchport trunk allowed vlan xx" ?

Edit: just noticed you are using VLAN 1, this is usually reserved for management VLAN and can't be tagged on Cisco devices. Change that service console's VLAN to something else and see if it works.

View solution in original post

Reply
0 Kudos
11 Replies
Texiwill
Leadership
Leadership
Jump to solution

Hello,

One of the things I would do first is not have multiple Service Console Portgroups. That is not necessary. I would do the following:

vSwitch0 -> SC/VMotion (requires VLANs)

vSwitch1 -> VM Network

Having a SC on your VM network vSwitch raises a security risk as your network attack surface for the SC has now doubled. As for the 802.3ad I will leave that to others.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Derekkk
Contributor
Contributor
Jump to solution

Hi,

I will suggest following:

2 x NIC on vswitch0 for console and vkernel. 2 x NIC on vswitch1 for production vm.

no need to configure port channel group on realted switch.

vswtich0 can just setup one access vlan (VLAN 10) on cisco switch port and configure console and vkernel network on ESX (no need assign VLAN ID).

vswitch1 configrue trunk port on cisco switch port and configure multiple network with VLAN ID on ESX.

I have setup following configure and work fine.

16 x ESX server on HP Blade with 4 x blade cisco switch. Each server has 6 x NIC. I stack all cisco switch into one (call virtual switch tech). Then create port channel group 1 (for console and vkernel traffic go to core and use 4 external ports), group 2 (for production VM trunk traffic go to core and use 8 external ports). Then cabling them to core switch

vswitch0 = NIC1 (SW-P1), NIC2(SW-P2) -- Console (192.168.1.0) (no VLAN ID), Vkernel (192.168.2.0) (no VLAN ID)

vswitch1 = NIC3 (SW-P3), NIC4(SW-P4), NIC5 (SW-P5), NIC6(SW-P6), (VLAN ID101, 102, 103, 104, 105, 106)

External port (SW-P17 ~ SW-20)

On blade switch, for internal port, just setup access vlan and trunk port on realted switch port.

For external port, setup channel group 1 and 2 and cabling them to core switch.

For failover fine tuning, I use beacom probing instead of link status on NIC teaming which get good result.

Hope this can give you idea.

Regards,

Derek

Reply
0 Kudos
atbnet
Expert
Expert
Jump to solution

Hi Edward.

The only reason I have two SCs is while I migrate from no use of vlans and trunk ports, to the use of vlans and trunk ports. So if the trunk port and vlans do not work, like at the moment, I can use the original service console to reconnect.

Basically I have the vSwitch setup you recommend just with the extra SC2, which will be deleted after. I understand the security risks hence my plan to seperate everything onto different VLANs/vSwitches, specifially SC and VMotion traffic.

I might try installing one of the ESX servers from scratch and when giving the SC an IP set the VLAN ID also. That way it rules out any issues with the current ESX network config.

Is there anyone can see any issues with the Cisco config that might be causing a problem, anything we missed out?

The idea was to migrate to the trunk ports one ESX at a time as to reduce downtime of the few production VMs running.

Thanks,

Andy, VCP,

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos
atbnet
Expert
Expert
Jump to solution

Hi Derek,

Thats what ive got really just the other way round vSwitch0/1. The first SC will be deleted once I migrate.

Do you have an example of your channel and port config on your core switch just so I can compare with ours? Are you using 802.3ad link aggregation?

Thanks,

Andy, VCP,

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos
Rumple
Virtuoso
Virtuoso
Jump to solution

one thing i see missing from the config is the setting of a Native VLAN that does not exist, or a native vlan that will not be configured on the portgroups.

If you do not add in a native vlan thats not on any portgroups then nothing will be tagged and probably not work...

Reply
0 Kudos
Dean_Holland
Enthusiast
Enthusiast
Jump to solution

Silly question(s) - are the VLAN's in the Cisco's VLAN database, and have you tried specifying the VLANs to trunk with "switchport trunk allowed vlan xx" ?

Edit: just noticed you are using VLAN 1, this is usually reserved for management VLAN and can't be tagged on Cisco devices. Change that service console's VLAN to something else and see if it works.

Reply
0 Kudos
Smiddie
Enthusiast
Enthusiast
Jump to solution

In this case it is indeed the native VLAN that is causing the trouble. Changing that should fix it. It is also best practise to not use VLAN1

Regards,

Raymond

Regards, Raymond
atbnet
Expert
Expert
Jump to solution

Hi,

I went on the service console and removed all the SC port groups and vswifs etc and set them up from scratch. When I did this I used the VLAN I created for Vmotion rather than VLAN1 and it worked. It must have been the the fact that VLAN1 is not tagged that caused the problem.

As for the load balancing using 802.3ad and ip hash accross two physical switches also appears to be working, however yet to test pulling cables out. Smiley Happy Looking over my old student books for VCP 3.5 it says you cant ip hash over two physical switches and then says most switches!

Now just got to vmotion all the VMs to the new trunk config ESX server and setup the other. However im getting "operation timed out" for the motion. Each ESX ca see each other vmkernel (vmotion) ip. Any ideas?

Cheers,

Andy, VCP,

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

I would make vSwitch0 the one with the SC/Vmotion with 2 pNICS and vSwitch1 with 2 PNICs for VMs. This way you do not need to have SC2 at all and the primary vSwitch0/pNIC0 combination wins out every time. CHanging this is not something I would normally do.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
atbnet
Expert
Expert
Jump to solution

I got the service console sorted, I also just have the one SC now. This is how it looks....

Each vSwitch has 2 x physical gigabit nics, each nic goes to a different core switch.

vSwitch0 = For Virtual Machines(currently one VM port group assigned a VLAN)

vSwitch1 = Service Console and VMotion (both now using vlans)

I also tested pulling out cables briefly simulating a failure, only one ping is lost while it flicks to the other physcial nic/switch.

The service console command line is easy, especially if your familliar with red hat like me :smileygrin: .

Bascially dont use VLAN1 with ESX it wont work. As for the migrating of the VMs so I can update the second ESX servers network config, we have arranged a few mins downtime to cold migrate them while I update the SC and VMotion settings.

Thanks,

Andy, VCP,

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos
atbnet
Expert
Expert
Jump to solution

For reference I have noted down what to do to set this up:

802.1q Trunking and 802.3ad Link Aggregation using Cisco Switches

Andy, VMware Certified Professional (VCP),

If you found this information useful please award points using the buttons at the top of the page accordingly.

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
Reply
0 Kudos