Vlan configuration with Trunks

[5474]

Hi,

We've got ESX 3.0.1 running on two HP 465 blades that tie into two c-Class GbE2c switches. Up till now I've only had one network (Vlan 90) running in ESX on vSwitch0 which also holds the SC vswif0.

I now want to tie in another VLAN (10) to that switch on vSwitch1 which currently has the VMKernal (vmotion etc)

I add the uplink ports on the GbE2c to the trunk group 2 and enable it. I've also switched the port on the network core to Trunk and added those two vlan's to it.

Then I've added the VLAN tags to the two vswitches. I'm getting responses anywhere from the whole ESX server dropping off line or I can get the server and console in but I can't ping in or out of the servers.

Any suggestions as to where I'm going wrong?

[bggb29]

Try running this command and verify what trunking information that you are seeing from the switch

esxcfg-info -n | grep -E -i "_name|Hint"

the out put will show you what trunks are seen on a nic by nic basis, then compare what the nics see to what you have added to the vswitch.

What vlan besides 10 are you running on vswitch1.

[5474]

Right now there isn't another VLAN on that switch. I guess I could tag the actual NIC through hardware and do it that way, but from reading the VMware PDF I got the impression that tagging on the vswitch was the way to go.

[VirtualNoitall]

Hello,

tagging on the vswitch is the way to go.

Here are a couple of links that might help:

http://www.vmware.com/pdf/esx3_vlan_wp.pdf

http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

Message was edited by:

VirtualNoitall

[bggb29]

Are you planning on adding other vlans to the vswitch, Yes tagging is best if you will add aditional vlans.

Otherwise take the vlan portgroup out of the vswitch and let it connect to the pswitch without tagging.

All my vswitches with the exception of vswfo are tagged.

We use one vswitch for dmz's and one vswitch for internal traffic

If you do not need the complexity of tagging and have enough pnics to handle your subnets now and into the forseeable future then leave it off.

[5474]

Unfortunately on the c-class blades we have 16 internal NICs and they are connected to 5 uplink ports. To maximize bandwith I want to trunk several of these and let the router handle passing them off to the correct vlan

[bggb29]

In that case have you had the oppurtunity to run the esxcfg command from my first post to see exactly what vlans your nics are seeing upstream from the pswitch.

Do you see vlan 10 on the vswitch you expect to see it on

[5474]

Command just seems to hang, no output is generated

[mikeddib]

Based on some best practices and info from VMWare on ESX networking I would think you would want to separate the VMotion and COS NICs from your VM NICs. It should still work with the trunking but 5 uplinks x 2 switches should be plenty of throughput to alleviate IO concerns. It also depends I guess on if the blade chassis is used exclusively for ESX or has other blades in it for other purposes.

I would think per switch you would want one uplink for COS, one for VMotion, and use the other three as trunks. No tagging or portgroups necessary for COS vSwitch and VMotion vSwitch (keeps the config simple) and then tag your portgroups on your VM vSwitch with the trunked uplinks. To see where you are now I would use the esxcfg-vswitch -l command. This should list out all our virtual switches, the uplinks connected to them, and the portgroups.

Are those the Cisco or HP interconnects for the switches? I remember HP terminology of vlan tagging being a little more confusing coming from the Cisco world.

[TiBoReR · Accepted solution]

I have a setup like yours.

I didn't go the same way as you did. I teamed my 2 host nics. I created 1 vSwitch and I created multiple VM port groups with different VLAN in it with VMotion port group and SC port group.

On my blade ethernet switchs, I put all my internal ports to VLAN tagging enabled and PVID tagging enabled with my default vlan id. I created all VLANs and assigned all VLANs to my internal ports and my trunk ports (i think you don't need help for trunked ports, if so, let me know)

I hope it's clear, if not, ask me questions..

I did that configuration on HP blade systems and IBM bladecenter and it worked like a charm.

I would have prefer to separe SC, VMotion and VMs but with only 2 NIC, it's difficult...

Hi,

We've got ESX 3.0.1 running on two HP 465 blades that

tie into two c-Class GbE2c switches. Up till now

I've only had one network (Vlan 90) running in ESX on

vSwitch0 which also holds the SC vswif0.

I now want to tie in another VLAN (10) to that switch

on vSwitch1 which currently has the VMKernal (vmotion

etc)

I add the uplink ports on the GbE2c to the trunk

group 2 and enable it. I've also switched the port

on the network core to Trunk and added those two

vlan's to it.

Then I've added the VLAN tags to the two vswitches.

I'm getting responses anywhere from the whole ESX

server dropping off line or I can get the server and

console in but I can't ping in or out of the

servers.

Any suggestions as to where I'm going wrong?

[5474]

They are HP interconnects and getting used to them in a PITA. Basically want Port 1 on Blade one to go to Trunk Group 2 Ports 20 and 21. Port 2 goes to Trunk group 3 Ports 22 and 23. Both Trunk groups should have access to VLAN 10 and 90.

Things are duplicated over on the second interconnect. Then I should be able to team the NIC inside ESX and have 2GB through each interconnnect. But I'm getting all messed up with vswitch tagging, PVID tagging and Vlan tagging 😕

Did you have to set VLAN ID's on the VM and SC port groups?

[bggb29]

We do not tag the sc port groups or the vmotion port group.

Also sent you a PM showing what we see when running the esxcfg with the hints.

The output displays the vmnic and the subnets as well as vlans on a per vmnic basis

[5474]

Did a bunch of work today. Basically Ports 1 and 2 on the interconnect were put to vlan tagging but PVID tagging off. Default PVID set to 90.

Uplink ports set to tagging and Pvid tagging. Also on VLAN 90 and trunked together.

Trunked the ports on the core switch and added the VLANs.

We founds that the VLAN 90 vswitch didn't need the VLAN added to it. Only VLAN other then 90.

So now I've got the two physical NICS combined into one vSwitch. It's got two Portgroups for Servers, the SC and VMKernal. I know it's not the best configuration but with the limited NICs and them being Blades it's the best we can do for now. I'm looking forward to building the production boxes so we can play configure it with Physical NICs and switch ports.

Thanks to everyone for helping out!