VMware Cloud Community
smonborgne
Enthusiast
Enthusiast
Jump to solution

VLAN tagging during installation phase

Hi,

I'm using VLAN tagging on our VMWare hosts to be able to send traffic to the right VLAN for each VM we use. To do so, my network engineers configure the ports on the switch on which we connect the esx hosts in trunk with the requested VLAN IDs.

However, during the installation phase I'm not able to set a VLAN ID on the network interface and I'm unable to perform a network install.

For the moment, the only workaround I have is to ask them to configure the ports on my management network VLAN during the installation phase and to change the setting to trunk when esx is installed. But it's not very efficient because it requires 2 interventions to make it usable and I also have to change the service console parameter.

I read some posts refering to native VLAN but i'm not sure it is the way to go.

Can someone explain me what I need to do to be able to communicate with VLAN during this installation phase and in production phase without having to mess with the network equipements settings several time?

Thanks for your help.

Sylvain

Tags (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
dinny
Expert
Expert
Jump to solution

Hiya,

As Jae and distorted say - Native Vlans are definitely the way to go - but that does not necessarily need to be the default vlan.

If you are using cisco kit - then you can use whatever vlan you like as the Native VLAN.

Configuring the cisco port for the service console like this will do the trick:

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 77

switchport trunk allowed vlan 72,77,95

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

In the example above PXE can happily initially take place on VLAN 77 (make sure an IP helper statement on that vlan directs PXE packets to wherever your PXE/dhcp servers are)

Then you can amend the SC portgroup to use vlan 72 via your kickstart script (setting it to run in rc.local)

vlan 95 can then be used (in this case) as a backup connection for vmotion.

Doubt you'll find anything documented on it by VMware - though there are a few posts on the forums - but its all cisco stuff really - should mean something to your network guys anyway.

Cheers

Dinny

View solution in original post

Reply
0 Kudos
17 Replies
RParker
Immortal
Immortal
Jump to solution

> I'm using VLAN tagging on our VMWare hosts to be able to send traffic to the right VLAN for each VM we use. To do so, my network engineers configure the ports on the switch on which we connect the esx hosts in trunk with the requested VLAN IDs.

Read the documentation for VLANS, you can't setup physical VLANs on the Physical switch, you MUST use VLANs on ESX, ONLY. Otherwise they won't work.

Reply
0 Kudos
stvkpln
Virtuoso
Virtuoso
Jump to solution

Actually, that's very very very untrue. If you make use of the native VLAN configuration, you can achieve the tagged requirement for multiple VLAN's while having an untagged VLAN for the Anaconda portion of the VI3 installation. What I had our network team is trunk the necessary VLAN's that I needed for vSwitch0 (where we have both SC and VMkernel reside), then set the VLAN for the SC to be the native VLAN. This allowed our installation routine (which pulls cd source from an NFS volume) to run as expected. Make sure that you enable portfast (or the equivalent for your switch if you aren't using Cisco gear)... Yes, you can enable portfast for trunked ports on Cisco gear, we do it every day Smiley Happy

-Steve
dwight
Enthusiast
Enthusiast
Jump to solution

It's to bad the installer doesn't support vlan tags. In order to get it to work we've had to configure the switches so the vlan containing the kickstart server is the default vlan (no vlan tags) and all the other vlans are vlan tagged. Also make sure the switch has portfast disabled and bpdu protection to off.

However, if you are installing to a blade chassis. The switches in your chassis may be smart enough that they can be configured to handle the vlan tagging/untagging for the installation process without having to change the settings in the core switches.
RHCE, VCP

Blog: http://computing.dwighthubbard.info

RHCE, VCP Blog: http://computing.dwighthubbard.info
Reply
0 Kudos
Aricade
Contributor
Contributor
Jump to solution

Never mind my post sorry

Should read more careful

Reply
0 Kudos
Jae_Ellers
Virtuoso
Virtuoso
Jump to solution

Using a native vlan during the install is the way to go on a trunked port. The native vlan will receive all untagged traffic on the physical switch. Once the install is done you can add your networks with the other vlans that the physical switch sends over the trunked port.

It's great to have several vlans coming in on each physical switch port or pair of physical swtich ports if your security model supports this kind of media sharing.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-=-=-=- http://blog.mr-vm.com http://www.vmprofessional.com -=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Reply
0 Kudos
Ken_Cline
Champion
Champion
Jump to solution

It's to bad the installer doesn't support vlan tags.

It's not the installer that's at fault. In order to perform a bare metal install using a VLAN, your boot PROM would have to support VLANs - or you would need to boot a loader from a remote management solution (iLO/Director/DRAC/LOM/etc.) with virtual media support. I'm not sure either of these would make your life any easier. I think diztorted has the right answer (if your networking and security teams will buy off on it) - PXE enable your home VLAN and put your SC there.

Ken Cline

Technical Director, Virtualization

Wells Landers

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

Hi,

Thank you all for your answers. As I understood I have no other choice than using the default VLAN to achieve this.

Can someone indicate me a good site or documentation dealing with this subject that I could forward to my network team to make them comfortable with this kind of configuration?

Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Hiya,

As Jae and distorted say - Native Vlans are definitely the way to go - but that does not necessarily need to be the default vlan.

If you are using cisco kit - then you can use whatever vlan you like as the Native VLAN.

Configuring the cisco port for the service console like this will do the trick:

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 77

switchport trunk allowed vlan 72,77,95

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk

In the example above PXE can happily initially take place on VLAN 77 (make sure an IP helper statement on that vlan directs PXE packets to wherever your PXE/dhcp servers are)

Then you can amend the SC portgroup to use vlan 72 via your kickstart script (setting it to run in rc.local)

vlan 95 can then be used (in this case) as a backup connection for vmotion.

Doubt you'll find anything documented on it by VMware - though there are a few posts on the forums - but its all cisco stuff really - should mean something to your network guys anyway.

Cheers

Dinny

Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

Sorry, I meant Native Vlan and not default. My mistake.

I'm going to review the configuration with my network team and see if we can do that.

Last question before trying to configure this native Vlan : why is it advised to enable portfast? If I remember well, one of the network engineers told me once that it could interfere with spanning tree when enabling this setting on a trunk. Maybe it's different because esx provides vswitch that have a different behavior than real switch but I'm pretty sure they're going to ask me this question.

Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Hiya,

If you don't have portfast set then there can be big delays (say 30 secs) for example when your SC network fails over to a backup one (if you have resilience) - or more often when the backup one fails back to the initial one.

If you have HA enabled then this can be enough to trigger the HA isolation response - so all your VMs are powered off and brought back up on another ESX server.

Less than ideal in most cases Smiley Happy

Can't remember for certain off the top of my head - but I'm fairly sure that ESX vswitches don't suffer from the spanning tree issues anyway.

There is a good presentation from one of the VMworlds or TSXs by VMware about it - if you search the forums you should find a few links to it...

Dinny

Ken_Cline
Champion
Champion
Jump to solution

Can't remember for certain off the top of my head - but I'm fairly sure that ESX vswitches don't suffer from the spanning tree issues anyway.

vSwitches don't participate in spanning tree, so they can't cause a problem. The only way to create a loop is to have a rogue VM bridging between two vSwitches.

Ken Cline

Technical Director, Virtualization

Wells Landers

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

Hi,

Sorry for the delay but we finally had the opportunity to get a new test server on which we could try to configure the Native VLAN.

It almost worked.

We configured a DHCP in this VLAN with forwarding to a PXE server that send the good image and parameter to the server.

We manage to get this part working but then the ESX 3.5u1 installer part fail to get an IP address. Then it is unable to catch our config file stored on an HTTP server.

It seems that the response from the DHCP doesn't come fast enough for the installer to get its IP and then it fails and asks me to enter the network parameters manually. If I do an Alt-F3 I get the message "Pump told us: No DHCP reply received"

Then, if I try to set the IP manually, it neither bring up the network fast enough to communicate with my http server.

It's the same kind of behaviour we have when portfast is disabled.

It's weird because "spanning-tree portfast trunk" is set in the config.

Someone has an idea to make this work?

Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Hiya,

I'm afraid it may well be down to your physical network switches (or the OS running on them).

Our environment runs across two separate Datacentres - my PXE builds all work fine in the newer datacentre - using Cisco IOS (ver 12.2) on the physical switches.

In the older datacentre myPXE builds never worked (2nd or 3rd dhcp request in the build process always fails or times out..) - using older Cisco CATOS (ver 7.6) switches.

I spent days trying various solutions to fix or work around it - none worked - we now have a new Cisco IOS switch in there - and the ESX servers on that switch build just fine....

So if you have a similar issue - unless you have any control over the physical switch env you may be struggling...

Dinny

Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

Thanks for your fast answer.

We use C7000 HP blade with integrated Cisco Catalyst Blade Switch 3020.

In fact, the problem is probably not PXE related because this part goes perfectly well.

It’s really in the ESX setup that we cannot get the DHCP.

Even if I boot with the CD and I try to get the config file with a command line like :

esx ks=http://xxx.xxx.xxx.xxx/ks.cfg

I have the same behaviour.

Are there some parameters we can change to make this work?

Reply
0 Kudos
dinny
Expert
Expert
Jump to solution

Indeed - the problems aren't in PXE they are in the Anaconda part of the base Linux set up.

There are several docs around suggesting lots of params to try tweaking to make it work (google for them - or look at the hude UDA thread on vmtn - I think there have been a few posts on there...) - as I said I spent days (if not weeks?) trying out all the suggestions I could find - none had any effect for me in the one datacentre. Only good thing was I knew my side of things was fine as it all worked fine in the other datacentre...

I was using ESX 3.02 - which is based on RH 3 - not sure what version of RH ESX 3.5 is based on - but I imagine it is still a pretty old one, with a fairly old Anaconda version integrated into it? Quite possibly the Anaconda developers have sorted out some of the the bugs in later versions of Anaconda...?

Is your dhcp server also on the HP blades - or is the dhcp request traversing some old(er) cisco kit too? If so that could be where the issue lies?

Know it doesn't help much - but at least you know it's not just you...

Dinny

Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

I also think the problem lie in the Anaconda part... it's a shame.

I remember having the same kind of problem when I first deployed ESX but setting portfast resolved it. However in this case with VLAN and trunks, it doesn't seem to be enough.

I built a temporary DHCP server in a VM hosted on a server connected to the same blade center and the same network switch to avoid issues you mention. But as I said: even if I use fixed IP address I encounter the problem so it's probably not DHCP related either. It really looks like the interface doesn't bring up fast enough to catch my config file. The most irritating part is: if I let a ping running in the background to check if the interface is up, it times out when the setup tries to get the config file but as soon as he failover and reach the default installation process (Choose a language), the ping starts to answer... frustrating...

I'm going to dig the posts you suggested.

Meanwhile, if there are other advices, feel free to tell me!

Reply
0 Kudos
smonborgne
Enthusiast
Enthusiast
Jump to solution

Hi,

Just wanted to let you know that we finally manage to make it work!

The problem was (don't laugh...) that our network admins did set the port to "portfast" but not to "portfast trunk"! That's why the DHCP request was timing out and we didn't reach the network.

So everything is working flawlessly and our ESX installations are now a breeze!

Thank you all for your help

Sylvain

Reply
0 Kudos