PXE build of ESX server on a NIC/port configured for 801.2q vlan tagging

[dinny]

Hiya,

I have spent some time recently trying to put together a PXE boot environment to build ESX servers.

This was helped no end with the release last week of the Universal Deployment Appliance:

http://www.rtfm-ed.co.uk/?page_id=366

If I do the build on a standard network patch configured with a single VLAN, connected to the ESX hardware it works perfectly.

Unfortunately my ESX server network connections use VLAN tagging to provide standby capabilities for the SC and Vmotion.

I have two physical network patches with 802.1q trunking set at the Cisco end which provide VLAN tagging for two VLANs. One for the SC and one for vmotion.

Once the ESX server is built I then have a portgroup for each VLAN - this all works fine.

Unfortunately if I try to PXE boot the ESX server off a network patch with 802.1q tagging set at the Cisco end, then it fails to find a DHCP server.

I am guessing that this is because the PXE boot agent on the NIC can not deal with the 802.1q trunking?

Can anyone confirm if my suspicions are correct? - or even better does anyone know of a way of getting it to work in this config?

I appreciate I have some other options:

a) I could remove the 802.1q trunking during the build process (but that is another team's responsibility - so they would not be too happy to keep having to reconfigure the cisco ports for me)

b) I could just use a CD or USB mem stick to boot from instead of a PXE server. I will revert to this option if necessary.

Cheers

Dinny

[Chris_Lynch · Accepted solution]

In your switch config, you should have a Native VLAN. On this VLAN, I would use this for your build network, which would have DHCP, PXE and your deployment host.

You could also use the Default VLAN config. This is used when the interface is no longer participating in an 802.1q trunk.

The following is an example config:

interface <interface-id>

switchport mode trunk

switchport access vlan <vlan-id>

switchport trunk native vlan <vlan-id>

switchport trunk allowed vlan add <vlan-list>[/i]

[dinny]

Thanks Chris,

I'll have a look at those suggestions in the morning - once our Network team are back in - as I have no Cisco access myself.

Dinny

[dinny]

Hi Chris,

I spoke to our network team and they suggested just adding one of the already specifically defined VLANs as the Native VLAN

They configured the cisco port as below:

switchport trunk encapsulation dot1q

switchport trunk native vlan xx

switchport trunk allowed vlan xx,yy

switchport mode trunk

switchport nonegotiate

spanning-tree portfast trunk[/i]

Where xx is the vlan for the SC (which has the IP forwarders to our DHCP servers) and yy is the vlan for vmotion.

This appeared to work perfectly - the PXE boot agent worked fine - the build script ran and the server is accessible via VLAN xx on the SC.

I don't know too much about Cisco - so I just wondered if you might be aware of any downsides to such a configuration?

I guess that it means all the VLAN xx traffic is available both untagged and tagged on this cisco port.

Can you think of any problems/issues that this particular config might cause?

Dinny

[Chris_Lynch]

Only if you tell your network team to apply the same config to another port that will service VM's. You do not want VM's to have access to the CoS network/VLAN.

[dinny]

Thanks Chris,

My VMs would not have direct access to ports configured in such a way.

They all use different virtual switches with different physical NIcs

Dinny

[dinny]

Hiya,

Quick update:

I spoke to our network team and they suggested just

adding one of the already specifically defined VLANs

as the Native VLAN

Once I got this up and running and actually set my SC and vmotion portgroups to use one pNIC as active and one pNIC as standby I lost all comms to my SC.

I can only presume the virtual switch gets confused by the mixture of tagged and untagged packets for the same VLAN?

I then tried various solutions - the one that seemed to work was using a native VLAN ID (for the PXE boot) - but a different[/b] VLAN to the one I was to use for the actual SC or vmotion portgroups.

In effect the native VLAN is just used to allow the PXE boot client on the NIC to point to a DHCP server - and hence be allocated an IP address to contact the UDA appliance to start the PXE boot install process.

Dinny

[moberle]

Could someone please help with the DHCP/PXE configuration issues. I have to same problem with the PXE client not receiving the DHCP Offer from the DHCP server. We have run a sniff on the network and determined that the DHCP server receives the request and replies with an Offer. The packets never get back to the PXE client.

I am trying to install ESX 3.0.1 using a PXE boot to a MS DHCP/PXE server. The ESX server (Dell 1955 blade) is connected to the network via a switch port on a 6509. The DHCP/PXE server is a Windows 2003 server running in a VM on another ESX host on the same subnet and vlan (Native) as the PXE client.

The cisco port configuration of the ESX server containing the DHCP server VM is as follows:

interface GigabitEthernet3/39

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 322

switchport trunk allowed vlan 2-4094

switchport mode trunk

switchport nonegotiate

no ip address

no cdp enable

spanning-tree portfast

spanning-tree bpduguard enable

spanning-tree guard root

end

The cisco port configuration of PXE client machine is as follows:

interface GigabitEthernet3/42

switchport

switchport trunk encapsulation dot1q

switchport trunk native vlan 322

switchport trunk allowed vlan 2-4094

switchport mode trunk

switchport nonegotiate

no ip address

no cdp enable

spanning-tree portfast

spanning-tree bpduguard enable

spanning-tree guard root

end

We have the Virtual Switch on the ESX host that the DHCP VM is running on, set up with a pxe vlan defined with 322 (the vlan defined as native) to allow the Virtual Switch to see the native vlan. and a mgmt (vlan 310) vlan defined to allow tagged vlan traffic to go through to our subnet.

We have 2 vNICs defined in the DHCP VM one using the pxe vlan and one using the mgmt vlan to allow RDP and other network type connections to work across the tagged vlans.

I have tried disconnecting the mgmt vlan connected vNIC to make sure there was not conflict there. (Both Nics have IP addresses on the same subnet I KNOW I KNOW If someone can tell me how to add a second vlan to the one vNIC i would appreciate that as well.) DHCP is setup to only service the vNIC using the the pxe vlan. With this setup the RDP traffic fails and the DHCP Offer is still outbound and not received by the PXE client.

If anyone can please point me to some documentation with the complete configurations needed to setup the Cisco ports and the vSwitches and vNICs or can help me with this directly I would greatly appreciated it.