Good morning-
We have two Lenovo RD640 servers running ESXi 5.5 and using the first onboard nic for Management network and an external nic for failover/HA. We have a major issue that the management networks on both servers come up as not working and they won't ping the core switch. Even stopping and restarting management services from the GUI will not kick start the management network and allow it to ping.
Here is how I have to get the management network working after a reboot:
Change the selected management NICS to two other nics, even ones that aren't connected to anything.
Change the IP Address to some random IP address.
Change the management NICS back to the original ones
change the IP Address back to the original correct ones.
Test the management network and it pings the core switch.
This is repeatable after a reboot.
Please provide some more information, e.g. a screenshot of the virtual network configuration. How are the policies configured on the vSwitch and the port groups? In addition to the virtual network setup, provide the configuration of the uplink ports on the physical switch.
André
Sorry for the lack of information, here is how it's configured.
Let me know if you need anything else. I'm still looking into the Core switch configuration but our network engineer set that part up.
I'm pretty sure the physical port configuration will help. Until you get that let me ask you a few questions.
André
Just saw the second reply with the physical configuration, where switchport access vlan 99 most likely causes the issue, and should be removed.
Here's an example how this should look like:
interface GigabitEthernet 4/35
description ESXi2 #1
switchport trunk encapsulation dot1q (may not be displayed on some switches if it's the default)
switchport trunk allowed vlan 1, 20, 30, 40, 50, 99
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
With both uplink ports being configured this way, you need to set the VLAN-IN on all port groups, except for VLAN 1 which seems to be the native VLAN.
André
a.p. wrote:
Just saw the second reply with the physical configuration, where switchport access vlan 99 most likely causes the issue, and should be removed.
Here's an example how this should look like:
interface GigabitEthernet 4/35
description ESXi2 #1
switchport trunk encapsulation dot1q (may not be displayed on some switches if it's the default)
switchport trunk allowed vlan 1, 20, 30, 40, 50, 99
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
With both uplink ports being configured this way, you need to set the VLAN-IN on all port groups, except for VLAN 1 which seems to be the native VLAN.
André
Andre, let me explain how this is setup, although I'm not sure why my network engineer did it this way. He's the one who said we needed to add the VLAN99 ID to the management network.
The 10gig adapter is on the default VLAN. The 1gig adapter is on the 99 vlan. There is routing in place from the default vlan (192.168.0.xx) to the 99 vlan (192.168.128.xx) and I can access the management network on 192.128.128.8 and .9 from the default VLAN. The 10 gig adapter is also in use for all the VMS as their primary NIC on the default VLAN.
That being said, do you still feel the same way about this setup? When you say set the VLAN ID on all port groups, what VLANID do you mean?
I'm afraid it won't work this way. You may either configure the physical port as an "access port" or a "trunk port" on a Cisco switch. If it is configured as an access port, then the VLAN tag is handled by the physical switch and the ESXi host will receive untagged traffic. In case of a trunk port (802.1Q), it's the ESXi host (i.e. the port groups) which takes care of the VLAN tags. I'm pretty sure that the 10 Gig port is configured differntly!?
>>> When you say set the VLAN ID on all port groups, what VLANID do you mean?
One of the VLANs that's on the "vlan allowed" list in the switch port configuration (except for the native VLAN). In case of the native VLAN do not configure a VLAN-ID on the port group, or you will loose connection!
André
Andre, 10gig config below:
In case you are using the vmnics in a failover mode (what you actually doing), both physical ports need to be configured that same way. You cannot configure one port as an access port and the other one as a trunk port. With the default teaming policy "Route based on originating virtual port ID" the vmnics are assigned in a round-robin manner, and if the Management Network gets assigned to the access port, it won't work, because the physical switch port doesn't know how to handle the VLAN tag (VLAN 99).
I'd suggest that you either configure both physical ports the same way (my recommendation), or use two seperate vSwitches for the differently configured ports.
André
a.p. wrote:
In case you are using the vmnics in a failover mode (what you actually doing), both physical ports need to be configured that same way. You cannot configure one port as an access port and the other one as a trunk port. With the default teaming policy "Route based on originating virtual port ID" the vmnics are assigned in a round-robin manner, and if the Management Network gets assigned to the access port, it won't work, because the physical switch port doesn't know how to handle the VLAN tag (VLAN 99).
I'd suggest that you either configure both physical ports the same way (my recommendation), or use two seperate vSwitches for the differently configured ports.
André
Andre-
I believe this is partially my fault. When I setup HA, the system prompted me to use the 10gig nic for the failover NIC, and when that happened, this is what VMWare did. I'm going to have to ask my network engineer to take a look at this.
I had two separate VSwitches configured for those NIC ports until I tried setting up HA, I had to move them to the same switch port for that. That being said, how would you handle this?
a.p. wrote:
In case you are using the vmnics in a failover mode (what you actually doing), both physical ports need to be configured that same way. You cannot configure one port as an access port and the other one as a trunk port. With the default teaming policy "Route based on originating virtual port ID" the vmnics are assigned in a round-robin manner, and if the Management Network gets assigned to the access port, it won't work, because the physical switch port doesn't know how to handle the VLAN tag (VLAN 99).
I'd suggest that you either configure both physical ports the same way (my recommendation), or use two seperate vSwitches for the differently configured ports.
André
Oh forgot to mention HA and VMotion does work with this configuration..
There's unfortunately no rule of thumb for how to setup networking. Anyway, what I do in most cases is to configure the physical network ports as trunk (802.1Q) ports and manage the VLAN tagging on the virtual port groups. Depending on the environment and the requirements, I then check whether it makes sense to configure the failover order for some of the port groups, in order to distribute the network traffic across the different vmnics.
As mentioned, it depends on the requirements and the environment (i.e. number and speed of uplinks, vSphere features used, number of physical switches, ...)
André
a.p. wrote:
There's unfortunately no rule of thumb for how to setup networking. Anyway, what I do in most cases is to configure the physical network ports as trunk (802.1Q) ports and manage the VLAN tagging on the virtual port groups. Depending on the environment and the requirements, I then check whether it makes sense to configure the failover order for some of the port groups, in order to distribute the network traffic across the different vmnics.
As mentioned, it depends on the requirements and the environment (i.e. number and speed of uplinks, vSphere features used, number of physical switches, ...)
André
I'm thinking it might make more sense to use one of the other extra available NICS on the servers and run a second line to the switch and setup that switch port on access 99 and move the 10 gig to it's own VSWITCH? That way we have dedicated failover on vlan 99. I have 2 extra empty NICS on the server.
That makes sense, and is what I meant with " ...it depends on the requirements and the environment ...". Simply make sure all uplink ports attached to a single vSwitch are configured the same way.
André
Andre, I found this information online about using Switchport Access VLAN and Switchport mode Trunk at the same time regarding the switch port on that host which is in this config:
interface GigabitEthernet5/35
description ESXi2 #2
switchport access vlan 99
switchport trunk allowed vlan 1,20,30,40,50,99
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
spanning-tree bpduguard enable
end
"No this isn't doing anything. When the port is in trunk mode, then any "switchport access" statements are not used. The same is true with a port in access mode with any "switchport trunk" statements."
Therefore, the VLAN tagging in the host shouldn't affect anything if the switch is disregarding the switchport access VLAN99 in the config.
I also checked the other cluster setup and all the other VLANS are tagged in the vswitches and the physical switch ports are setup without switchport access VLAN, just switchport mode trunk, like this.
interface GigabitEthernet4/41
description ESXi #2 VMs
switchport trunk allowed vlan 1,20,50,99
switchport mode trunk
switchport nonegotiate
speed 1000
duplex full
end
So that being said, I'm not sure what do do at this point, I'm waiting on the network engineer to get in touch with me.
Full switchport config
CHKNRNCH-CORE-01#show interfaces gi5/35 switchport
Name: Gi5/35
Switchport: Enabled
Administrative Mode: trunk
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: Off
Access Mode VLAN: 99 (MANAGEMENT)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: 1,20,30,40,50,99
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL
>>> I found this information online about using Switchport Access VLAN and Switchport mode Trunk at the same time ...
Thanks for this information. I didn't know this (actually I never configured a port this way), but I'm willing to learn ![]()
Unless the fixed port and duplex setting are required, I'd remove them. Also make sure each ESXi uplink port has spanning-tree portfast [trunk] configured.
André
Hi André,
This is the network engineer responding with Brad's account. The 'switchport access vlan 99' is not doing anything since the port is configured as a trunk, that's left over from the initial config when the ESXi box was originally IP'd and connected to the switch. In this case, all uplinks are trunked (native VLAN is 1) and that's why we have the vSwitches tagged (except for the default).
I agree that the speed/duplex should never be manually configured, I'll have to see when that got changed.
Thanks for your help, we'll work to get this resolved with your info.
