We have a Dell M1000e blade chassis with a number of M605 blade servers running ESXi 5.0 using the Nexus 1000V for networking. We are using 10G Ethernet in fabrics B and C for a total of 4 10G NICs per server. We are not using the 1G NICs on fabric A. We currently use one NIC from fabrics B and C for VM traffic and the other NIC from each fabric for VM management/vMotion/iSCSI traffic. We currently use EqualLogic PS6010 iSCSI arrays and have two port-groups setup with iSCSI bindings (one to physical NIC vmnic3 and one to physical NIC vmnic5).
We have added an EMC VNX 5300 unified array to our setup and we configured three additional VLANs on our networking setup - two for iSCSI and one for NFS. We added addition vEthernet port-profiles for the three new VLANs but when we added new vmk# ports on some of the ESXi servers, they couldn't ping anything. We did a TAC case with Cisco and it was determined that only a single port-group with iSCSI bindings can be tied to one physical uplink at a time.
We decided we would temporarily add the new VLANs to the list of allowed VLANs on the physical switch trunk ports currently used only for VM traffic. We need to remove the new VLANs from the current ethernet port-profile but are running into an issue.
The current Nexus 1000V port-profile we need to change is:
port-profile type ethernet DenverMgmtSanUplinks
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 2306-2308,2311-2315
channel-group auto mode passive
no shutdown
system vlan 2306-2308,2311-2315
description MGMT SAN UPLINKS
state enabled
We need to remove VLANs 2313-2315 from the 'system vlan' list in order to be able to remove them from the 'switchport trunk allowed vlan' list.
However, when we try to do so, we get an error about the port-profile currently being in use:
vsm21a# conf t
Enter configuration commands, one per line. End with CNTL/Z.
vsm21a(config)# port-profile type ethernet DenverMgmtSanUplinks
vsm21a(config-port-prof)# system vlan 2306-2308,2311-2312
ERROR: Cannot remove system vlans, port-profile currently in use by interface Po2
We have 6 ESXi servers connected to this Nexus 1000V. Originally they were VEM 3-8 but apparently when we did a firmware upgrade, they got re-designated as VEMs 9-14 and the old 6 VEMs and associated port-channels, are orphaned.
For example, if we look at port-channel 2 in more detail, we see its tied to the orphaned VEM 3 and it has no ports associated with it:
vsm21a(config-port-prof)# sho run int port-channel 2
!Command: show running-config interface port-channel2
!Time: Fri Apr 26 18:59:06 2013
version 4.2(1)SV2(1.1)
interface port-channel2
inherit port-profile DenverMgmtSanUplinks
vem 3
vsm21a(config-port-prof)# sho int port-channel 2
port-channel2 is down (No operational members)
Hardware: Port-Channel, address: 0000.0000.0000 (bia 0000.0000.0000)
MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s
Beacon is turned off
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth3/4, Eth3/6
Last clearing of "show interface" counters never
102 interface resets
We can probably delete port-channel 2 but assume the error about the port-profile being in use will cascade to the other port-chanels. We can remove the other orphaned port-channels 4,6,8,10 and 12 as they are associated with the orphaned VEMs but we expect we wil then also get errors about port-channels 13,15,17,19,21 and 23 that are associated with the active VEMs.
We're looking to see if there is an easy way to fix this on the VSM or if we need to break off one of the physical uplinks on each server, connect them to a vSS or vDS, and migrate all of the vmkernel ports off of the Nexus 1000V so we can clean up the VLAN issue.
