dgoldsmith
Contributor
Contributor

Removing 'system vlans' from Nexus 1000V port-profile

Jump to solution

We have a Dell M1000e blade chassis with a number of M605 blade servers running ESXi 5.0 using the Nexus 1000V for networking.  We are using 10G Ethernet in fabrics B and C for a total of 4 10G NICs per server.  We are not using the 1G NICs on fabric A.  We currently use one NIC from fabrics B and C for VM traffic and the other NIC from each fabric for VM management/vMotion/iSCSI traffic.  We currently use EqualLogic PS6010 iSCSI arrays and have two port-groups setup with iSCSI bindings (one to physical NIC vmnic3 and one to physical NIC vmnic5).

We have added an EMC VNX 5300 unified array to our setup and we configured three additional VLANs on our networking setup - two for iSCSI and one for NFS.  We added addition vEthernet port-profiles for the three new VLANs but when we added new vmk# ports on some of the ESXi servers, they couldn't ping anything.   We did a TAC case with Cisco and it was determined that only a single port-group with iSCSI bindings can be tied to one physical uplink at a time.

We decided we would temporarily add the new VLANs to the list of allowed VLANs on the physical switch trunk ports currently used only for VM traffic. We need to remove the new VLANs from the current ethernet port-profile but are running into an issue.

The current Nexus 1000V port-profile we need to change is:

port-profile type ethernet DenverMgmtSanUplinks

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 2306-2308,2311-2315

  channel-group auto mode passive

  no shutdown

  system vlan 2306-2308,2311-2315

  description MGMT SAN UPLINKS

  state enabled

We need to remove VLANs 2313-2315 from the 'system vlan' list in order to be able to remove them from the 'switchport trunk allowed vlan' list.

However, when we try to do so, we get an error about the port-profile currently being in use:

vsm21a# conf t

Enter configuration commands, one per line.  End with CNTL/Z.

vsm21a(config)# port-profile type ethernet DenverMgmtSanUplinks

vsm21a(config-port-prof)# system vlan 2306-2308,2311-2312

ERROR: Cannot remove system vlans, port-profile currently in use by interface Po2

We have 6 ESXi servers connected to this Nexus 1000V.  Originally they were VEM 3-8 but apparently when we did a firmware upgrade, they got re-designated as VEMs 9-14 and the old 6 VEMs and associated port-channels, are orphaned.

For example, if we look at port-channel 2 in more detail, we see its tied to the orphaned VEM 3 and it has no ports associated with it:

vsm21a(config-port-prof)# sho run int port-channel 2

!Command: show running-config interface port-channel2

!Time: Fri Apr 26 18:59:06 2013

version 4.2(1)SV2(1.1)

interface port-channel2

  inherit port-profile DenverMgmtSanUplinks

  vem 3

vsm21a(config-port-prof)# sho int port-channel 2

port-channel2 is down (No operational members)

  Hardware: Port-Channel, address: 0000.0000.0000 (bia 0000.0000.0000)

  MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  auto-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  Members in this channel: Eth3/4, Eth3/6

  Last clearing of "show interface" counters never

  102 interface resets

We can probably delete port-channel 2 but assume the error about the port-profile being in use will cascade to the other port-chanels.  We can remove the other orphaned port-channels 4,6,8,10 and 12 as they are associated with the orphaned VEMs but we expect we wil then also get errors about port-channels 13,15,17,19,21 and 23 that are associated with the active VEMs.

We're looking to see if there is an easy way to fix this on the VSM or if we need to break off one of the physical uplinks on each server, connect them to a vSS or vDS, and migrate all of the vmkernel ports off of the Nexus 1000V so we can clean up the VLAN issue.

0 Kudos
1 Solution

Accepted Solutions
lwatta
Hot Shot
Hot Shot

You won't be able to remove the system vlans until nothing using that port-profile. We are very protective of any vlan that is designated on the system vlan command line.

You should clean up the old port-channels and the old VEMs. You can safely do a "no int port-channel" and "no vem" on the devices that are no longer being used.

What you can do is create a new uplink port-profile with the settings you want. Then swap the interfaces over to the new port-profile. It's usually easier to create a new one then to try and clean up and old port-profile with system vlan settings.

I would do the following steps.

Create a new port-profile with the settings you want

Put the host in maintenance mode if possible

Pull one nic out of the old N1Kv eth port-profile

Add the nic to the new N1Kv eth port-profile

Pull the second nic out of the old eth port-profile

Add the second nic to the new port-profile

You'll get some duplicate packet error messages but it should work.

The other option is to remove the host from the N1Kv and add it back using the new eth port-profile.

Another option is to just leave it. Unless it's really bothering you no VMs will be able to use those port-profile unless you create a veth port-profile on that VLAN.

louis

View solution in original post

0 Kudos
2 Replies
lwatta
Hot Shot
Hot Shot

You won't be able to remove the system vlans until nothing using that port-profile. We are very protective of any vlan that is designated on the system vlan command line.

You should clean up the old port-channels and the old VEMs. You can safely do a "no int port-channel" and "no vem" on the devices that are no longer being used.

What you can do is create a new uplink port-profile with the settings you want. Then swap the interfaces over to the new port-profile. It's usually easier to create a new one then to try and clean up and old port-profile with system vlan settings.

I would do the following steps.

Create a new port-profile with the settings you want

Put the host in maintenance mode if possible

Pull one nic out of the old N1Kv eth port-profile

Add the nic to the new N1Kv eth port-profile

Pull the second nic out of the old eth port-profile

Add the second nic to the new port-profile

You'll get some duplicate packet error messages but it should work.

The other option is to remove the host from the N1Kv and add it back using the new eth port-profile.

Another option is to just leave it. Unless it's really bothering you no VMs will be able to use those port-profile unless you create a veth port-profile on that VLAN.

louis

View solution in original post

0 Kudos
dgoldsmith
Contributor
Contributor

Ok, thanks.  We got similar information from a VMware Support call yesterday and we started implementing the changes.

0 Kudos