vlans unreachable problem

[diegosoloaga]

Hi all,

I have an IBM Blade Chassis H with 2 BNT Layer 2/3 Copper Gigabit Ethernet Switch Modules firmware 5.3.0.5. Each switch is connected to one upstream switch using two trunks with two external ports. Each trunk is tagged with two different vlans (100 and 200). Blade servers are running ESXi 5.1. Internal ports are vlan tagged, pvid is 100 and tagging-pvid is disabled. Each blade is configured with one standard switch that has Management network. This standard switch is connected to vmnic0 and vmnic1.

The problem is that vmnic0 and vmnic1 don´t detect networks other than management. I am trying to use the two gigabit ethernet switches for iSCSI traffic but when I add a new vkernel with iscsi vlan tagged I can´t reach the network even though is connected and can be pinged from the switches. The kernels are connected following documentation, two vkernels with one vmnic available and the other disabled.

Thanks in advance

[JPM300]

Hey diegosoloaga,

If you could supply us with a screenshot of your networking screen from the VI that would help greatly.

However from what you have explained it seems like your vSwitch0 is a VSS with one port group assisgned to it, which was Management.  This port group has a VLAN tag of 100.  You then added two other port groups, iSCSI0, ISCSI1 however these will not communicate.

What VLAN tag did you give the iSCSI vfk kerenels?

Also I don't understand what you mean by VLAN tag 100 for management is tagged as a pvid but pvid is disabled.  If those ports are indeed in pvid mode you may not be able to assign other VLAN's to them.  It depends on the switch.

What you will need to do is either have you iSCSI kerenels on VLAN 100, 200, or a different VLAn tag all together, and have this tagged all the way up to your physical switches

Another way you can test the setup is once you have the VLAN's on your iSCSI kerenels see if you can do a vmkping from the CLI to each iSCSI kernel IP.  If they can communicate with each other but not the exteranl world then you have a external switch problem.

One last note, the iSCSI kerenels should have dedicated nics for best performacne / best practices.

[diegosoloaga]

Hi JPM300,

Thanks for yuor reponse, I´ll try to give more details

pSwithch

External ports

EXT1 and EXT2 are a trunk tagged with vlan 100, vlan tagging disabled

EXT3 and EXT4 are a trunk tagged with vlan 200, vlan tagging disabled

External ports

INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.

VSS

     vmnic1 active,

     vmnic2 active

portgroup mgmt, no vlan, this is working

     vmnic1 active,

     vmnic2 stand by

portgroup iscsi 1, vlan 200, ip 10.0.0.1

     vmnic1 active,

     vmnic2 unused,

portgroup iscsi 2, vlan 200, 10.0.0.2

     vmnic2 active,

     vmnic1 unused,

Storage Adapter

vmhba34

     port binding

     iscsi 1 vmk1 -> vmnic1 (sais compilant but "path status" is unused)

     iscsi 2 vmk2 -> vmnic2 (sais compilant but "path status" is unused)

Storage IP 10.0.0.5

Management is on 192.168.0.x network and iscsi is on 10.0.0.0. When I go to vss properties-> Netwokr Adapters and check vmnic 1,2 the observed IP ranges is only 192.168.0.x-192.168.0.y.

The problem:

If I connect to the pswitches I can ping the storage but not the iscsi portgroups.

If I tag the mgmt portgroup with vlan 100 I loose conectivity

Thanks

I

[JPM300]

Hey diegosoloaga,

Are these Nortel Switches or Cisco Switches on the back of the chassis in the IO slots?

I have a feeling they are Nortel Switches due to the way things are setup:

This confuses me a little:

External ports

INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.

So internal ports 1,2,3,4,5...etc VLAN 100 and VLAN 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.

^This to me says all your internal ports(AKA the ports your blades connect into on the back plane in the chassis) are tagged for VLAN 100 and VLAN 200, however pvid is 100 so anything without VLAN 100 or VLAN 200 will get VLAN 100 as its the pvid.  The ports also allow tagging dispite the pvid.  Many switches allow this which urks me a little as if you have a pvid setup or a port in access mode it should ONLY ever get 1 VLAN, but i know HP switches do this as well.  None the less this is how I read it. 

So since your management portgroup doesn't have a VLAN set on it, it will get VLAN 100 due to the pvid being 100.  Now when you tag your management group you loose control as I'm assuming this is because vCenter and or your management system lives outside the chassis?  If so this tells me that for some reason your EXT1 and EXT2 ports are not passing tagged VLAN's for some reason.  I'm not sure what this means exactly:

EXT1 and EXT2 are a trunk tagged with vlan 100, vlan tagging disabled

EXT3 and EXT4 are a trunk tagged with vlan 200, vlan tagging disabled

this to me says EXT1 and EXt2 are in access mode for VLAN 100 as vlan tagging is disabled? 

this to me says EXT3 and EXT4 are in access mode for VLAN 200 as vlan tagging is disabled?

If you could shed more light on this that would be great.

I have a feeling your problem is probably with the EXT ports as I used to have constant problems with these back when we deployed IBM Blade chassis a lot

SSH into your hosts and check to see if you can do a vmkping from host1 to host2 on the iSCSI network.  Your East/West traffic shouldn't have to leave the internal switch so it should work. 

[JPM300]

Further more we need to make sure your VLAN 200 / 100 route properly all the way to your core

It's been awhile since I have had to work with the IBM Chassis so some of this might be wrong but you get the idea.

[rcporto]

Any chance to attach the configuration of BNT switches (removing the sensitive data) ?

[diegosoloaga]

Hi JPM399,

I am not really sure if the switches are Nortel, here is the description:

switch-type "BNT Layer 2/3 Copper Gigabit Ethernet Switch Module for IBM BladeCenter"

INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.

For what I know the ports with vlan tagged enable become trunk ports as they are called in cisco swithces, meaning they don´t remove tags. So when you tag the ports with vlans 100 and 200  they onlye let those vlans go through.

As you say, pvid is the vlan that is tagged if the packet has no vlan.

So since your management portgroup doesn't have a VLAN set on it, it will get VLAN 100 due to the pvid being 100. 

Yes, the port group doesn´t have vlan, so it gets pvid vlan as it goes through the switch port.

Now when you tag your management group you loose control as I'm assuming this is because vCenter and or your management system lives outside the chassis?  If so this tells me that for some reason your EXT1 and EXT2 ports are not passing tagged VLAN's for some reason.  I'm not sure what this means exactly:

No, vcenter lives inside blade.

this to me says EXT1 and EXt2 are in access mode for VLAN 100 as vlan tagging is disabled? 

this to me says EXT3 and EXT4 are in access mode for VLAN 200 as vlan tagging is disabled?

Yes, both ports should be access

If I tag iscsi porgtroups and ssh to the host, I can´t ping any host on iscsi network. If I connect to the switch I can ping the storage but not the blade.

The idea is to use vst so vlans should be tagged at portgroup level and switches internal ports should be trunk (let all vlans pass).

Here is the switch config

!

spanning-tree mstp name "STP-DC"

!

interface port INT1

        pvid 100

        fastforward

        exit

!

interface port INT2

        pvid 100

        fastforward

        exit

!

interface port INT3

        pvid 100

        fastforward

        exit

!

interface port INT4

        pvid 100

        fastforward

        exit

!

interface port INT5

        pvid 100

        fastforward

        exit

!

interface port INT6

        pvid 100

        fastforward

        exit

!

interface port INT7

        fastforward

        exit

!

interface port INT8

        fastforward

        exit

!

interface port INT9

        fastforward

        exit

!

interface port INT10

        fastforward

        exit

!

interface port INT11

        fastforward

        exit

!

interface port INT12

        fastforward

        exit

!

interface port INT13

        fastforward

        exit

!

interface port INT14

        fastforward

        exit

!

interface port EXT1

        pvid 100

        exit

!

interface port EXT2

        pvid 100

        exit

!

interface port EXT3

        pvid 200

        exit

!

interface port EXT4

        pvid 200

        exit

!

vlan 1

        member INT1

        member INT2

        member INT3

        member INT4

        member INT5

        member INT6

        member INT7

        member INT8

        member INT9

        member INT10

        member INT11

        member INT12

        member INT13

        member INT14

        no member EXT1

        no member EXT2

        no member EXT3

        no member EXT4

        member EXT5

        member EXT6

!

vlan 100

        enable

        name "V100-MGMT"

        member INT1

        member INT2

        member INT3

        member INT4

        member INT5

        member INT6

        member EXT1

        member EXT2

!

vlan 200

        enable

        name "V200-ISCSI"

        member INT1

        member EXT3

        member EXT4

!

no spanning-tree pvst-compatibility

spanning-tree stp 1 vlan 1

spanning-tree stp 1 vlan 100

spanning-tree stp 2 vlan 200

!

portchannel 1 port EXT1

portchannel 1 port EXT2

portchannel 1 enable

!

portchannel 2 port EXT3

portchannel 2 port EXT4

portchannel 2 enable

!

!

sflow enable

sflow server 10.100.200.130

!

interface port EXT1

        sflow polling 20

!

interface port EXT2

        sflow polling 20

!

interface port EXT3

        sflow polling 20

!

interface port EXT4

        sflow polling 20

!

failover vlan

failover trigger 1 amon portchannel 1

failover trigger 1 enable

!

failover trigger 2 amon portchannel 2

failover trigger 2 enable

!

!

!

interface ip 2

        ip address 10.100.150.200 255.255.255.0

        enable

        vlan 200

        exit

!

ntp enable

ntp primary-server 10.100.200.82

!

end

[JPM300]

Hey,

I forget in this blade chassis how does the INT ports have a relationship with the EXT is it souly based off VLAN settings?  Can every INT port leave out any EXT port?  I forget how this works in the IBM chassis as its been awhile.

Also from your switch config you only have INT1, EXT3 and EXT4 set with a VLAN of 200, so all your int ports only have pvid 100 or VLAN 100.  Meaning if you are putting a VLAN tag of 200 on your port group it will drop it once it hits the internal ports as they only have pvid of 100.  Since you want to do all of your tagging on your Port groups this will mean all traffic hitting your internal / ext ports will already have a VLAN tag on them.  So your INT ports will need to be trunked with VLAN100/200 same with your EXT ports.   Also if you only want certain INT / EXT ports to only have certain VLANs then you would trunk just the VLAN you want for that port.

With this switch aside from the VLAN membership do you have to set the options on the ports as well or does the membership do it for you?

You have two setup options here:

Figure out which INT port each nic on the blades hard wires to and setup pvids for the proper vlans to route everything correctly.  This would mean you would not set any VLANs at the port group level.

or

Trunk all the INT ports for the proper VLAN along with the EXT ports as the VLAN tag will be coming from the vSwitch/Portgroup from VMware.

If you have any other questions please let me know.

[JPM300]

Here is another example of how the trunk ports would work.

As your INT ports are the first ports any traffic from your blades hit.  If they where to never route out of the blade chassis, they would stay within the INT ports on that switch.  When traffic has to route out it will use the EXT ports.  In which case all VLANs required out will need to be passed long to your core switch.  From there your storage node will connect to the switch but seeing as most SANs typically don't set VLANS from the SP's you would put the ports  your SAN connects into in access mode or PVID200.  This would then be able to talk to the traffic on VLAN 200.  Or should.

I hope some of this has helped.

You will also need to find the IBM redpaper on how the blades nic's map to the INT ports on the internal switch/switches

[JPM300]

Did some thinking on your switch settings the other night as well.

It appears your switches are like HP's where you can have a pvid/access mode port(meaning untag traffic hits this port and the switch places the pvid VLAN on it) and also tag a vlan on this as well.

For instance:

port 1:

pvid 100

tag 200

fastforward

In this case if anything plugged into this port would get VLAN 100, however if this port see's a packet come into this port with a VLAN tag of 200 it will route it accordingly.  I know some switches allow this and it bugs me, but its an option.

Going back to what I said, and looking at your switch settings, it looks like you will either need to figure out how your NIC's in your blade map to the INT ports and then use the appropriate pvids, and not use VLAN's on your port groups

or

continue to use pvids and tag some of the ports required for the iSCSI network like I discussed at the top

or

switch the ports over to Trunk mode and pass only the VLAN's required then tag all your traffic from each port group.  Aka MGMT VLAN 100, iSCSI VLAN 200

here is some links as well that may help:

Tag/Untag Ports on HP Switches - Spiceworks

http://networkengineering.stackexchange.com/questions/6483/what-are-tagged-and-untagged-packets

anyhow hope this has helped

[diegosoloaga]

Hi JPM300,

What I know about pvid tagging is that if anything comes with pvid vlan then vlan tag is removed.

vlan 100 tagging -> port pvid 100 with tagging-pvid enabled -> forward without tagging

I found somthing it might be the problem:

http://delivery04.dhe.ibm.com/sar/CMA/XSA/03mph/0/elx_fw_cna_ibm1206-4.2.433.3-1_linux_32-64.chg

Regards

[diegosoloaga]

Here is the answer:

There´s a bug in Emulex driver version VMware_bootbank_net-be2net_4.1.255.11-1vmw.510.0.0.799733 that stops nic from seeing vlans.

I downloaded the Drivers and install it on one hosts and EUREKA. All works now.

IBM Bladecenter H Nic on Vmware

Thanks for your help.

[JPM300]

Lol,

Ohh jeez.  Well good find!

I have seen some odd bugs from emulex before.  In one of our environments the cards loose the iSCSI discovery portal after every reboot.... there still is no fix :smileysilly:

Thanks for posting the fix