Hi all,
I have an IBM Blade Chassis H with 2 BNT Layer 2/3 Copper Gigabit Ethernet Switch Modules firmware 5.3.0.5. Each switch is connected to one upstream switch using two trunks with two external ports. Each trunk is tagged with two different vlans (100 and 200). Blade servers are running ESXi 5.1. Internal ports are vlan tagged, pvid is 100 and tagging-pvid is disabled. Each blade is configured with one standard switch that has Management network. This standard switch is connected to vmnic0 and vmnic1.
The problem is that vmnic0 and vmnic1 don´t detect networks other than management. I am trying to use the two gigabit ethernet switches for iSCSI traffic but when I add a new vkernel with iscsi vlan tagged I can´t reach the network even though is connected and can be pinged from the switches. The kernels are connected following documentation, two vkernels with one vmnic available and the other disabled.
Thanks in advance
Hey diegosoloaga,
If you could supply us with a screenshot of your networking screen from the VI that would help greatly.
However from what you have explained it seems like your vSwitch0 is a VSS with one port group assisgned to it, which was Management. This port group has a VLAN tag of 100. You then added two other port groups, iSCSI0, ISCSI1 however these will not communicate.
What VLAN tag did you give the iSCSI vfk kerenels?
Also I don't understand what you mean by VLAN tag 100 for management is tagged as a pvid but pvid is disabled. If those ports are indeed in pvid mode you may not be able to assign other VLAN's to them. It depends on the switch.
What you will need to do is either have you iSCSI kerenels on VLAN 100, 200, or a different VLAn tag all together, and have this tagged all the way up to your physical switches
Another way you can test the setup is once you have the VLAN's on your iSCSI kerenels see if you can do a vmkping from the CLI to each iSCSI kernel IP. If they can communicate with each other but not the exteranl world then you have a external switch problem.
One last note, the iSCSI kerenels should have dedicated nics for best performacne / best practices.
Hi JPM300,
Thanks for yuor reponse, I´ll try to give more details
pSwithch
External ports
EXT1 and EXT2 are a trunk tagged with vlan 100, vlan tagging disabled
EXT3 and EXT4 are a trunk tagged with vlan 200, vlan tagging disabled
External ports
INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.
VSS
vmnic1 active,
vmnic2 active
portgroup mgmt, no vlan, this is working
vmnic1 active,
vmnic2 stand by
portgroup iscsi 1, vlan 200, ip 10.0.0.1
vmnic1 active,
vmnic2 unused,
portgroup iscsi 2, vlan 200, 10.0.0.2
vmnic2 active,
vmnic1 unused,
Storage Adapter
vmhba34
port binding
iscsi 1 vmk1 -> vmnic1 (sais compilant but "path status" is unused)
iscsi 2 vmk2 -> vmnic2 (sais compilant but "path status" is unused)
Storage IP 10.0.0.5
Management is on 192.168.0.x network and iscsi is on 10.0.0.0. When I go to vss properties-> Netwokr Adapters and check vmnic 1,2 the observed IP ranges is only 192.168.0.x-192.168.0.y.
The problem:
If I connect to the pswitches I can ping the storage but not the iscsi portgroups.
If I tag the mgmt portgroup with vlan 100 I loose conectivity
Thanks
I
Hey diegosoloaga,
Are these Nortel Switches or Cisco Switches on the back of the chassis in the IO slots?
I have a feeling they are Nortel Switches due to the way things are setup:
This confuses me a little:
External ports
INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.
So internal ports 1,2,3,4,5...etc VLAN 100 and VLAN 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.
^This to me says all your internal ports(AKA the ports your blades connect into on the back plane in the chassis) are tagged for VLAN 100 and VLAN 200, however pvid is 100 so anything without VLAN 100 or VLAN 200 will get VLAN 100 as its the pvid. The ports also allow tagging dispite the pvid. Many switches allow this which urks me a little as if you have a pvid setup or a port in access mode it should ONLY ever get 1 VLAN, but i know HP switches do this as well. None the less this is how I read it.
So since your management portgroup doesn't have a VLAN set on it, it will get VLAN 100 due to the pvid being 100. Now when you tag your management group you loose control as I'm assuming this is because vCenter and or your management system lives outside the chassis? If so this tells me that for some reason your EXT1 and EXT2 ports are not passing tagged VLAN's for some reason. I'm not sure what this means exactly:
EXT1 and EXT2 are a trunk tagged with vlan 100, vlan tagging disabled
EXT3 and EXT4 are a trunk tagged with vlan 200, vlan tagging disabled
this to me says EXT1 and EXt2 are in access mode for VLAN 100 as vlan tagging is disabled?
this to me says EXT3 and EXT4 are in access mode for VLAN 200 as vlan tagging is disabled?
If you could shed more light on this that would be great.
I have a feeling your problem is probably with the EXT ports as I used to have constant problems with these back when we deployed IBM Blade chassis a lot
SSH into your hosts and check to see if you can do a vmkping from host1 to host2 on the iSCSI network. Your East/West traffic shouldn't have to leave the internal switch so it should work.
Further more we need to make sure your VLAN 200 / 100 route properly all the way to your core
It's been awhile since I have had to work with the IBM Chassis so some of this might be wrong but you get the idea.
Any chance to attach the configuration of BNT switches (removing the sensitive data) ?
Hi JPM399,
I am not really sure if the switches are Nortel, here is the description:
switch-type "BNT Layer 2/3 Copper Gigabit Ethernet Switch Module for IBM BladeCenter"
INT1,2,3... vlan 100 and 200 are tagged, pvid is 100, vlan tagging enabled, tag-pvid is disabled.
For what I know the ports with vlan tagged enable become trunk ports as they are called in cisco swithces, meaning they don´t remove tags. So when you tag the ports with vlans 100 and 200 they onlye let those vlans go through.
As you say, pvid is the vlan that is tagged if the packet has no vlan.
So since your management portgroup doesn't have a VLAN set on it, it will get VLAN 100 due to the pvid being 100.
Yes, the port group doesn´t have vlan, so it gets pvid vlan as it goes through the switch port.
Now when you tag your management group you loose control as I'm assuming this is because vCenter and or your management system lives outside the chassis? If so this tells me that for some reason your EXT1 and EXT2 ports are not passing tagged VLAN's for some reason. I'm not sure what this means exactly:
No, vcenter lives inside blade.
this to me says EXT1 and EXt2 are in access mode for VLAN 100 as vlan tagging is disabled?
this to me says EXT3 and EXT4 are in access mode for VLAN 200 as vlan tagging is disabled?
Yes, both ports should be access
If I tag iscsi porgtroups and ssh to the host, I can´t ping any host on iscsi network. If I connect to the switch I can ping the storage but not the blade.
The idea is to use vst so vlans should be tagged at portgroup level and switches internal ports should be trunk (let all vlans pass).
Here is the switch config
!
spanning-tree mstp name "STP-DC"
!
interface port INT1
pvid 100
fastforward
exit
!
interface port INT2
pvid 100
fastforward
exit
!
interface port INT3
pvid 100
fastforward
exit
!
interface port INT4
pvid 100
fastforward
exit
!
interface port INT5
pvid 100
fastforward
exit
!
interface port INT6
pvid 100
fastforward
exit
!
interface port INT7
fastforward
exit
!
interface port INT8
fastforward
exit
!
interface port INT9
fastforward
exit
!
interface port INT10
fastforward
exit
!
interface port INT11
fastforward
exit
!
interface port INT12
fastforward
exit
!
interface port INT13
fastforward
exit
!
interface port INT14
fastforward
exit
!
interface port EXT1
pvid 100
exit
!
interface port EXT2
pvid 100
exit
!
interface port EXT3
pvid 200
exit
!
interface port EXT4
pvid 200
exit
!
vlan 1
member INT1
member INT2
member INT3
member INT4
member INT5
member INT6
member INT7
member INT8
member INT9
member INT10
member INT11
member INT12
member INT13
member INT14
no member EXT1
no member EXT2
no member EXT3
no member EXT4
member EXT5
member EXT6
!
vlan 100
enable
name "V100-MGMT"
member INT1
member INT2
member INT3
member INT4
member INT5
member INT6
member EXT1
member EXT2
!
vlan 200
enable
name "V200-ISCSI"
member INT1
member EXT3
member EXT4
!
no spanning-tree pvst-compatibility
spanning-tree stp 1 vlan 1
spanning-tree stp 1 vlan 100
spanning-tree stp 2 vlan 200
!
portchannel 1 port EXT1
portchannel 1 port EXT2
portchannel 1 enable
!
portchannel 2 port EXT3
portchannel 2 port EXT4
portchannel 2 enable
!
!
sflow enable
sflow server 10.100.200.130
!
interface port EXT1
sflow polling 20
!
interface port EXT2
sflow polling 20
!
interface port EXT3
sflow polling 20
!
interface port EXT4
sflow polling 20
!
failover vlan
failover trigger 1 amon portchannel 1
failover trigger 1 enable
!
failover trigger 2 amon portchannel 2
failover trigger 2 enable
!
!
!
interface ip 2
ip address 10.100.150.200 255.255.255.0
enable
vlan 200
exit
!
ntp enable
ntp primary-server 10.100.200.82
!
end
Hey,
I forget in this blade chassis how does the INT ports have a relationship with the EXT is it souly based off VLAN settings? Can every INT port leave out any EXT port? I forget how this works in the IBM chassis as its been awhile.
Also from your switch config you only have INT1, EXT3 and EXT4 set with a VLAN of 200, so all your int ports only have pvid 100 or VLAN 100. Meaning if you are putting a VLAN tag of 200 on your port group it will drop it once it hits the internal ports as they only have pvid of 100. Since you want to do all of your tagging on your Port groups this will mean all traffic hitting your internal / ext ports will already have a VLAN tag on them. So your INT ports will need to be trunked with VLAN100/200 same with your EXT ports. Also if you only want certain INT / EXT ports to only have certain VLANs then you would trunk just the VLAN you want for that port.
With this switch aside from the VLAN membership do you have to set the options on the ports as well or does the membership do it for you?
You have two setup options here:
Figure out which INT port each nic on the blades hard wires to and setup pvids for the proper vlans to route everything correctly. This would mean you would not set any VLANs at the port group level.
or
Trunk all the INT ports for the proper VLAN along with the EXT ports as the VLAN tag will be coming from the vSwitch/Portgroup from VMware.
If you have any other questions please let me know.
Here is another example of how the trunk ports would work.
As your INT ports are the first ports any traffic from your blades hit. If they where to never route out of the blade chassis, they would stay within the INT ports on that switch. When traffic has to route out it will use the EXT ports. In which case all VLANs required out will need to be passed long to your core switch. From there your storage node will connect to the switch but seeing as most SANs typically don't set VLANS from the SP's you would put the ports your SAN connects into in access mode or PVID200. This would then be able to talk to the traffic on VLAN 200. Or should.
I hope some of this has helped.
You will also need to find the IBM redpaper on how the blades nic's map to the INT ports on the internal switch/switches
Did some thinking on your switch settings the other night as well.
It appears your switches are like HP's where you can have a pvid/access mode port(meaning untag traffic hits this port and the switch places the pvid VLAN on it) and also tag a vlan on this as well.
For instance:
port 1:
pvid 100
tag 200
fastforward
In this case if anything plugged into this port would get VLAN 100, however if this port see's a packet come into this port with a VLAN tag of 200 it will route it accordingly. I know some switches allow this and it bugs me, but its an option.
Going back to what I said, and looking at your switch settings, it looks like you will either need to figure out how your NIC's in your blade map to the INT ports and then use the appropriate pvids, and not use VLAN's on your port groups
or
continue to use pvids and tag some of the ports required for the iSCSI network like I discussed at the top
or
switch the ports over to Trunk mode and pass only the VLAN's required then tag all your traffic from each port group. Aka MGMT VLAN 100, iSCSI VLAN 200
here is some links as well that may help:
Tag/Untag Ports on HP Switches - Spiceworks
http://networkengineering.stackexchange.com/questions/6483/what-are-tagged-and-untagged-packets
anyhow hope this has helped
Hi JPM300,
What I know about pvid tagging is that if anything comes with pvid vlan then vlan tag is removed.
vlan 100 tagging -> port pvid 100 with tagging-pvid enabled -> forward without tagging
I found somthing it might be the problem:
http://delivery04.dhe.ibm.com/sar/CMA/XSA/03mph/0/elx_fw_cna_ibm1206-4.2.433.3-1_linux_32-64.chg
Regards
Here is the answer:
There´s a bug in Emulex driver version VMware_bootbank_net-be2net_4.1.255.11-1vmw.510.0.0.799733 that stops nic from seeing vlans.
I downloaded the Drivers and install it on one hosts and EUREKA. All works now.
IBM Bladecenter H Nic on Vmware
Thanks for your help.
Lol,
Ohh jeez. Well good find!
I have seen some odd bugs from emulex before. In one of our environments the cards loose the iSCSI discovery portal after every reboot.... there still is no fix :smileysilly:
Thanks for posting the fix