Cannot pass Tagged traffic from vmware to Virtual Fabric 10 gb switch

[nevilleonline]

Hi ,

I need to understand how to pass tagged traffic from vmware VST to these Virtual Fabric switch. The IBM HS22 blades connect internally to the Virtual switch from ports 1 to 14. I am using 2 external ports (17-18) one connects to the Netgear switch and another to the other Virtual fabric switch. Same is done on the other Virtual Fabric switch. My Synology rackstation is configured with iscsi luns that connect to the Netgear switch and I would like to connect my rackstation to the HS 22 blades. My main concern is i cannot ping to the netgear IP interface on the same vlan. I can ping from my Synology diskstation to the netgear which are in the same vlan. Netgear and BNT switches are connected through DAC SFP+ cables.

Same vlans are also configured on Netgear switch. The pvid by default is set as 1 on all the interfaces, should I disable this ?. Do I need to use tagpvid-ingress on all the interfaces.

sh run

Current configuration:
!
version "7.8.7"
switch-type "IBM Networking OS Virtual Fabric 10Gb Switch Module for IBM BladeCenter"
iscli-new
!
system timezone 295
! Europe/Denmark
system daylight
!

snmp-server name "BNT01"
!
hostname "BNT01"
!
!
access userbbi enable
!
interface port INT1
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT2
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT3
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT4
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT5
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT6
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT7
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!

interface port INT8
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT9
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT10
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT11
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!

interface port INT12
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT13
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port INT14
switchport trunk allowed vlan 1,16-50,3998-4000,4095
exit
!
interface port EXT1
switchport mode trunk
switchport trunk allowed vlan 1,16-50,3998-4000
exit
!
interface port EXT2
switchport mode trunk
switchport trunk allowed vlan 1,16-50,3998-4000
exit
!

!
vlan 1
name "Default"
!
vlan 16
name "VLAN16"
!
vlan 17
name "VLAN17"
!
vlan 18
name "VLAN18"
!
vlan 19
name "VLAN19"
!
vlan 20
name "VLAN20"
!

.

.

.

.

.

.

..

vlan 46
name "VLAN46"
!
vlan 47
name "VLAN47"
!
vlan 48
name "VLAN48"
!
vlan 49
name "VLAN49"
!
vlan 50
name "VLAN50"
!
vlan 3998
name "iscsi"
!
vlan 3999
name "vmotion"
!
vlan 4000
name "mgmt"
!
!
!
spanning-tree mst configuration
name "lethosting-region1"
revision 2
exit
!
spanning-tree mode mst
!
spanning-tree mst configuration
instance 1 vlan 16-50
instance 2 vlan 3997,4000
instance 3 vlan 3998-3999
exit

The configuration is for pNIC. I donno what I am missing here? Any ideas will be really appreciated.

[aguerriero]

How are you doing NIC teaming for each vmk? You should only have one active adapter per VMK and the other setup as unused. Or you have two different standard switches each with its own vmk and adapter. If you have both vmnics as active in the vmk settings then the switch will want a non LACP ether channel setup. Without the ether channel or vmnic specification you are probably getting MAC address flaps between your two ports on the switch.

vmware says not to use ether channels for iSCSI though.

[cjscol]

INT1-14 require

switchport mode trunk

[nevilleonline]

I had already tried that. The command executes but doesn't appear in show run configuration file. By default INT 1-14 are tagged to vlan 1 and 4095. I have added the rest of the vlans.

The BNT switches are upgraded to the latest firmware 7.8.

Switch# configure terminal

Switch# interface port 1-14                                    By Default Tagging is enabled for vlan 1 and 4095 on internal ports 1 to 14

Switch (config-if)# tagging

Switch# ^Z

Switch# configure terminal

Switch# interface port 17-18

Switch (config-if)# tagging

Switch# ^Z

Switch# configure terminal

Switch(config)# vlan 3998

Switch(config-vlan)# member 1-14,17-18

Switch(config-vlan)# name iscsi

Switch(config-vlan)# enable

Switch# configure terminal

Switch(config)# vlan 3999

Switch(config-vlan)# member 1-14,17-18

Switch(config-vlan)# name vmotion

Switch(config-vlan)# enable

Switch# configure terminal

Switch(config)# vlan 1,16-50

Switch(config-vlan)# member 1-14,17-18

Switch(config-vlan)# enable

Switch# configure terminal

Switch(config)# vlan 4000

Switch(config-vlan)# name mgmt

Switch(config-vlan)# member 1-14,17-18

Switch(config-vlan)# enable

[nevilleonline]

I have 2 BNT switches and each adapter is connected to a different switch. You are right one there should be one active adapter and one inactive. Thats what is done.

Is there any way to check if the vmware traffic is even arriving at IBM Virtual Fabric Switches.

[aguerriero]

Can you create a vlan interface 3998 and put it in the same subnet as the VMK. Then try to ping the storage and the host interfaces from the switch.

[cjscol]

Are the ports he BNT modules are connected to on the Netgear configured as trunked ports with all of the VLANs allowed on them?

[nevilleonline]

Yes,first BNT Switch is connected to port 11 of Netgear and second BNT switch is connected to port 12.

interface xg11

no auto-negotiate

speed 10G full-duplex

spanning-tree bpduflood

spanning-tree port mode

vlan participation include 16-50,3998-4000

vlan tagging 1,16-50,3998-4000

exit

interface xg12

no auto-negotiate

speed 10G full-duplex

spanning-tree bpduflood

spanning-tree port mode

vlan participation include 16-50,3998-4000

vlan tagging 1,16-50,3998-4000

exit

[nevilleonline]

@aguerriero yes I have tried with vlan 1, 16-50 and 3998-4000. I slowly getting the feeling its some config I am missing in BNT switches. I have got vmware tagging working with brocade switches before, but BNT switches need something more.

I have this working with Brocade switch where the port to which this interface is connected is tagged to vlan 4000. But this is on a IBMx3850 server ,not a HS22 blade.I have done similar thing with vlan 3998 on the HS 22 blades. Does anyone you know have experience in configuring BNT or IBM virtual Fabric Switch module in Physical NIC mode with ESXi 6.0 will be helpful.

[cjscol]

It has been a while since I have worked with IBM Bladecenter but from what I remember there is an option in the Bladecenter management interface to enable or disable the external ports on the I/O modules. Have a look at the Bladecenter management and check for this option.I think it is under I/O Module Tasks -> Admin/Power/Restart

[nevilleonline]

@cjscol Yep those are enabled too for both the BNT switches. All the ports are up on the BNT switches.

[aguerriero]

VMware Compatibility Guide: System Search

It appears that the IBM HS22 last supported ESXi version is 5.5U3.

[aguerriero]

HS22V ESXi 5.5 Upgrade 10Gb Network Problem | vcdx133.com

Could be the same problem this guy is experiencing. Is your driver the same version listed?

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI60-EMULEX-ELXNET-10425513&productId=...

[nevilleonline]

@aguerrieroaguerriero Interesting, I think maybe this is it, that this particular blade (HS22) is not supported plus I found out that Emulex OCm10102-F-X iSCSI/FCoE Virtual fabric adapter (firmware version 4.6.281.21) is not supported by ESXi 6.0 from their support.Looks like this is a very old adapter and not manufactured anymore. I am now trying to find out which is the last ESXi version this adapter works with and to which firmware code version this can be updated to.

[nevilleonline · Accepted solution]

Yesssss. Finally got it to work. The tagged traffic now runs from Blades with ESXI 5.5 U2 to the Synology rackstation.

It was the same thing that we suspected. ESXi 6.0 is not supported by this emulex adapter. Also ESXi 5.5 does not come I think with the iSCSI driver. So I updated the drivers using esxcli.

VMware

Update Network driver

/tmp # esxcli software vib install -v /tmp/elxnet-10.0.575.9-1OEM.550.0.0.1331820.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Emulex_bootbank_elxnet_10.0.575.9-1OEM.550.0.0.1331820
VIBs Removed: VMware_bootbank_elxnet_10.0.100.0v-1vmw.550.0.0.1331820
VIBs Skipped:

Update iSCSI driver

/tmp # esxcli software vib install -v /tmp/scsi-be2iscsi-4.6.261.0-1OEM.550.0.0.1198611.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Emulex_bootbank_scsi-be2iscsi_4.6.261.0-1OEM.550.0.0.1198611
VIBs Removed:
VIBs Skipped:
/tmp # esxcli software vib install -v /tmp/ima-be2iscsi-4.6.261.0-1OEM.550.0.0.1198611.i386.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Emulex_bootbank_ima-be2iscsi_4.6.261.0-1OEM.550.0.0.1198611
VIBs Removed:
VIBs Skipped:

esxcli system maintenanceMode set -e true
esxcli system shutdown reboot -r=driverupdate -d=10

After this i created the iSCSI vmkernel ports with teaming. Ping started working and now I can connect to storage