VMware Cloud Community

ESXi 5.0 Cluster Utilizing Two iSCSI SANs, Some iSCSI Routing Involved

Looking for advice or recommendations.  Here's the setup:

We had a vSphere 4.1 implementation, with a cluster consisting of four ESXi 4.1 hosts that utilized two different SANs.  The first SAN is an Equallogic PS5XXX series that we own.  The second SAN is a Falconstor NSS appliance from which the hosts access two LUNs that we "lease" from a larger campus department.  Each of the ESXi hosts had four vmkernel ports dedicated for iSCSI.  All of these port groups use the same vSwitch.  There are four vmnics utilized by the vSwitch, with a one-to-one mapping for the vmnic to the port group.  Cue the esxcfg-vmknic output:

~ # esxcfg-vmknic -l
Interface  Port Group/DVPort      IP Family IP Address            Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type
vmk1       Software iSCSI3          IPv4      172.17.28.xxx 172.17.28.xxx    00:50:56:72:3d:95 1500    65535     true    STATIC
vmk2       Software iSCSI1          IPv4      172.16.52.xxx    172.16.52.xxx   00:50:56:75:fc:66 1500    65535     true    STATIC
vmk3       Software iSCSI2          IPv4      172.16.52.xxx    172.16.52.xxx   00:50:56:7e:54:0f 1500    65535     true    STATIC
vmk4       Software iSCSI4          IPv4      172.17.28.xxx 172.17.28.xxx    00:50:56:7b:92:0d 1500    65535     true    STATIC

The esxcfg-vswitch output:

~ # esxcfg-vswitch -l


Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1             128               9                128                   1500      vmnic3,vmnic7,vmnic4,vmnic5

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Software iSCSI4          0            1                 vmnic5
  Software iSCSI2          0            1                 vmnic3
  Software iSCSI1          0            1                 vmnic7
  Software iSCSI3          0            1                 vmnic4


Interfaces vmk2 and vmk3 are used to connect to the Equallogic, and all devices are connected the same subnet (  Interfaces vmk1 and vmk4 are used to connect to the Falconstor.  These interfaces (subnet 172.17.28.xxx/27) are not on the same subnet as the Falconstor (subnet, so the iSCSI traffic is routed.  In order to connect to the Falconstor, I added a route with esxcfg-route:

~ # esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
172.17.28.xxx  Local Subnet     vmk1    Local Subnet     vmk6    Local Subnet     vmk2    Local Subnet     vmk0    172.17.28.xxx     vmk1    <<== added here
default     vmk0

All the vmkernel ports were bound to the ESXi software iSCSI inititiator, which allowed us to multi-path to each SAN.  Now, I am aware that iSCSI port binding does not support routing (http://kb.vmware.com/kb/1009524), however, under 4.1 it seemed to work.  Earlier this week, I began to upgrade to vSphere 5.0 update 2, and after performing an upgrade in place of one of the ESXi hosts, I found that the connections to the Equallogic came back fine, but the connection to the Falconstor did not.  In checking the new "Network Configuration" tab under the Software iSCSI Initiator properties window, I see that vmk1 and vmk4 show a "Path Status" of "Not Used".  The route I added is still in place.  I double-checked the connection properties for the Falconstor iSCSI target (CHAP is used, so I re-entered the secrets).  So I'm left with the fact that it really now is the case that routing isn't supported in this configuration.

I'm wondering what my configuration options are right now, assuming we will still want to utilize the storage from the Falconstor?  The physical nics are Broadcom 5709, which support TOE and show as hardware iSCSI adapters.  All hosts are licensed for vSphere Enterprise (so no dvSwitches).  I don't believe there is any flexibility for changing adapter subnets in order to avoid having to route (the hardware is located in a datacenter that my department does not manage).

Can the two vmkernel ports used for the Falconstor be unbound from the software initiator while the other two used for the Equallogic remain bound?  Can the hardware initiator be used in conjunction with the software initiator?  I'd like to be able to continue to take advantage of multipathing to both SANs, or will that be lost?


0 Kudos
0 Replies