We have been using a couple of cheaper iSCSI SANs for a while now as a low-tier compliment to our FC SAN. This has been running well up until recently. As we needed more storage at a higher tier, and also because our FC SAN will be end of life at the end of this year we decided to install an Equalogic PS4110x with its own switching hardware. This converges at an HP c7000 blade environment. The Equalogic SAN is on a new VLAN and as such has two new vmk ports that are bound to specific physical nics to get multipathing. The old iSCSI SAN storage has a single vmk port on a different VLAN. Neither of the iSCSI VLANs have any routing.
I use port binding on 2x ESXi 5.1 hosts each with 2 vmk ports for iSCSI each on a different broadcast domain/subnet without issue. I do manually enter the iSCSI targets on the ESXi hosts though rather than dynamic discovery so they only know about targets they can reach.
Problem resolved, in case anyone is interested:
The resolution was to also bind the original vmk port to a nic. Once I had discovered that this solved the problem, logic dictated the creation of a second vmk port to be bound to the other nic connected to the vswitch. So I now have 4 vmk ports servicing two iSCSI VLANs - time to move the old storage on to the new iSCSI network.
