Problems with iSCSI Multipath I/O

[justgraham]

Hi Folks,

I've just started testing with ESXi 4 and have hit a snag with the Software iSCSI Adaptor and Multipath I/O.

From a bit of background reading, I understand that the storage architecture is a little different to 3.5, with the new PSA.

Anyway, here's the set-up.

Single ESXi 4.0 Host

- vSwitch1

- Two physical uplink NICs

- Two port groups, with a VMKernel interface in each (See attachment 'Storage4-Network.jpg')

- 10.42.80.107 & 10.42.80.108

- The adaptor order has been over-ridden in the port groups so that only 1 adaptor is active in each respective group

- Added both of the VMKernel interfaces (vmk1 & vmk2) to the iSCSI Adaptor, using the 'esxcli swiscsi nic' commands

- Basically, everything described in the VMWare "iSCSI SAN Configuration Guide"

Thecus R4500i iSCSI SAN

- Two physical NICs

- Each physical NIC has it's own IP Address

- 10.42.80.200 & 10.42.80.201

So in this configuration, I should end up with 4 paths:

ESXi 10.42.80.107 - SAN 10.42.80.200

ESXi 10.42.80.107 - SAN 10.42.80.201

ESXi 10.42.80.108 - SAN 10.42.80.200

ESXi 10.42.80.108 - SAN 10.42.80.201

Which is what I get if I look in "Configuration\Storage Adaptors\iSCSI Software Adaptor" (see attached 'Storage2-Paths.jpg').

But here's the problem, when I take a look in "Configuration\Storage\Devices\iSCSI Software Adaptor\Manage Paths", I only get a single path (see attached "Storage3-Paths.jpg").

Can anyone help explain this? Have I missed something?

This same set-up (Servers, NICs, SAN, etc) is all fine with multiple paths in ESXi 3.5

(And yes, I know that the 'Thecus' isn't in the supported HCL, but we're just talking standard iSCSI here, no additional Storage Plugins, etc)

Not uber-urgent, but annoying, as this lack of multipathing would stop a deployment of vSphere.

Thanks,

Graham.

[rnp2000]

Excellent Thread. Graham, your diagrams in the attachment Understanding iSCSI.pdf were very clear and useful. Could you briefly explain the comment on page 5: "Have to use static iSCSI targets". What happened when you tried to use dynamic discovery for the targets?

Thanks,

Rich

[justgraham]

Hi Rich

Good question, it's not very clear is it!

So the background was that I was testing both our production Thecus i4500R (see earlier in the thread) and FreeNAS.

In this scenario the storage has 2 IP addresses in different subnets.

With the Thecus storage, if I entered one of the IP Addresses in Dynamic Targets, ESXi would discover both IP Addresses / Targets on the storage. I.e. they'd appear in the Static Targets tab.

With FreeNAS, this didn't work. If I entered one of the IP Addresses in Dynamic Targets, ESXi would only discover that IP Address / Target. Therefore to get the other path, I entered it manually in the Static Targets tab. In hindsight I could have also tried entering the second IP Address in Dynamic Targets, which would probably have worked too.

Glad you found it useful. Just remember that it's not gospel; try the scenarios against your own storage to be sure.

Cheers,

Graham.

[rnp2000]

Thanks Graham,

I have found similar differences in how different iscsi devices respond to dynamic target discovery. I have recently tested with a Netapp and Openfiler.

For NetApp, when I specified a single IP address as a send target using dynamic discovery, all (3) of the IP portal addresses where discovered; however on the Openfiler device I had to explicitly include each IP portal address in the dynamic discovery configuration list.

I did not test at this time as to whether difference is due to the vendor implementation differences or whether it has to do with configuration differences, such as the mapping between IP portal addresses and iscsi targets. For Netapp test, I had three portal addresses tied to the same iscsi target. For Openfiler test, I had two iscsi targets (managing the same set of luns) each with its own IP portal address

-Rich

[DSTAVERT]

Since both the Thecus and the Netapp devices support it have a look at NFS. Since NFS is a very light weight protocol you may end up with a little better performance. You can view the same storage volume from multiple ESXi hosts which means you also have the added DR benefits. It becomes easy to pause a VM on one host and imediately unpause on the second host. Great for maintenance or load balancing.

[justgraham]

Thanks for the suggestion.

Unfortunately in my specific case, the Thecus that we're using (the i4500R) only supports iSCSI. It also doesn't support Etherchannel / Link Aggregation.

In hindsight, it possibly wasn't the best purchase we've ever made! But no worries, we'll soon be moving to NetApp FAS940, so a whole new world of opportunity will open up, which will include NFS.