NFS 4.1 and Multipathing

[tomsmig]

Hi, we are trying to mount PowerScale (former: Isilon) storage to esxi hosts with multipathing.

We have 3 ESXi nodes, each has one VMkernel adapter connected to 3 external IPs from PowerScale.

After mounting new NFSdatastore for the first 5 minutes we have 3 TCP sessions established to the coresponding nodes.

After 5 minutes, two sessions are dropped. The only session that stays is the one established to first node on the list.

From vmkernel.log:

2022-08-24T20:33:31.569Z cpu62:2130292 opID=95111ab7)NFS41: NFS41_VSIMountSet:417: datastore_ssd1 mounted successfully
2022-08-24T20:38:31.570Z cpu12:2098629)NFS41: NFS41_ConnectionRemove:1091: Connection: 0x4316aa41f9b0 [83] addr: tcp 10.44.174.202.8.1
2022-08-24T20:38:31.572Z cpu12:2098629)WARNING: NFS41: NFS41DestroyClientIDNFSProcess:2136: DESTROY_CLIENTID error: NFS4ERR_STALE_CLIENTID

Therefore no multipathing is working.

[NilangaK]

Hi Tomsmig

Did you get this issue resolved. Running into the same issue. How did you get Multipath working. I have tested with a Windows NFS server and works as expected but not powerscale.

[connecttoduy]

Hello

If you want enable multipath on PowerScale , you need to using FQDN for storage - Smart Connect featured, and FQDN will round robin on 3 IPs

Doc ref : https://infohub.delltechnologies.com/l/dell-emc-isilon-storage-with-video-management-systems-configu...

Regards, 

Duy

[NilangaK]

Hi Duy

I tried using Smartconnect FQDN but it creates multiple datastores.

I have 4 ESXI nodes in a cluster. When I add each of the hosts to the datastores, if the smartconnect returns a new IP address when the host looks up the FQDN SSIP, it creates a new Datastore and associates the host with the IP address attached to the datastore. See attached file. 

How did you get around this issue? Also tried using the SmarConnect IP (instead of the FQDN) and it errors. I don't believe accessing via the IP address is supported. 

[btechit]

This is probably a Dell EMC support ticket, assuming vmware is a supported Host. 

[tomsmig]

From what I remember from the times of implementation, multipathing works only if IP switches between NICs within one node.

If IP switches to another node in the cluster, multipathing does not work.

What else came out was that "Session Persistence" in NFSv 4.1 is "diferently" implemented in PowerScale than we expected.

What it means is that PowerScale presents the same shares as different objects in vSphere.

It was confirmed with DellEMC support, but I do not have official clear statement about this.

That's why we returned to NFSv3.

I think you are facing similar situation, and should double check with DellEMC.

[connecttoduy]

Hello How to step you add datastore to VMware cluster ? And just using FQDN for NFS while you adding. Regards, Duy

[tomsmig]

We tried both.

[NilangaK]

We tried both SSIP FQDN as well as IP address without any luck. Would love to hear anyone who got SSIP to work.

The storage team enabled Dynamic IP address on each of the 4 nodes on the PowerScale cluster. We added 4 separate datastores with separate NFS shares. It's not ideal but it still allowed some level of High availability (across ports) and to spread the throughput across the PowerScale nodes.