ESXi 7.0 U2 - SunRPC: Socket space full. rpc queued.

[crispyduck00]

Hi,

I am playing around with NFS4.1 trunking. So far I had a older ESXi 6.7 U3 host connected to the storage and after eliminating some issues it seems to work fine.

Now I added another Server with 7.0 U2 to the same storage exactly the same way (same config), where I see in disk benchmarks from inside a VM that small read writes are a bit slower and I also get following vmkernel.log messages:

2022-02-28T15:00:20.573Z cpu15:262826)SunRPC: 4727: Socket space full. rpc queued. 0x110692ad
2022-02-28T15:00:37.824Z cpu18:262826)SunRPC: 4727: Socket space full. rpc queued. 0x1107a352

Couldn't find anything to this message. Have never seen it on 6.7. Is it just a new leg message as also the NFS performance monitoring?

I compared the advanced settings and also played around a bit with them, but still bit slower on small read/writes and always this message in the log.

Only SunRPC.SetNoDelayedAck 1 brought a bit better read, but still not as good as 6.7 and still the log messages.

Network is ident.

Any Ideas? 

Br
cd

[crispyduck00]

Hi, can someone just explain why this is available on 7.0 u2 and not on 6.7 u3 when both hosts have the same config?

Br,

cd

[crispyduck00]

No one else?

[TRottig]

Have you ever solved this?

I get  SunRPC: 4727: Socket space full. rpc queued. 0x71ed51b3  too on ESXi-7.0U3sd-19482531-standard.

This is accompanied by abysmal read performance (1.8 MB/s on a 6 vdev mirror ZFS box with SAS3 SSDs).

Its a multipathed config too, with Chelsio T62100 100G nics in iWarp mode on both sides

Write performance is okish, nothing to write home about though, i think its significantly better in 6.7 (but have not verified on same hw).

[Martjasek]

Hi,

maybe it's already solved, but I would check NFS MaxQueueDepth directly on that NFS datastore. In version 7 there was an issue that even if you set ESXi advanced parameter to some value it wasn't properly set on datastore itself.

esxcli storage nfs param get -v all

Martin

[jwckauman]

I'm troubleshooting an issue with ESXi 7.0 U3d and random 'all paths down' warnings from all our hosts (attempting to connect to our datastores).  I've checked all our host advanced settings and they match NFS best practices, and we've updated all our network adapter drivers and firmware, but the issue continues to persist.  One of the warnings I'm seeing in the vmkernel.log brought me to your post.  

SunRPC: 4727: Socket space full. rpc queued. 0x1ee84601

I have no idea what it means,  but wanted to join in the convo to see what you had found or if you had managed to get this warning to go away.