error messages for non-existing NFS datastore in vmkernel.log

[casaub]

Hello forum,

a while ago we re-configured our network. Part of that was the migration of a NFS device/datastore from a public network to a private network.

We re-connected that NFS share as datastore in vCenter using the new, private  IP address and it is working fine. But ESXi hosts still 'remember' the old, unaccessible <public-IP> of it. We removed the old lines of the NFS share already from esx.conf but we see these errors in the vmkernel.log continous and very often:

2022-03-10T06:57:53.699Z cpu5:2097603)NFS: 171: NFS mount <public-IP>:/Backup failed: Unable to connect to NFS server.

2022-03-10T06:58:03.652Z cpu9:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7045 attempt 1 of 3
2022-03-10T06:58:13.652Z cpu17:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7068 attempt 2 of 3
2022-03-10T06:58:23.652Z cpu17:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7075 attempt 3 of 3

2022-03-10T08:11:03.664Z cpu4:2098401)CpuSched: 699: user latency of 2103653 RPC-tx-<public-IP>.0.111 0 changed by 2098401 NFSv3-RemountHandler -6

Because that public network is not accessible anymore, we cannot migrate that NFS share/datastore back to <public-IP> (and remove it again from ESXi hosts/vCenter).

For the NFS share with <public-IP> we had a separet TCP/IP stack, separate vmk (vmk1) and separate vSwitch. We deleted that vmk1. TCP/IP stack and vSwitch still exist, both are unused.

What could be the source of these error messages? Were else could the ESXi host take the <public-IP> from?

The ESXi host is running version 6.7.0 Update 3 (Build 19195723). vCenter is a vCSA 6.7.0 build 19299595.

Thank you.

[alantz]

Since /Backup is listed maybe its vcsa scheduled backup ?

--Alan--

[casaub]

Great idea.

Was checking that - but unfortunately no, we have scheduled tasks for backup, but using internal storage of ESXi host now (to avoid data loss in case the NFS device is updating firmware automatically).

[alantz]

Do you see it with esxcli storage nfs list ?  Maybe the mount just needs removed.

--Alan--

[casaub]

Command esxcli storage nfs list is showing one NFS share, which is /Backup, but with the new, private IP address. We re-use the same NFS device:

esxcli storage nfs list
Volume Name Host Share Accessible Mounted Read-Only isPE Hardware Acceleration
------------------ --------------- ------- ---------- ------- --------- ----- ---------------------
NAS    <private-IP> /Backup true true false false Not Supported

Is there a possibility to search ESXi configuration (files) for <public-IP> or the client (0x430589d63180)?

What could lead to that SunRPC tries to access <public-IP>?

[alantz]

See how that compares with this command.

esxcfg-nas -l

Not sure how to get rid of that stale entry. Maybe someone else can jump on here and say how they have done it.

--Alan--

[casaub]

The output of that command confirms that there is only on NFS share:

esxcfg-nas -l
NAS is /Backup from <private-IP> mounted available

Thank you, Alan.

[DeviVmware]

Hi @casaub ,

Is it possible for you to unmount current nfs from esxi host and mount again ?

[casaub]

Hello,

we will try that. And report back.

[casaub]

Looks like this and a reboot fixed it. No <public-IP> in vmkernel.log anymore. Thank you.