a while ago we re-configured our network. Part of that was the migration of a NFS device/datastore from a public network to a private network.
We re-connected that NFS share as datastore in vCenter using the new, private IP address and it is working fine. But ESXi hosts still 'remember' the old, unaccessible <public-IP> of it. We removed the old lines of the NFS share already from esx.conf but we see these errors in the vmkernel.log continous and very often:
2022-03-10T06:57:53.699Z cpu5:2097603)NFS: 171: NFS mount <public-IP>:/Backup failed: Unable to connect to NFS server.
2022-03-10T06:58:03.652Z cpu9:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7045 attempt 1 of 3
2022-03-10T06:58:13.652Z cpu17:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7068 attempt 2 of 3
2022-03-10T06:58:23.652Z cpu17:2098401)SunRPC: 3303: Synchronous RPC abort for client 0x430589d63180 IP <public-IP>.0.111 proc 3 xid 0x7d8d7075 attempt 3 of 3
2022-03-10T08:11:03.664Z cpu4:2098401)CpuSched: 699: user latency of 2103653 RPC-tx-<public-IP>.0.111 0 changed by 2098401 NFSv3-RemountHandler -6
Because that public network is not accessible anymore, we cannot migrate that NFS share/datastore back to <public-IP> (and remove it again from ESXi hosts/vCenter).
For the NFS share with <public-IP> we had a separet TCP/IP stack, separate vmk (vmk1) and separate vSwitch. We deleted that vmk1. TCP/IP stack and vSwitch still exist, both are unused.
What could be the source of these error messages? Were else could the ESXi host take the <public-IP> from?
The ESXi host is running version 6.7.0 Update 3 (Build 19195723). vCenter is a vCSA 6.7.0 build 19299595.
Since /Backup is listed maybe its vcsa scheduled backup ?
Was checking that - but unfortunately no, we have scheduled tasks for backup, but using internal storage of ESXi host now (to avoid data loss in case the NFS device is updating firmware automatically).
Do you see it with esxcli storage nfs list ? Maybe the mount just needs removed.
Command esxcli storage nfs list is showing one NFS share, which is /Backup, but with the new, private IP address. We re-use the same NFS device:
esxcli storage nfs list
Volume Name Host Share Accessible Mounted Read-Only isPE Hardware Acceleration
------------------ --------------- ------- ---------- ------- --------- ----- ---------------------
NAS <private-IP> /Backup true true false false Not Supported
Is there a possibility to search ESXi configuration (files) for <public-IP> or the client (0x430589d63180)?
What could lead to that SunRPC tries to access <public-IP>?
See how that compares with this command.
Not sure how to get rid of that stale entry. Maybe someone else can jump on here and say how they have done it.
The output of that command confirms that there is only on NFS share:
NAS is /Backup from <private-IP> mounted available
Thank you, Alan.
we will try that. And report back.
Looks like this and a reboot fixed it. No <public-IP> in vmkernel.log anymore. Thank you.