vcenter backup to nfs fail (no such device)

[ojw_nc]

Trying to configure vCenter 7 backup to NFS fails with the following error in logfile /var/log/vmware/applmgmt/applmgmt.log:

2021-02-11T17:03:18.223 [2712]DEBUG:vmware.appliance.backup_restore.schedule_impl:Validating schedule spec.
2021-02-11T17:03:18.243 [2712]ERROR:plugins.MountedFSStorageIOLib:Cmd ['/usr/bin/mount', '-t', 'nfs', '(IP):/mount/point/', '/storage/remote/backup /nfs/(IP)/rx3sIz27/ryH2KfMS', '-o', 'hard,intr'] failed. rc: 32, stdout: , stderr: mount.nfs: No such device
2021-02-11T17:03:18.244 [2712]INFO:plugins.MountedFSStorageIOLib:Failed to mount the nfs share (IP):/mount/point/ at /storage/remote/backup/nfs/(IP)/rx3sIz27/ryH2KfMS. Err- rc: 32, out=, err=mount.nfs: No such device

The same error occurs trying manually with mount -v ...

Interestingly, the modules present don't seem to correlate to the kernel version running:

root@172 [ /mnt ]# uname -a
Linux (IP) 4.19.115-1.ph3 #1-photon SMP Tue Apr 21 18:56:03 UTC 2020 x86_64 GNU/Linux

root@172 [ /mnt ]# ls -l /lib/modules/
total 8
drwxr-xr-x 2 root root 4096 Feb 10 18:53 4.19.115-1.ph3
drwxr-xr-x 4 root root 4096 Feb 10 18:50 4.19.148-5.ph3

root@172 [ /mnt ]# ls -l /lib/modules/4.19.115-1.ph3/
total 0

root@172 [ /mnt ]# find /lib/modules/ | grep nfs
/lib/modules/4.19.148-5.ph3/kernel/drivers/xen/xenfs
/lib/modules/4.19.148-5.ph3/kernel/drivers/xen/xenfs/xenfs.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/nfs.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/nfsv2.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/nfsv3.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/blocklayout
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/filelayout
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/nfsv4.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/flexfilelayout
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfsd
/lib/modules/4.19.148-5.ph3/kernel/fs/nfsd/nfsd.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs_common
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs_common/nfs_acl.ko.xz
/lib/modules/4.19.148-5.ph3/kernel/fs/nfs_common/grace.ko.xz

 Is this expected to work; is it possible to fix?

[vXav]

Can you show a screenshot of the configuration page of the backup in the VAMI?

Also, have you checked that kb? https://kb.vmware.com/s/article/80570

[ojw_nc]

Thanks very much for your reply. The error from the web interface is not especially informative:

I can see from the shell that such is indeed exported:

root@172 [ /mnt ]# showmount -e 172.19.NFS.IP | grep vcb
/opt/vcb   172.19.VC.IP

[vXav]

Have you tried putting something (anything) in the user/password fields? I know in 6.7 I had to I would just put "a" and "a", silly but...

[ojw_nc]

Thank you for the suggestion; I have tried that now but the behaviour is the same.

This page and others suggest that the problem is the kernel modules missing: https://wiki.archlinux.org/index.php/NFS/Troubleshooting#mount.nfs4:_No_such_device

[vXav]

I compared with my vcsa7 in my lab and your modules look ok.

I tried a quick and dirty backup on an NFS share hosted on freenas which worked after I set the maproot user field.

Are you able to mount the share with :

mkdir /mnt/test
mount -t nfs 192.168.10.13:/mnt/vcsa /mnt/test

If yes are you then able to write to /mnt/test ?

[Lalegre]

Hey @ojw_nc,

If you are not even able to mount it manually please check the next first:

• That you can reach over L2 or L3 to the NFS server.
• That you have port 111 and 2049 TCP and UDP allowed.
• That you do not have any ACL on the NFS Server.
• That you have r/w permissions on the share to an specific user or to everyone (in case you did not specify user)

Once you solve the issue of mounting it manually then maybe the backup will work.

[ojw_nc]

I  think the showmount indicates the connectivity is there - or else what commands are best to check?

Manually I get the same error:

root@172 [ /mnt ]# mount -t nfs -v 172.19.NFS.IP:/opt/vcb temp
mount.nfs: timeout set for Thu Feb 18 18:56:48 2021
mount.nfs: trying text-based options 'vers=4.2,addr=172.19.NFS.IP,clientaddr=172.19.VC.IP'
mount.nfs: mount(2): No such device
mount.nfs: No such device

root@172 [ /mnt ]# mount -t nfs -o v3 -v 172.19.NFS.IP:/opt/vcb temp
mount.nfs: timeout set for Thu Feb 18 18:56:54 2021
mount.nfs: trying text-based options 'v3,addr=172.19.NFS.IP'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.19.NFS.IP prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 172.19.NFS.IP prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): No such device
mount.nfs: No such device

Other servers in the same subnet can mount the NFS share sucessfully.

[Lalegre]

If you check on /var/log/messages do you see any other entry?

[vXav]

What about the logs on the NFS server. Anything at all ?

[ojw_nc]

There was nothing else useful in the logs (on either side).

To followup on this, for the benefit of others in future: after the appliance was rebooted [before upgrading to 7.0.2] it began to work!