Fedora 11 with recent updates ESXi can not connect to via NFS?

[Schorschi]

I am posting this issue in hopes that someone else has seen it. In our lab, for experimental work, we use Fedora 10 and 11 to serve NFS. This is not for serious work, but a sandbox environment. With the latest updates to Fedora 11 proper (not rawhide) ESXi 3.5 U4 and ESXi 4 can not establish a lock via nfslock. Fedora 10 works, Fedora 11 straight off the media or initial build release, work. The connection via NFS works. The datastore is there, I can create small files, move them around, delete them, etc. But anything of signficance, like creating disk file for VM, dies. vCenter client (VIclient) as well as CLI methods for creating a VM all fail. Sometimes the VMX file is created sometimes not, but always fail when attempting to create disk file. We consistently get the following error in the vmkernel logs. The following example is from ESXi 3.5 U4 classic for ESXi same results same error. The error below is misleading, the NFS mount is in front of an EXT3 550GB drive that is empty. I am not a NFS guru but someone that is, I think might see something we are missing? The issue is consistent, on Fedora 11 with latest updates.

Jul 21 08:49:34 Host01 vmkernel: 0:01:17:04.020 cpu2:1036)NFS: 107: Command: (mount) Server: (192.168.15.73) IP: (192.168.15.73) Path: (/datastore/nfs/store0) Label: (Server01:Store0) Options: (None)

Jul 21 08:49:49 Host01 vmkernel: 0:01:17:18.819 cpu2:1035)BC: 2672: Failed to flush 1 buffers of size 131072 each for object b00f 40 1 1070001 18c0002 0 28b49f1e 854b745c 34640eb3 b302966b aa573132018c0006 0 0 0: Limit exceeded

Jul 21 08:49:49 Host01 vmkernel: 0:01:17:18.921 cpu2:1035)WARNING: NFSLock: 583: .lck-06008c0100000000 0xa601e38 updateFailureCount 1: bytesWritten 0

Jul 21 08:49:49 Host01 vmkernel: 0:01:17:18.921 cpu2:1035)WARNING: NFSLock: 1346: Unable to create lock file: .lck-06008c0100000000, no space left on device

Jul 21 08:49:49 Host01 vmkernel: 0:01:17:18.921 cpu2:1035)NFSLock: 1480: Failed to create lock info for lock file .lck-06008c0100000000 : No space left on device

[DSTAVERT]

Is the drive mounted?? Does the exports file point to the drive mount point?

[Schorschi]

Yes, I can create files on the mount point, delete them, etc. Any other Linux box can access the mount point as exported. Here is the weird part. I built a Fedora 11 box, but did not update it, straight of the initial release media only. I only used yum to install nfs (rpcbind, nfslock, nfs-utils). I am still testing, but I was able to create a VM via VIclient logged into vCenter, where on a fully patched Fedora 11 box this failed during VM creation. As noted above, the datastore connects, on either ESXi 3.5 U4 or ESXi 4, but both are impacted by fully patched instance of Fedora 11. I thought it was because NFSv4 is not support by VMware, but I am not 100% sure how to tell if NFSv4 is active, I believe it is, since process list shows a worker thread as "nfsd4", since I did use yum, I figured I have the latest version of NFS for Fedora. I am created a number of partitions to see if there is some quirky issue with the partition geometry of the target below given exported mount point. So far, 1.5 TB and 1.0+ TB as either ext3 and ext4 partitions under NFS failed let ESX acquire a lock as noted above.

[AntonVZhbankov]

Looks like permissions problem.

Make chmod 777 on directory you sharing. Also check that you enabled RW access to NFS share from this host.

>root@warehouse ~ # cat /etc/exports

>/spare *(rw)

>/spare 10.xx.xx.xx/32(rw)

You can use both ways, allow to all or allow just for particular host.

---

VMware vExpert '2009

http://blog.vadmin.ru

[SuperGrobi73]

Hallo Schorchi,

any ESX Server cannot access any NFSv4 Volumes, this is depending on ESX only supporting NFSv3!

You can run your NFSv4 Server in NFSv3 Mode, then ESX can again access the volumes.

Good Luck

Carsten

--

Mein Blog: http://www.datenfront.de

[DSTAVERT]

ESXi does NOT support NFS4. If three is disabled by default you will need to enable it.

[Schorschi]

The mount is valid, it works. It exists. It is exported correctly. RedHat or different Fedora box can use the mount. It is only ESXi (3.5 U4 or 4) that can not actually use the mount. ESXi connects to the mount, can browse the datastore once connected, however the VM creation wizard attempts to create the VM, but fails to complete the VM creation sequence when the lock is not acquried. This problem only happens on Fedora 11 fully patched. Fedora 11 baseline works. So there is something, which I have yet to find, that is being patched in Fedora that breaks ESXi ability to acquire a NFS file lock. So it is not an export file permissions issue, but that was a good idea to check, it not a question of the mount not existing it does, it shows as a valid datastore. I suspect I will have to do a control update of Fedora 11 per package to find the package or packages that create the issue. It is a recent update that is doing this. This was not an issue a couple of weeks ago.

[Schorschi]

Yeah, fear that is the real issue, and that it works with the baseline as a fluke or luck, but some update breaks the fact that it just happens to work. Going to be fun to figure out how to force NFS to use v3 only.

[AntonVZhbankov]

So, a couple of weeks ago you have successfully created VM on this NFS share?

---

VMware vExpert '2009

http://blog.vadmin.ru

[Schorschi]

Yes, and, when I create a baseline Fedora 11 box, as well. It is only when I do a full update of all pending updates to Fedora 11, for example # yum update, which applies all updates that exist for Fedora 11, the problem with VM creation via a lack of NFS lock acquire, appears. I have a box right now, that has only the baseline code installed with only NFS installed, and it works, but if I full update the Fedora 11 box, the issue occurs.

[DSTAVERT]

Just check to see if NFS3 is enabled. You don't need to disable NFS4 just as long as it will accept 3 as well.

[Schorschi]

Maybe I am missing something? I disabled NFSv4 I believe but still see a thread for nfsd4? See below...

# vi /etc/sysconfig/nfs

<changed settings, saved file>

# cat /etc/sysconfig/nfs

MOUNTD_NFS_V3="yes" RPCNFSDARGS="-N 2 -N 4"

# service nfs --full-restart

Shutting down NFS mountd:

Shutting down NFS daemon:

Shutting down NFS services:

Starting NFS services:

Starting NFS daemon:

Starting NFS mountd:

# ps -el | grep -i nfs

1 S 0 1486 2 0 75 -5 - 0 worker ? 00:00:00 nfsd4 1 S 0 1487 2 0 75 -5 - 0 svc_re ? 00:00:02 nfsd

1 S 0 1488 2 0 75 -5 - 0 svc_re ? 00:00:02 nfsd

1 S 0 1489 2 0 75 -5 - 0 svc_re ? 00:00:02 nfsd

1 S 0 1490 2 0 75 -5 - 0 svc_re ? 00:00:02 nfsd

1 R 0 1491 2 0 75 -5 - 0 - ? 00:00:02 nfsd

1 S 0 1492 2 0 75 -5 - 0 svc_re ? 00:00:03 nfsd

1 R 0 1493 2 0 75 -5 - 0 - ? 00:00:02 nfsd

1 S 0 1494 2 0 75 -5 - 0 svc_re ? 00:00:02 nfsd

[DSTAVERT]

Just for fun. Try changing selinux to permissive.

[Schorschi]

Ha! Slightly ahead of you on this one... our standard open default configuration... for the lab sandbox... is setenforce 0, grub.conf 'selinux=0' and selinux config set to 'disabled' which trumps permissive right? Was the first thing I double checked!

.

[DSTAVERT]

Try -o noacl option in the mount command.

[Schorschi]

vCenter GUI does not support optional parameters for datastore mounts? Try the mount via the ESXi embedded CLI?

[Schorschi]

Still working to narrow this down, built a Fedora 11 box again using baseline media only, and works, the minute I update via 'yum update' then kaboom! Disable of NFSv4 appears to be inconsistent, turned it off per the documentation, but the NFS daemon still shows a NFSv4 thread in the process list.