12 Replies Latest reply on Jan 16, 2019 4:18 PM by texcor

    NFS 4.1 on ESXi 6.5: unable to browse datastore

    giox Novice

      I mounted a NFS 4.1 share on ESXi 6.5a (single host, no vcenter).

       

      This is the CLI command, but I can also mount it via web interface:

       esxcli storage nfs41 add -a AUTH_SYS -H 192.168.98.4 -v MyLinuxNFS -s /mnt/ds-esxi
      

      My linux /etc/exports is

           /mnt/ds-esxi    192.168.98.0/27(rw,sync,no_subtree_check,anonuid=1002,anongid=1002)

       

      After correctly mounting it, I can see it on the web client:

       

      But I'm unable to browse MyLinuxFS, the "Datastore browser" stays in the following state forever:

      Permission are correctly set, because I can write files and see the datastore via the CLI:

      [root@vm2:~] echo "123test" > /vmfs/volumes/MyLinuxNFS/testfile.txt
      [root@vm2:~] ls -la /vmfs/volumes/MyLinuxNFS/
      total 24
      drwxr-xr-x    4 root     root             5 Feb  5 19:32 .
      drwxr-xr-x    1 root     root           512 Feb  5 19:32 ..
      drwxr-xr-x    2 root     root            10 Feb  4 17:10 testdir1
      drwxr-xr-x    2 root     root             4 Feb  4 22:26 testdir2
      -rw-r--r--    1 root     root             8 Feb  5 19:32 testfile.txt
      
      

      NFSv3 is working fine, but I would like to use NFSv4.1 because I'm experimenting to workaround bug KB2010953 in a bigger infrastructure.

      I also tried with FreeBSD 11, it seems to support NFSv4.1, but I have the same problem described here: FreeBSD11 NFSv4.1 is not compatible with ESXi 6.5.

       

      Any idea to make NFS 4.1 work on ESXi 6.5 ?

        • 1. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
          dalong4net Lurker

          The datastore is read only in this case. I encountered exactly the same issue when using NFS 4.1 share from FreeBSD 11. I haven' t figured out any solution yet.

           

          If someone can shine some light on this, it will be very helpful. Hopefully VMware can address this soon, since I'm evaluting the vsphere for production now.

          • 2. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
            JelleHissink Lurker

            I am having the same issue, have spent a few days now.

            My setup uses nfs4 from an ubuntu 16.04.1 server.

             

            The thing is on ssh on the esxi 6.5a server I can read and write files to the nfs server, using workstation 12.5 I am able to browse the datastore.

            Using PowerCli reading from the datastore also works.


            So the issue seems to be located in the web interface.

             

            Hope this gets fixed soon, it is almost unworkable like this.

            • 3. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
              JelleHissink Lurker

              Hmmm, digging some further, not all seems well.

              I created a new vm on the datastore:

              -rw------- 1 nobody nogroup 2147483648 Feb 10 23:39 test_0-flat.vmdk

              -rw------- 1 nobody nogroup        492 Feb 10 23:39 test_0.vmdk

              -rw-r--r-- 1 nobody nogroup          0 Jan  2  1980 test.vmsd

              -rwxr-xr-- 1 nobody nogroup       2009 Feb 10 23:40 test.vmx

              -rw-r--r-- 1 nobody nogroup      52502 Feb 10 23:40 vmware.log

               

              so the files are on the ubuntu server.

              Starting the vm however fails, the vmware.log of the vm:

              2017-02-10T22:34:48.827Z| vmx| I125: DISK: OPEN scsi0:0 '/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk' persistent R[]

              2017-02-10T22:34:48.831Z| vmx| I125: DISKLIB-VMFS  : VmfsExtentCommonOpen: possible extent truncation (?) realSize is 0, size in descriptor 4194304.

              2017-02-10T22:34:48.831Z| vmx| I125: DISKLIB-VMFS  : "/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0-flat.vmdk" : failed to open (The file specified is not a virtual disk): Size of extent in descriptor file larger than real size. Type 3

              2017-02-10T22:34:48.831Z| vmx| I125: DISKLIB-LINK  : "/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk" : failed to open (The file specified is not a virtual disk).

              2017-02-10T22:34:48.831Z| vmx| I125: DISKLIB-CHAIN : "/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk" : failed to open (The file specified is not a virtual disk).

              2017-02-10T22:34:48.831Z| vmx| I125: DISKLIB-LIB   : Failed to open '/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk' with flags 0xa The file specified is not a virtual disk (15).

              2017-02-10T22:34:48.831Z| vmx| I125: DISK: Cannot open disk '/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk': The file specified is not a virtual disk (15).

              2017-02-10T22:34:48.831Z| vmx| I125: DISK: Opening disks took 5 ms.

              2017-02-10T22:34:48.831Z| vmx| I125: Module 'Disk' power on failed.

              2017-02-10T22:34:48.831Z| vmx| I125: VMX_PowerOn: ModuleTable_PowerOn = 0

              2017-02-10T22:34:48.831Z| vmx| I125: SVMotion_PowerOff: Not running Storage vMotion. Nothing to do

              2017-02-10T22:34:48.833Z| vmx| I125: Vix: [90009 mainDispatch.c:1175]: VMAutomationPowerOff: Powering off.

              2017-02-10T22:34:48.834Z| vmx| W115: /vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test.vmx: Cannot remove symlink /var/run/vmware/0/80444906393_90009/configFile: No such file or directory

              2017-02-10T22:34:48.834Z| vmx| I125: WORKER: asyncOps=1 maxActiveOps=1 maxPending=0 maxCompleted=0

              2017-02-10T22:34:48.839Z| vmx| I125: Vix: [90009 mainDispatch.c:4234]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0

              2017-02-10T22:34:48.839Z| vmx| I125:

              2017-02-10T22:34:48.839Z| vmx| I125+ Power on failure messages: File system specific implementation of LookupAndOpen[file] failed

              2017-02-10T22:34:48.839Z| vmx| I125+ File system specific implementation of LookupAndOpen[file] failed

              2017-02-10T22:34:48.839Z| vmx| I125+ File system specific implementation of LookupAndOpen[file] failed

              2017-02-10T22:34:48.839Z| vmx| I125+ The file specified is not a virtual disk

              2017-02-10T22:34:48.839Z| vmx| I125+ Cannot open the disk '/vmfs/volumes/18630e34-de98dd50-0000-000000000000/test/test_0.vmdk' or one of the snapshot disks it depends on.

              2017-02-10T22:34:48.839Z| vmx| I125+ Module 'Disk' power on failed.

              2017-02-10T22:34:48.839Z| vmx| I125+ Failed to start the virtual machine.

              • 4. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                takenalias Lurker

                I am having the exact same problem. I can not browse the datastore ect. Would be good to hear from VMware on this issue.

                • 5. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                  rmanning Lurker

                  I'm simply trying to migrate VMs from an older 5.5 hypervisor to a new 6.5 box and half of them get the same error you are seeing on startup (File system specific implementation of LookupAndOpen[file] failed).  No solution in sight so far, this is ridiculous...

                  • 6. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                    am68 Lurker

                    I encounter exactly the same issue with a NFS 4.1 volume on ESXi 6.5a when trying to browse the NFS v4.1 datastore using the ESXi Web Client (not vCenter).

                     

                    Everything else works fine (creating and running VMs on the NFS v4.1 datastore, copying files, etc.) - only the datastore browsing in the Web Client seems to be a problem.

                     

                    However, on a ESXi 6.0 U2 host, the datastore browsing works fine (when using the same NFS v4.1 shared volume).

                    It also works on both ESXi systems (6.5a and 6.0 U2) when ESXi connects to the NFS volume via NFS v3.

                     

                    In my case, the NFS server is a CentOS 7.3 system and the NFS connection is using AUTH_SYS (as described here: VMware vSphere 6.5 Documentation Library).

                    Typically the NAS servers use the no_root_squash, as described here: VMware vSphere 6.5 Documentation Library.

                     

                    The NFS export is defined as follows:

                    /nfsstorage     *(rw,sync,insecure,no_root_squash,no_subtree_check,no_wdelay).

                     

                    Just wanted to note that I cannot reproduce JelleHissink's problem with a CentOS 7 based NFS server.

                    • 7. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                      ESantelices Lurker

                      I encountered this issue as well. I'm using an Intel NUC with a Synology in my home lab and ESXi 6.5. I encountered the problem when trying to Register VM's. Switching back to NFS v3 and the issue goes away.

                      • 8. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                        jojo Novice

                        I have the same problem - esxi host webclient can't browse nfs 4.1 data storages. Version 3 of nfs data storage work fine. If I use the virtual center I can browse the v4.1 nfs storages too.

                         

                        ESXi Host WebClient

                        Version 1.8.0

                        Build 4516221

                         

                        ESXi

                        Version 6.5.0

                        Build 4887370

                         

                        Regards

                         

                            Jochen

                        • 9. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                          yanivhs Lurker

                          I'm having the same problem as well

                          Using NFSv4.1 share from an EMC Unity 300 storage

                           

                          Using SSH I can browse it and create files.

                          Also i was able to deploy VMs from OVF or by copying directly to the datastore using scp.

                           

                          Any new ideas?

                          • 10. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                            enoch85 Novice

                            I'm also having this exact same issue on (Updated) ESXi-6.5.0-20170304101-standard (VMware, Inc.) + FreeNAS 11.0-U2.


                            I can read files created on the FreeNAS NFS share, but I can't delete, create or modify from the WebUI. Haven't tried to run a VM from that storage yet, but should be impossible as I can't write to the share.

                             

                            Any help would be appreciated.

                            • 11. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                              solgaeDK Enthusiast

                              I have a similar issue, but a little more insight.

                               

                              It seems there's an issue when using datastore browser in vSphere Web Client (or HTML5 client). The datastore browser works fine when you attempt to go into file browser (e.g. root directory of datastore), but if you attempt to browse to the VM files that are powered on, it hangs like the thread author described. However, the VMs themselves power on and operate just fine.

                               

                              It seems when using datastore browser from vSphere Web Client (HTML5 Client), vCenter will use the last host that has access to the datastore (https://kb.vmware.com/s/article/1034573) as proxy for browsing datastores. When you browse to a VM folder that is powered on from that last host, the VM file browsing works fine. If the VM is powered on to any other host, the issue occurs. I'm guessing read requests are getting rejected when using datastore browser from the web client due to file lock.

                               

                              This isn't exactly deal-breaking issue since the VM operations work fine otherwise, and you can still browse the files via shell access to ESXi host, but it's an annoying issue nonetheless. None of the issue happens on NFS 3.

                              • 12. Re: NFS 4.1 on ESXi 6.5: unable to browse datastore
                                texcor Lurker

                                For those who still have this problem,

                                This is a webgui bug fixed in 1.10.0v (see changelog).

                                Upgrade procedure:

                                esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxui/esxui-signed-latest.vib