5 Replies Latest reply on Feb 7, 2019 2:30 PM by wila

    NFS datastore slow from ESXi, fast from VMs inside ESXi

    IvanFdz Lurker

      So I have this (rather upsetting) situation in which I have an ESXi 6.0 accessing an NFS datastore at 7 Mbit/s while a vm *inside* the ESXi accesses the same NFS shared folder at a (more tolerable) speed of 150Mbit/s.

       

      Is there anything (the more rare or unexpected the better, I have already tried all the average and easy to find "solutions") I can do to fix this? I've been using this method to make my backups with ghettoVCB since I can't remember, but it all started failing some weeks ago...

       

      Thanks everyone reading!

        • 1. Re: NFS datastore slow from ESXi, fast from VMs inside ESXi
          wila Guru
          vExpertCommunity WarriorsUser Moderators

          Hi,

           

          Curious.. are you using FreeNAS and did you update that a few weeks ago perhaps?

           

          --

          Wil

          | Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
          | Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
          | More info at https://www.vimalin.com
          | Twitter @wilva
          | VMware Wiki at http://www.vi-toolkit.com
          • 2. Re: NFS datastore slow from ESXi, fast from VMs inside ESXi
            IvanFdz Lurker

            Hi Vil,

             

            not using FreeNAS. I've been using an OpenDedupe Appliance on a different ESXi host for the backups, and so far it was working OK (not top speed, but OK for ghettoVCB). I thought it got somehow corrupted and deleted it completelty, only to find the same problem after reinstalling a new (updated version) appliance. Then deleted that and installed a plain ubuntu server vm and exported a folder. Same results. Finally I'm testing against a dockerized nfs server at my own place (ESXi is at OVH) and there is where I'm getting steady 7Mbit/s from ESXi vs. 150Mbit/s from a vm inside that ESXi. Beautiful nightmare....

             

            -Iván.

            • 3. Re: NFS datastore slow from ESXi, fast from VMs inside ESXi
              wila Guru
              Community WarriorsUser ModeratorsvExpert

              Hi,

               

              OK, I was just wondering as about 2 weeks ago I saw my vmkfstools based backup scripts take double the time without any indicators of why the extra time.

              No hardware failures that I can find and switching all VMs over to another host gives the same -OK, even slower- result.

               

              Only thing in common between the 2 hosts is the FreeNAS server and I did run an update to the latest version around that time.

              Upgraded the ZFS pool too, so cannot go back to an older version to exclude that being the issue (oops).

               

              As for your problem. Difficult to say. Running performance tests from within the service console is awkward as the service console is constrained on things like copying files around.

              For good reasons as it is a management console, not designed for doing a lot of work, it has limited memory and very limited CPU.

              You should run tests with the commands designed for moving data in the console, such as vmkfstools, not just via cp / mv etc..

               

              --

              Wil

              | Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
              | Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
              | More info at https://www.vimalin.com
              | Twitter @wilva
              | VMware Wiki at http://www.vi-toolkit.com
              • 4. Re: NFS datastore slow from ESXi, fast from VMs inside ESXi
                IvanFdz Lurker

                Well, just to clarify, this is the command I use at the ESXi console for cloning a vmdk:

                screenshot651.png

                ...and this is what my NFS Server sees:

                screenshot653.png

                While this is the command at the console of the linux vm inside that ESXi:

                screenshot654.png

                and what the same NFS server sees:

                screenshot655.png

                 

                Shouldn't both graphs be at least on the same scale? why the ESXi only gets 1/13th of the speed?

                Unless I'm doing something stupid here (which is probably the correct explanation because it's indeed the simplest and Occam Razor's-ish, I'll keep on thinking it's aliens...)

                 

                -Iván.

                • 5. Re: NFS datastore slow from ESXi, fast from VMs inside ESXi
                  wila Guru
                  User ModeratorsCommunity WarriorsvExpert

                  Hi,

                   

                  It is pretty much expected behavior.

                   

                  There are a couple of things you can do to improve throughput.

                   

                  Disable atime, so that you don't update meta data on reads. Usually you don't need that feature and it does make a difference.

                   

                  Disable sync.

                  vSphere wants to aggressively sync when writing data. While that is a good thing for if you run a VM of that disk, it might be less interesting if you are using it for backup purposes.

                   

                  The following article is for FreeNAS, but depending on your storage implementation it might be relevant there too.

                   

                  https://forums.freenas.org/index.php?threads/sync-writes-or-why-is-my-esxi-nfs-so-slow-and-why-is-iscsi-faster.12506/

                  IOW, don't disable sync writes if you intend to run VMs from that datastore.

                   

                  If you're not using ZFS then look into the NFS async mount option. The same warning as above counts of course.

                   

                  --

                  Wil

                  | Author of Vimalin. The virtual machine Backup app for VMware Desktop Products
                  | Vimalin : Automated backups for VMware Fusion and VMware Workstation Professional
                  | More info at https://www.vimalin.com
                  | Twitter @wilva
                  | VMware Wiki at http://www.vi-toolkit.com