4 Replies Latest reply on Dec 12, 2011 10:48 AM by daveytech

    My NFS Experiences: Windows NFS vs. CentOS 5.5

    daveytech Lurker

      I wanted to post some info regarding my recent NFS experiences while I had a moment to spare.

       

      When I started using ghettoVCB I initially created an NFS share on an eSATA disk attached to a Windows 2003 R2 32 bit box connected to the network at 1 Gbps. I did not have any issues with this configuration at first, and all seemed well. I was averaging around 100 Mbps when writing to the NFS share while creating my backups. While this speed isn't anything to write home about, it was sufficient.

       

      I hit my first snag when I couldn't backup a particular VM. It would just bomb out at 48%. I ran through every single troubleshooting measure I could think of without success. I scoured the internet looking for others who had the same issue to see if they had found a solution, and I simply could not find any reliable "fix".

       

      I then tried some 3rd party NFS for Windows apps, haneWin, and Allegro NFS. I couldn't get either of them to work at a level that was satisfactory.

       

      So, I decided that I had spent enough time trying to get NFS working on the Windows platform and installed CentOS 5.5 32 bit on the same box.

       

      I installed CentOS, fdisk'd and mounted my eSATA disk, and then created an NFS export on the mount folder. I had never done this before, so I was a bit hesitant as I'm not a Linux expert, but I must say that I am extremely happy that I did.

       

      That same VM that was giving me grief during the backup to the Windows NFS share had zero issues while backing up to the linux NFS share. Zero.

       

      Not only have I not had a single issue while backing up my VM's, I have seen a 600% increase in backup speed. Where I used to average 100 Mbps, I am now averaging 600 Mbps. In one case I had a VM that took 9.5 minutes to backup on Windows NFS which now backs up in 1.2 minutes on the CentOS box. That's more than 600% faster, and I'm not exactly sure where the extra time savings is coming from, but I'll gladly accept it.

       

      I can post more information about my config/environment if anyone is interested. The point of this story is to help those of you who are battling with Windows NFS. Drop it and start thinking about implementing NFS on Linux.

        • 1. Re: My NFS Experiences: Windows NFS vs. CentOS 5.5
          lamw Guru
          VMware EmployeesCommunity Warriors

          Thanks for sharing this, I'm sure this will be very helpful for others!

          • 2. Re: My NFS Experiences: Windows NFS vs. CentOS 5.5
            tohe Enthusiast

            Yes, this is a very good report about your NFS experience.

             

            Please can you give us some more details about your config, e.g which filesystem you are using and parameters to mount this and your NFS configuration.

             

            Thank you!

            • 3. Re: My NFS Experiences: Windows NFS vs. CentOS 5.5
              daveytech Lurker

              I want to state that since I am still in the design stages of implementing this backup system, I still have work to do to make this a full production backup system. Like converting my VM backup storage device(s) to a RAID array for resiliency. I also disabled the Linux firewall during setup. I didn't want any software not related to NFS to potentially interfere. In a nutshell, I can't guarantee that anything I've listed here is either secure or will work in all environments.

               

              For completeness I have included most of my server configuration in this post.

               


              Backup Server Config

               

              This is a really old server that I wanted to make use of. It's a perfect example of how you can turn an old server you have sitting in a corner in to a great backup tool.

               

              - Server Model: Dell PowerEdge 2600 Tower.

               

              - CPU: 2x Intel Zeon 2.4 Ghz (Pentium 4 equivalent w/Hyperthreading)

               

              - RAM: 2 GB

               

              - OS: CentOS 5.5 32-bit. Installed using netinstall wth minimal packages selected. I did install KDE and some GUI tools though.

               

              - Local disks: 3x 33.6 GB Seagates. (This is where CentOS is installed.)

               

              - eSATA RAID Controller: SYBA-PCXSA2-2E2R. This controller uses the Sil3124 chipset. CentOS 5.5 has built in drivers for this card.

               

              - eSATA Disk: 2 TB Cavalry CAXH3702T3.

               

              (Both the controller and disk were purchased from NewEgg business for a few hundred dollars. Dirt cheap storage solution.)

               

               

               

              Disk / Backup Storage Config

              I learned how to do all of this in 15 minutes by searching google. These commands will likely not work for your configuration as your device names, folder names, and even Linux commands may be different. I added this purely for the sake of example.

               

              - Erased and created partitions on the eSATA disk using the CLI utility fdisk.

               

              - Ran the following script to format the disk with the EXT3 file system.

               

              mkfs.ext3 -b 4096 /dev/sdb1

               

              - Added disk to fstab and mounted the disk to the /ESATA folder (which I created earlier):

               

              - Added the following line to /etc/fstab

               

              /dev/sdb1 /ESATA ext3 defaults 1 1

               

              - Command to mount the new disk:

               

              mount /ESATA

               

               

               

              NFS Config

               

              I created the NFS share by editing the /etc/exports file. This file essentially contains the entire NFS configuration in a single line. Two services are required for NFS to work properly on the Linux platform. One is the actual NFS service and the other is the portmap service. This is all well documented on various web sites. I encourage you to do some searching and read up on it.

               

              /etc/exports config

               

              - The IP part of the line allows you to specify which IP's or subnets should be allowed to access ths share. The "no_root_squash" option is required for ESXi to gain full access to the share. I know that this is well known, but it is also stated in the following VMWare document, which is worth reading for beginners. http://vmware.com/files/pdf/VMware_NFS_BestPractices_WP_EN.pdf

               

              - I added the following line to /etc/exports:
              /ESATA/ 10.x.x.x/24(rw,async,no_root_squash)

               

              - I manually started the portmap service using the CLI/terminal
              /etc/init.d/portmap start

               

              - I manually started the NFS service using the CLI:
              /etc/init.d/nfs start

               

              - If you have to make changes to the exports file for any reason, you will have to restart the NFS service with the following command. You will also need to re-mount the NFS share on your ESXi server if you already have it mounted. I had to do this several times as I was getting things fine-tuned.

               

              /etc/init.d/nfs restart

               

              - I had to configure the portmap and NFS services to start during boot up. I know there is a way to do this in a terminal, but I did it through the 'Services' GUI tool by checking the box next to those two services.

               

              - A command I found useful for verifying your NFS share was actually up and running on your NFS box. If it's not displayed in the output then you either have an /etc/exports configuration problem, or the NFS or portmap services are not running.


              showmount -e

               

               

               

              ghettoVCB Config

               

              Again, I have listed this information for example purposes. You will need to decide how to organize your scripts and related files.

               

              - ESXi seems to erase any foreign files created on the filesystem after a reboot, so I have stored all of my ghetto-related scripts on the NFS share.

               

              - You'll want to copy the ghetto .zip to the NFS server, using WinSCP or some other tool, and unzip it using the CLI. This will preserve the file's attributes.

               

              - I created a simple directory structure within my NFS share for my scripts, logs, and backups.

               

              Backups

              /ESATA/VMBACKUPS

               

              Scripts

              /ESATA/VMBACKUP-SCRIPTS

               

              Logs

              /ESATA/VMBACKUP-LOGS

               

               

              - I will be using plink to execute the backup commands on the ESXi box remotely. I will use Windows task scheduler on my current production backup server, running BackupExec, to execute the ghettoVCB backups via plink.

               

              - If you mount the NFS share on the ESXI box, using vSphere client, you can run all of your scripts, vm backup lists, and store your logs on the NFS share. I highly recommend it. This is all well documented in the ghettoVCB documentation.

               

               

              - Example .bat file run on my Windows box using plink to execute the ghettoVCB.sh script remotely.

               

              c:\VMBACKUP-SCRIPTS\plink.exe root@x.x.x.x -pw youresxirootpasswordhere "/vmfs/volumes/YourNFS/VMBACKUP-SCRIPTS/ghettoVCB.sh -f /vmfs/volumes/YourNFS/VMBACKUP-SCRIPTS/VMLIST.DAILY.TXT -g /vmfs/volumes/YourNFS/VMBACKUP-SCRIPTS/ghettoVCB.DAILY.conf > /vmfs/volumes/YourNFS/VMBACKUP-LOGS/BACKUP.DAILY.ghettoVCB.ESXI-ServerName.log"

               

               

              - Example .bat file run on my Windows box using plink to execute the ghettoVCB-restore.sh script remotely.

               

              c:\VMBACKUP-SCRIPTS\plink.exe root@x.x.x.x -pw youresxirootpasswordhere "/vmfs/volumes/YourNFS/VMBACKUP-SCRIPTS/ghettoVCB-restore/ghettoVCB-restore.sh -c /vmfs/volumes/YourNFS/VMBACKUP-SCRIPTS/ghettoVCB-restore/VM-RESTORE-LIST.TXT"

               

               

               

              Notes:

               

              After yesterday's post I am seeing an error in the ghetto backup logs for two of my VM's. The error is related to the script being unable to delete the previous backup's vmdk file. I have my rotation count set to 1 currently. However, this error seems to be irrelevant as the previous backup does actually get deleted. I'm a bit paranoid so I went ahead and verified I could restore the backed up VM using ghettoVCB-restore.sh and it worked perfectly. All is well.

               

              I encounted another issue when trying to restore VM's using ghettoVCB-restore.sh. For some reason it didn't like the directory names where the backups were stored. Such as /vmfs/volumes/YourNFS/VM-NAME/VM-NAME-DATETIMESTAMP. I had to rename the /VM-NAME-DATETIMESTAMP directory to be shorter and unique to the local datastore on the ESXi server to get the restore script to run. The path ended up looking something like /vmfs/volumes/YourNFS/VM-NAME/VM-NAME2. After the rename the script ran extremely well. I was restoring a "thin" disk as a "zeroedthick" disk and achieved 950 Mbps average speed during the restore process. Since it was cloning the disk as a zeroedthick, the restore operation did take quite a bit longer than the backup process, but at 950 Mbps, it was as efficient and fast as one could possibly hope for with a gigabit connection.

               

              I posted that I was seeing a decrease in backup time beyond just the boost in network performance going from Windows NFS to Linux NFS. It appears that the "thin" disk type is actually working correctly when backing up to the Linux NFS share. I was led to this conclusion by comparing different sized VMDK's backup times and noticed that the actual amount of data stored in the VMDK determined the amount of time it takes to back up the file.

               

              For example, I have two zeroedthick VMDK's. One is 40 GB, the other is 100 GB. Both VMDK's have roughly 6 GB's of actual data. Both VMDK's took roughly the same amount of time to back up.

               

              However, the VMDK file size is still reported to be the full size on the backup file system. Though when I check the properties of the folder in Linux, it shows that actual disk space usage does not reflect the reported file size. My knowledge of Linux file systems is weak at best, so this may be something others already understand.

               

              This clearly demonstrates another drawback to Windows NFS. Backups to Windows NFS appeared to be backing up the empty parts of VMDK's as if data existed, even though the backup format was set to "thin". I'm starting to wonder if NFS just doesn't mix well with NTFS.

               

               

              Update 3/8/11

              I ran a backup on a VM with two VMDK's. One VMDK is 20 GB, with 10 GB of actual data. The other VMDK is 500 GB, with 1 GB of actual data. This backup took 16.5 minutes. I was not expecting this backup to take this long, based on the information I gathered previously and posted above. I'm guessing that the cloning operation scans the entire VMDK for data because after about 5 minutes into the backup, the network activity dropped off completely, yet the cloning process had not reached 20%. Sustained network activity did not return at any point in the backup once it had stopped and the backup completed successfully.

               

              Message was edited by: daveytech

              • 4. Re: My NFS Experiences: Windows NFS vs. CentOS 5.5
                daveytech Lurker

                Well, it's been some time since I implemented this system and I am happy to say that it's been running non-stop without any issues since March of this year.

                 

                I ended up purchasing 2x eSATA enclosures and 2x higher-end eSATA Seagate disks for the enclosures. I then set up the disks in a Linux softraid 1 configuration on my CentOS box. I'm currently backing up VM's from 7 different ESXi servers to these eSATA disks (which host my NFS share).

                 

                I have been approached by several companies who offer products that perform a similar function to what the ghettoVCB scripts and my NFS share currently does. I have to say that I haven't found a compelling reason to shift away from this set up in favor of a more quote "refined/professional product". I work for a public school district, and money is extremely tight, so once again, I want to extend my gratitude to lam for his ghettoVCB scripts. Many thousands of people are benefiting from your efforts in my environment alone.

                 

                If folks are interested, I can post the process I used to set up my eSATA disks as a softraid and/or re-write my big post based on the softraid configuration instead of the single disk configuration.

                 

                Would anyone be interested in a more comprehensive or easy-to-follow guide on how I configured everything from start to finish? I can't help but think that there are others out there like myself who would benefit from a guide that is geared towards those with little Linux experience.