8 Replies Latest reply on Feb 9, 2015 10:01 PM by michaeldaly

    Failed to clone disk: Input/output error (327689).

    valsimots Novice

      I am able to successfully backup multiple VM's using GhettoVCB.

      However, this one in particular has been troublesome.

       

      I can confirm the following:

      1. Ample disk space on both host datastore and NAS

      2. Other VM's backup successfully.

      3. No Snapshots exist.

       

      Any suggestions as to what I can do to resolve or if anyone has any resources to point me to would be very much appreciated. Thanks in advance. 

       

      Below are the logs for this particular VM.

       

      2014-08-11 16:40:54 -- info: ============================== ghettoVCB LOG START ==============================

       

      2014-08-11 16:40:54 -- debug: Succesfully acquired lock directory - /tmp/ghettoVCB-work.weekly

       

      2014-08-11 16:40:54 -- debug: HOST VERSION: VMware ESXi 5.0.0 build-469512

      2014-08-11 16:40:54 -- debug: HOST LEVEL: VMware ESXi 5.0.0 GA

      2014-08-11 16:40:54 -- debug: HOSTNAME: uranus.hds

       

      2014-08-11 16:40:54 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = /vmfs/volumes/datastore1/vmconfig-DataSet1-weekly

      2014-08-11 16:40:54 -- info: CONFIG - VERSION = 2013_01_11_0

      2014-08-11 16:40:54 -- info: CONFIG - GHETTOVCB_PID = 28700400

      2014-08-11 16:40:54 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/Vault2-DataSet1/URANUS/Weekly

      2014-08-11 16:40:54 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 2

      2014-08-11 16:40:54 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2014-08-11_16-40-54

      2014-08-11 16:40:54 -- info: CONFIG - DISK_BACKUP_FORMAT = thin

      2014-08-11 16:40:54 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0

      2014-08-11 16:40:54 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0

      2014-08-11 16:40:54 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 4

      2014-08-11 16:40:54 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5

      2014-08-11 16:40:54 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15

      2014-08-11 16:40:54 -- info: CONFIG - LOG_LEVEL = debug

      2014-08-11 16:40:54 -- info: CONFIG - BACKUP_LOG_OUTPUT = /vmfs/volumes/Vault2-DataSet1/URANUS/Logs/ghettoVCB.Weekly_Backup.log

      2014-08-11 16:40:54 -- info: CONFIG - ENABLE_COMPRESSION = 0

      2014-08-11 16:40:54 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0

      2014-08-11 16:40:54 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0

      2014-08-11 16:40:54 -- info: CONFIG - ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0

      2014-08-11 16:40:54 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all

      2014-08-11 16:40:54 -- info: CONFIG - VM_SHUTDOWN_ORDER =

      2014-08-11 16:40:54 -- info: CONFIG - VM_STARTUP_ORDER =

      2014-08-11 16:40:54 -- info: CONFIG - EMAIL_LOG = 0

      2014-08-11 16:40:54 -- info:

      2014-08-11 16:40:56 -- debug: Storage Information before backup:

      2014-08-11 16:40:56 -- debug: SRC_DATASTORE: datastore3-hdd

      2014-08-11 16:40:56 -- debug: SRC_DATASTORE_CAPACITY: 931.2 GB

      2014-08-11 16:40:56 -- debug: SRC_DATASTORE_FREE: 324.9 GB

      2014-08-11 16:40:56 -- debug: SRC_DATASTORE_BLOCKSIZE: 1

      2014-08-11 16:40:56 -- debug: SRC_DATASTORE_MAX_FILE_SIZE: 256 GB

      2014-08-11 16:40:56 -- debug:

      2014-08-11 16:40:56 -- debug: DST_DATASTORE: Vault2-DataSet1

      2014-08-11 16:40:56 -- debug: DST_DATASTORE_CAPACITY: 2662.4 GB

      2014-08-11 16:40:56 -- debug: DST_DATASTORE_FREE: 864.6 GB

      2014-08-11 16:40:56 -- debug: DST_DATASTORE_BLOCKSIZE: NA

      2014-08-11 16:40:56 -- debug: DST_DATASTORE_MAX_FILE_SIZE: NA

      2014-08-11 16:40:56 -- debug:

      2014-08-11 16:40:57 -- debug: getVMDKs() - DXT.ifarmer.vmdk###37:

      2014-08-11 16:40:57 -- info: Initiate backup for DXT.ifarmer

      2014-08-11 16:40:57 -- info: Creating Snapshot "ghettoVCB-snapshot-2014-08-11" for DXT.ifarmer

      2014-08-11 16:40:59 -- debug: Waiting for snapshot "ghettoVCB-snapshot-2014-08-11" to be created

      2014-08-11 16:40:59 -- debug: Snapshot timeout set to: 900 seconds

      2014-08-11 16:40:59 -- debug: findVMDK() - Searching for VMDK: "DXT.ifarmer.vmdk" to backup

      2014-08-11 16:40:59 -- debug: /sbin/vmkfstools -i "/vmfs/volumes/datastore3-hdd/DXT.ifarmer/DXT.ifarmer.vmdk" -a "ide" -d "thin" "/vmfs/volumes/Vault2-DataSet1/URANUS/Weekly/DXT.ifarmer/DXT.ifarmer-2014-08-11_16-40-54/DXT.ifarmer.vmdk"

      Destination disk format: VMFS thin-provisioned

      Cloning disk '/vmfs/volumes/datastore3-hdd/DXT.ifarmer/DXT.ifarmer.vmdk'...

       

      Clone: 10% done.

      Clone: 11% done.

      Clone: 12% done.

      Clone: 13% done.

      Clone: 14% done.

      Clone: 15% done.

      Clone: 16% done.

      Clone: 17% done.

      Clone: 18% done.

      Clone: 19% done.

      Clone: 20% done.

      Clone: 21% done.

      Clone: 22% done.

      Clone: 23% done.

      Clone: 24% done.

      Clone: 25% done.

      Clone: 26% done.

      Clone: 27% done.

      Clone: 28% done.

      Clone: 29% done.

      Clone: 30% done.

      Clone: 31% done.

      Clone: 32% done.

      Clone: 33% done.

      Clone: 34% done.

      Clone: 35% done.

      Clone: 36% done.

      Clone: 37% done.

      Clone: 38% done.

      Clone: 39% done.

      Clone: 40% done.

      Clone: 41% done.

      Clone: 42% done.

      Clone: 43% done.

      Clone: 44% done.

      Clone: 45% done.

      Clone: 46% done.

      Clone: 47% done.

      Clone: 48% done.

      Clone: 49% done.

      Clone: 50% done.

      Clone: 51% done.

      Clone: 52% done.

      Clone: 53% done.

      Clone: 54% done.

      Clone: 55% done.

      Clone: 56% done.

      Clone: 57% done.

      Clone: 58% done.

      Clone: 59% done.

      Clone: 60% done.Failed to clone disk: Input/output error (327689).

       

      2014-08-11 17:12:43 -- info: ERROR: error in backing up of "/vmfs/volumes/datastore3-hdd/DXT.ifarmer/DXT.ifarmer.vmdk" for DXT.ifarmer

      2014-08-11 17:13:04 -- info: Removing snapshot from DXT.ifarmer ...

      2014-08-11 17:13:04 -- debug: Removing /vmfs/volumes/Vault2-DataSet1/URANUS/Weekly/DXT.ifarmer/DXT.ifarmer-2014-08-08_01-00-01

      2014-08-11 17:13:04 -- info: Backup Duration: 32.12 Minutes

      2014-08-11 17:13:04 -- info: ERROR: Unable to backup DXT.ifarmer due to error in VMDK backup!

       

      2014-08-11 17:13:04 -- info: ###### Final status: ERROR: No VMs backed up! ######

       

      2014-08-11 17:13:04 -- debug: Succesfully removed lock directory - /tmp/ghettoVCB-work.weekly

       

      2014-08-11 17:13:04 -- info: ============================== ghettoVCB LOG END ================================

        • 1. Re: Failed to clone disk: Input/output error (327689).
          JoergD Novice

          Could it possible that the maximum file size of the NAS is not sufficient?

          Which file system is used on the NAS?

          • 2. Re: Failed to clone disk: Input/output error (327689).
            valsimots Novice

            We are using FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty) using ZFS with 3 volumes, 1.8 TB each (6 TB drive capacity combined, 3 x 2TB WD Red drives).

             

            As mentioned before, other VM"s backing up using the Ghetto script (approx. 27 other VM's) - many others are much larger.

            This VM is about 40 GB, other VM's are as large as 530 GB that backup successfully.

             

            This VM did backup successfully some time ago - however, it is unknown as to when it stopped successfully backing up.

            The VM itself is a Windows XP VM that works without any issues as well.

            • 3. Re: Failed to clone disk: Input/output error (327689).
              JoergD Novice

              Ok, the datastore file system of the FreeNAS-8.3.1 normally is UFS or ZFS.  The maximum file size should not be the problem.

               

              Could you migrate the VM to another datastore on the ESX server?

               

              Should the migration failed, a file from the VM is not "clean".

               

              Could you the VM shutdown? Is the backup after the shutdown possible?

              • 4. Re: Failed to clone disk: Input/output error (327689).
                valsimots Novice

                Looks like I have a corrupt vmdk file... a copy of the DXT.ifarmer from DataStore3 to DataStore4 failed.

                A backup of the VM with the VM Shutdown also failed. Same issue at ~60%

                • 5. Re: Failed to clone disk: Input/output error (327689).
                  JoergD Novice

                  That's bad news.

                   

                  You've written, that the VM works without any issues.

                  In the past I have solved such a problem as follows:

                   

                  1. Check the SCSI controler setting from the failed hard disk (SCSI 0:0 or ...)

                  2. Shutdown the VM and add a new hard disk with the same size

                  3. Boot the VM from a CD with a cloning tool for example Acronis

                  4. Clone the old disk to the new disk

                  5. Shutdown the VM and delete the old hard disk from the VM (!!! NOT FROM THE DATASTORE !!!)

                  6. Reconfigure the SCSI controler settings from the new hard disk to the same value of the old hard disk (see 1.)

                  7. Boot the VM. It should work and a backup job also should work

                  8. Is everything ok, the old hard disk VMDK can delete from the datastore

                  • 6. Re: Failed to clone disk: Input/output error (327689).
                    valsimots Novice

                    I'm wondering if installing VMWare Converter on the VM can simply convert it again to another new VM.

                    • 7. Re: Failed to clone disk: Input/output error (327689).
                      JoergD Novice

                      Since the VM works, the problem lies outside the VM.

                      The VMDK is readable but not movable.

                       

                      Maybe a reboot of the ESX might help or you using the generated VM by VMware Converter.

                      Then the question arises whether the old VM can be deleted.

                       

                      I have not, unfortunately, more ideas.

                      • 8. Re: Failed to clone disk: Input/output error (327689).
                        michaeldaly Novice

                        My system has caught this disease too. I have a nice looking 200GB backup on an NAS datastore (QNAP NAS with ext4 file system), created by ghettoVCB from an ESXI4.1 system, but I can't get it to restore. The ghettoVCB.sh has a LAST_MODIFIED_DATE=2011_11_19 so I presume thats the date of the script. One thing I've noticed was a setting for DISK_BACKUP_FORMAT=thin but I was attempting to restore using the DISK_FORMATS of '1' ie 'zeroedthick'. I then instead tried the '2' option ie '2gbsparse' which did result in  a 100% completion but the restored folder was a collection of multiple 2Gig files that failed to start. So I might try a restore using the 'DISK_FORMAT' setting of '3' ie thin, and see what happens. Hopefully it will not hang at eg 18% restore, again

                        MD

                        Post Script...using the DISK_FORMATS option  of '3', which matched my DISK_BACKUP_FORMAT=thin option in the backup script, resulted in an operable restore in the early hours this morning!