ghettoVCBg2 - Free alternative for backing up VMs in ESX(i) 3.5 and 4.x (no SSH console required!)

ghettoVCBg2 - Free alternative for backing up VMs in ESX(i) 3.5 and 4.x (no SSH console required!)

Table of Contents:

    • Description
    • Features
    • Requirements
    • ESX/ESXi Version Support Table
    • Setup
    • Addings ESX/ESXi host(s) to VIMA/vMA
    • Configurations
    • Usage
    • Sample Execution 
      • Backup VMs located in a list
      • Backup VMs using individual backup policies
      • Dryrun Mode
    • Experimental Features
    • FAQ
    • Useful Links
    • Change Log

Description:

This tool is the follow up release of the ghettoVCB

backup utility which allows users to perform backups of virtual machines residing on ESX(i) 3.5+/4.x+

servers using methodology similar to VMware's VCB

tool. By incorporating highly constructive feedback from the VMware  community and utilizing the existing VI API, ghettoVCB’s framework was  completely rewritten to be harder, better, faster, stronger

.

The primary motivation for ghettoVCBg2 was to provide ESXi users with  access to the utility without relying on unlocking and utilizing the  unsupported console. To satisfy this requirement, the rebuilt framework  takes advantage of the

VI Perl Toolkit/vSphere SDK for Perl which is present in the

VIMA/vMA virtual appliance provided by VMware. As a result, ghettoVCBg2 provides  a more proper backup solution that administrators can utilize in their  virtual infrastructure.

As it stands, it is a requirement that ghettoVCB be invoked directly on  each of the ESX servers hosting virtual machines in need of a backup. By  taking advantage of VIMA/vMA, the entire backup process becomes  centralized in turn eliminating the abovementioned requirement.  Additionally, by leveraging the vi-fastpass library, unattended  authentication between VIMA/vMA and target ESX servers is made possible.  This alleviates the need to continually pass in credentials for each  ESX server.

By centralizing the backup process, individual virtual machine backup  lists pertaining to their respective ESX(i) servers are no longer  required. ghettoVCBg2 has the ability to identify virtual machines and  their respective hosts, potentially eliminating any past confusion  associated with maintaining multiple lists.

In its current configuration, the script will allow up to 3 unique  backups of the Virtual Machine before it will overwrite the previous  backups; this however, can be modified to fit procedures if need be.  Please be diligent in running the script in a test or staging  environment before using it on production live Virtual Machines; this  script functions well within our environment but there is a chance that  it may not fit well into other environments.

Features

  • Support for logging (normal & verbose)
  • Single VM backup list across multiple ESX/ESXi host(s)
  • Credential-less backups (so long as host(s) are being managed by VIMA/vMA)
  • Online back up of VM(s)
  • Only valid VMDKs presented to the VM will be backed up
  • Support for multiple VMDK backup per VM
  • Preserve original powerState of VM(s)
  • Ability to shutdown guestOS, initiate backup process and power on VM afterwards with the option of hardpower timeout
  • Ensure that snapshot removal process completes prior to continuing the backup process
  • VM(s) that initially contain snapshots and Physical RDMs (raw device mapppings) will not be backed up
  • Support for VM(s) with Virtual RDMs
  • Ability to specify the number of backup rotations per VM
  • Output back up VMDKs in ZEROEDTHICK (default behavior), EAGERZEROEDTHICK, 2GB SPARSE or THIN format
  • Ouput backup VMDKs using either BUSLOGIC or LSILOGIC adapter type
  • Fully support VMDKs stored across multiple datastores
  • VM snapshot memory and quiesce options
  • Individual VM backup policy (supported on ESX(i) 3.5u2+ & ESX(i) 4.x+)
  • Email logs (Experimental Support) NEW!
  • Ability to include/exclude specific VMDK(s) per VM (requires individual VM backup policy setup) NEW!
  • Independent disk aware (will ignore VMDK) NEW!
  • Additional debugging information including dry run execution NEW!


Requirements:

VMs running on ESX(i) 3.5u2+ or ESX(i) 4.x+

VMware VIMA 1.0 or VMware vMA 4.x


ESX/ESXi Version Support Table

VersionSupportAlternative
ESX 3.5u2Requires at least Foundation/Essentials licenseghettoVCB\
ESX 3.5u3Requires at least Foundation/Essentials licenseghettoVCB\
ESX 3.5u4Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 3.5u2Works for freeghettoVCB\
ESXi 3.5u3Works for freeghettoVCB\
ESXi 3.5u4Requires at least Foundation/Essentials licenseghettoVCB\
ESX 4.0Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 4.0Requires at least Foundation/Essentials licenseghettoVCB\
ESX 4.1Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 4.1Requires at least Foundation/Essentials licenseghettoVCB\



Setup:

1) Download ghettoVCBg2.tar.gz

to your VIMA 1.0/vMA 4.x system

2) Extract the contents of the tarball

[vi-admin@scofield ~]$ tar -zxvf ghettoVCBg2.tar.gz
ghettoVCBg2/
ghettoVCBg2/ghettoVCBg2-vm_backup_configuration_template
ghettoVCBg2/ghettoVCBg2.pl

3) The script should be in a directory called

ghettoVCBg2


Addings ESX/ESXi host(s) to VIMA/vMA

Prior to starting, ensure that the VMware VIMA/vMA host is managing the appropriate ESX(i) 3.5u2/4.x+ host(s)

1. Add ESXi hosts to VIMA/vMA managment using vifp

A password prompt for the vi-admin account and root password to the pertinent ESXi host(s) will be presented.

[vi-admin@vima-primp-industries ~]$ sudo vifp addserver olga.resnet.ucsb.edu
root@olga.resnet.ucsb.edu's password:
[vi-admin@vima-primp-industries ~]$

2. Verify that the appropriate hosts are being managed by VIMA/vMA

[vi-admin@vima-primp-industries ~]$ sudo vifp listservers
olga.resnet.ucsb.edu
himalaya.primp-industries.com


Configurations

The following variables (similar to that of ghettoVCB) need to be defined within the script prior to execution.

Defining the datastore in which the backups are stored:

my $VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups";

Defining the name of folder that will contain backups (if folder does not exist, it will automatically be created):

my $VM_BACKUP_DIRECTORY = "WILLIAM_BACKUPS";

Defining the backup rotation per VM:

my $VM_BACKUP_ROTATION_COUNT = "3";

Defining the backup disk format (zeroedthick, eagerzeroedthick, thin, and 2gbsparse are available):

my $DISK_BACKUP_FORMAT = "zeroedthick";

Defining the adapter type for backed up VMDK (buslogic, and lsilogic are available):

my $ADAPTER_FORMAT = "buslogic";

Defining whether the VM is powered down or not prior to backup (1 = enable, 0 = disable):

Note: VM(s) that are powered off will not require snapshoting

my $POWER_VM_DOWN_BEFORE_BACKUP = "0";

Defining whether virtual machine memory is snapped and if quiescing is enabled (1 = enable, 0 = disable):

Note: By default both are disabled

my $VM_SNAPSHOT_MEMORY = "0";
my $VM_SNAPSHOT_QUIESCE = "0";

Defining the detail of logging (verbose or normal):

my $LOG_LEVEL = "normal";

*Defining VMDK(s) to backup from a particular VM either a list of vmdks or "all"

VM_VMDK_FILES="myvmdk.vmdk"

Experimental Support Configs:

Note: Port 25 by default

my $SEND_MAIL = "no";
my $EMAIL_HOST = "emailserver";
my $EMAIL_DOMAIN = "localhost.localdomain";
my $EMAIL_TO = 'William Lam <william@primp-industries.com.com>';
my $EMAIL_FROM = 'ghettoVCBg2 <ghettoVCBg2@primp-industries.com.com>';

Ensure that you do not edit past this section:

########################## DO NOT MODIFY PAST THIS LINE ##########################


Usage:

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl
Required command option 'vmlist' not specified.

Synopsis: ./ghettoVCBg2.pl OPTIONS


Command-specific options:
   --config_dir
      Name of directory containing VM(s) backup configurations
   --dryrun (default '0')
      Set to 1 to enable dryrun mode (default 0)
   --output (default '/tmp/ghettoVCBg2.log')
      Full path to output log (default /tmp/ghettoVCBg2.log)
   --vmlist (required)
      A file containing a list of virtual machine(s) to be backed up on host

Common VI options:
   --config (variable VI_CONFIG)
      Location of the VI Perl configuration file
   --credstore (variable VI_CREDSTORE)
      Name of the credential store file defaults to <HOME>/.vmware/credstore/vicredentials.xml on Linux and <APPDATA>/VMware/credstore/vicredentials.xml on Windows
   --encoding (variable VI_ENCODING, default 'utf8')
      Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
   --help
      Display usage information for the script
   --passthroughauth (variable VI_PASSTHROUGHAUTH)
      Attempt to use pass-through authentication
   --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
      Pass-through authentication negotiation package
   --password (variable VI_PASSWORD)
      Password
   --portnumber (variable VI_PORTNUMBER)
      Port used to connect to server
   --protocol (variable VI_PROTOCOL, default 'https')
      Protocol used to connect to server
   --savesessionfile (variable VI_SAVESESSIONFILE)
      File to save session ID/cookie to utilize
   --server (variable VI_SERVER, default 'localhost')
      VI server to connect to. Required if url is not present
   --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
      Service path used to connect to server
   --sessionfile (variable VI_SESSIONFILE)
      File containing session ID/cookie to utilize
   --url (variable VI_URL)
      VI SDK URL to connect to. Required if server is not present
   --username (variable VI_USERNAME)
      Username
   --verbose (variable VI_VERBOSE)
      Display additional debugging information
   --version
      Display version information for the script

Note: There are some additional command line arguments that can be provided upon runtime: --debug, --output and --vmlist. --vmlist is the only parameter required. The rest will have defaults if not explicitly defined.

--dryrun

  • Dry run information will be written to the log and no backup will take place

--output

  • Name of the output log, if the same log is used, it will  automatically append (ensure you provide full path e.g.  /tmp/ghettoVCBg2.log)

--config_dir

  • This directory will contain the individual VM backup policy files.  If a virtual machine has an associated policy file, the global  configuration will be ignored for that backup instance.

The input to this script is a file that contains the display name of the  virtual machine(s) separated by a newline. When creating this file on a  non-Linux/UNIX system, you may introduce a ^M character which may cause  the script to misbehave. To ensure that this does not occur, please  create the file on the ESX/ESXi host.

Provided is a sample of what the file should contain:

[vi-admin@scofield ~]$ cat virtual_machine_backup_list
VIMA
vMA-2


Sample Execution

  • Backup VMs located in a list
  • Backup VMs based on individual VM backup policies
  • Debug Mode
  • Dry run Mode

Backup VMs located in a list

  • Log verbosity: info (default)
  • Log output: /tmp/ghettoVCBg2.log (default)

DISK_FORMAT2gbsparse
ADAPTER_FORMATbuslogic
LOG_LEVELinfo
VM_SNAPSHOT_MEMORY0
VM_SNAPSHOT_QUIESCE0
VMS_TO_BACKUP(VIMA,vMA-2)


[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --vmlist virtual_machine_backup_list

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-13-2009 23:34:53 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-13-2009 23:34:53 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-13-2009 23:34:53 --  info: CONFIG - DISK_BACKUP_FORMAT = 2gbsparse
        11-13-2009 23:34:53 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-13-2009 23:34:53 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-13
        11-13-2009 23:34:53 --  info: CONFIG - VM_VMDK_FILES = all

        11-13-2009 23:34:58 --  info: Initiate backup for VIMA found on himalaya.primp-industries.com
        11-13-2009 23:35:18 --  info: VIMA has 1 VMDK(s)
        11-13-2009 23:37:37 --  info: Backup completed for VIMA!

        11-13-2009 23:37:37 --  info: Initiate backup for vMA-2 found on himalaya.primp-industries.com
        11-13-2009 23:37:47 --  info: vMA-2 has 1 VMDK(s)
        11-13-2009 23:40:07 --  info: Backup completed for vMA-2!

        11-13-2009 23:40:10 --  info: ============================== ghettoVCBg2 LOG END ==============================

Backup VMs based on individual VM backup policies and log output to /tmp/ghettoVCB.log

  • Log verbosity: info (default)
  • Log output: /tmp/ghettoVCB.log (default)

VIMA

VM_BACKUP_DATASTOREdlgCore-NFS-bigboi.VM-Backups
VM_BACKUP_DIRECTORYWILLIAM_BACKUPS
DISK_FORMATzeroedthick
ADAPTER_FORMATbuslogic
LOG_LEVELdebug
VM_SNAPSHOT_MEMORY1
VM_SNAPSHOT_QUIESCE1
VM_VMDK_FILESVIMA_1.vmdk


vMA-2

VM_BACKUP_DATASTOREdlgCore-NFS-bigboi.VM-Backups
VM_BACKUP_DIRECTORYvGHETTO_BACKUPS
DISK_FORMATthin
ADAPTER_FORMATlsilogic
LOG_LEVELinfo
VM_SNAPSHOT_MEMORY0
VM_SNAPSHOT_QUIESCE0
VM_VMDK_FILESall


1. Create folder to hold individual VM backup policies (can be named anything):

[vi-admin@scofield ~]$ mkdir backup_config

2. Create individual VM backup policies for each VM and ensure each file  is named exactly as the display name of the VM being backed up (use  provided template to create duplicates):

[vi-admin@scofield backup_config]$ cp ghettoVCBg2-vm_backup_configuration_template VIMA
[vi-admin@scofield backup_config]$ cp ghettoVCBg2-vm_backup_configuration_template vMA-2

4. Using a preferred editor (nano or vi), edit each VM backup policy file

Listing of VM backup policy within backup configuration directory

[vi-admin@scofield backup_config]$ ls
ghettoVCBg2-vm_backup_configuration_template  VIMA  vMA-2

5. Pass in the value, backup_config, into the --config_dir parameter

(e.g.)
--config_dir backupConfigs

Example of two distinct VM backup policies will be applied to VIMA and vMA-2

Backup policy for "VIMA" (backup only 1 specific VMDKs)

[vi-admin@scofield backup_config]$ cat VIMA
VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups"
VM_BACKUP_DIRECTORY = "WILLIAM_BACKUPS"
VM_BACKUP_ROTATION_COUNT = "3"
DISK_BACKUP_FORMAT = "zeroedthick"
ADAPTER_FORMAT = "buslogic"
POWER_VM_DOWN_BEFORE_BACKUP = "0"
VM_SNAPSHOT_MEMORY = "1"
VM_SNAPSHOT_QUIESCE = "1"
LOG_LEVEL = "debug"
VM_VMDK_FILES = "VIMA_1.vmdk"

Backup policy for VM "vMA-2" (backup all VMDKs found)

VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups"
VM_BACKUP_DIRECTORY = "vGHETTO_BACKUPS"
VM_BACKUP_ROTATION_COUNT = "3"
DISK_BACKUP_FORMAT = "thin"
ADAPTER_FORMAT = "lsilogic"
POWER_VM_DOWN_BEFORE_BACKUP = "0"
VM_SNAPSHOT_MEMORY = "0"
VM_SNAPSHOT_QUIESCE = "0"
LOG_LEVEL = "info"
VM_VMDK_FILES = "all"

The following is an output of the script when utilizing the new --config_dir parameter with debug mode enabled.

Note: --vmlist is still a required  parameter with the exception that backup policy files must exist for  each VM in the list else the VM will be ignored in the backup process.

Execution of backup

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --config_dir backup_config --vmlist virtual_machine_backup_list

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-14-2009 00:18:41 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-14-2009 00:18:45 --  info: CONFIG - USING CONFIGURATION FILE = VIMA
        11-14-2009 00:18:45 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-14-2009 00:18:45 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
        11-14-2009 00:18:45 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-14-2009 00:18:45 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:18:45 --  info: CONFIG - VM_SNAPSHOT_MEMORY = YES
        11-14-2009 00:18:45 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = YES
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:18:45 --  info: CONFIG - VM_VMDK_FILES = VIMA_1.vmdk

        11-14-2009 00:18:45 --  info: Initiate backup for VIMA found on himalaya.primp-industries.com
        11-14-2009 00:18:55 --  debug: VIMA original powerState: poweredOn
        11-14-2009 00:18:55 --  debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-11-14" for VIMA
        11-14-2009 00:19:45 --  info: VIMA has 2 VMDK(s)
        11-14-2009 00:19:45 --  debug: findVMDKFile: Found VMDK File: VIMA_1.vmdk
        11-14-2009 00:19:45 --  debug: backupVMDK: Backing up "[http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk" to "[http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14/VIMA_1.vmdk"
        11-14-2009 00:19:45 --  debug: backupVMDK: Signal copyThread to start
        11-14-2009 00:19:45 --  debug: backupVMDK: Backup progress: Elapsed time 0 min
        11-14-2009 00:19:45 --  debug: copyTask: Wake up and follow the white rabbit, with status: doCopy
        11-14-2009 00:19:45 --  debug: CopyThread: Start backing up VMDK(s) ...
        11-14-2009 00:19:52 --  debug: copyTask: send copySuccess message ...
        11-14-2009 00:19:52 --  debug: copyTask: waiting for next job and sleep ...
        11-14-2009 00:20:46 --  debug: backupVMDK: Successfully completed backup for [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk Elapsed time: 1 min
        11-14-2009 00:20:49 --  debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-11-14" for VIMA
        11-14-2009 00:20:51 --  debug: checkVMBackupRotation: Starting ...
        11-14-2009 00:20:57 --  debug: Purging [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-13--3 due to rotation max
        11-14-2009 00:21:05 --  info: Backup completed for VIMA!

        11-14-2009 00:21:05 --  debug: reConfigureBackupParams: VM - vMA-2
        11-14-2009 00:21:05 --  info: CONFIG - USING CONFIGURATION FILE = vMA-2
        11-14-2009 00:21:05 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DIRECTORY = vGHETTO_BACKUPS
        11-14-2009 00:21:05 --  info: CONFIG - DISK_BACKUP_FORMAT = thin
        11-14-2009 00:21:05 --  info: CONFIG - ADAPTER_FORMAT = lsilogic
        11-14-2009 00:21:05 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:21:05 --  info: CONFIG - VM_VMDK_FILES = all

        11-14-2009 00:21:05 --  info: Initiate backup for vMA-2 found on himalaya.primp-industries.com
        11-14-2009 00:21:15 --  info: vMA-2 has 1 VMDK(s)
        11-14-2009 00:23:22 --  info: Backup completed for vMA-2!

        11-14-2009 00:23:25 --  info: ============================== ghettoVCBg2 LOG END ==============================

Dry run Mode (no backup will take place)

  • Log verbosity: drymode
  • Log output: /tmp/ghettoVCB.log (default)

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --config_dir backup_config --vmlist virtual_machine_backup_list --dryrun 1

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-14-2009 00:26:29 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-14-2009 00:26:34 --  info: CONFIG - USING CONFIGURATION FILE = VIMA
        11-14-2009 00:26:34 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-14-2009 00:26:34 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
        11-14-2009 00:26:34 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-14-2009 00:26:34 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_MEMORY = YES
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = YES
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:26:34 --  info: CONFIG - VM_VMDK_FILES = VIMA_1.vmdk

        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO VIMA ----------
        11-14-2009 00:26:34 --  info: DEBUG - Host Build: VMware ESX 4.0.0 build-164009
        11-14-2009 00:26:34 --  info: DEBUG - Host: himalaya.primp-industries.com
        11-14-2009 00:26:34 --  info: DEBUG - Virtual Machine: VIMA
        11-14-2009 00:26:34 --  info: DEBUG - VM ConfigPath: [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - VMX File: VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupConfigPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14/VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14
        11-14-2009 00:26:34 --  info: DEBUG - VM Datastore: himalaya-local-SAS.Savvio
        11-14-2009 00:26:34 --  info: DEBUG - VMDK(s):
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA.vmdk
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk
        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO VIMA ----------

        11-14-2009 00:26:34 --  debug: reConfigureBackupParams: VM - vMA-2
        11-14-2009 00:26:34 --  info: CONFIG - USING CONFIGURATION FILE = vMA-2
        11-14-2009 00:26:34 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIRECTORY = vGHETTO_BACKUPS
        11-14-2009 00:26:34 --  info: CONFIG - DISK_BACKUP_FORMAT = thin
        11-14-2009 00:26:34 --  info: CONFIG - ADAPTER_FORMAT = lsilogic
        11-14-2009 00:26:34 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:26:34 --  info: CONFIG - VM_VMDK_FILES = all

        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO vMA-2 ----------
        11-14-2009 00:26:34 --  info: DEBUG - Host Build: VMware ESX 4.0.0 build-164009
        11-14-2009 00:26:34 --  info: DEBUG - Host: himalaya.primp-industries.com
        11-14-2009 00:26:34 --  info: DEBUG - Virtual Machine: vMA-2
        11-14-2009 00:26:34 --  info: DEBUG - VM ConfigPath: [http://himalaya-local-SAS.Constellation|http://himalaya-local-SAS.Constellation] vMA-2/vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - VMX File: vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupConfigPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] vGHETTO_BACKUPS/vMA-2/vMA-2-2009-11-14/vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] vGHETTO_BACKUPS/vMA-2/vMA-2-2009-11-14
        11-14-2009 00:26:34 --  info: DEBUG - VM Datastore: himalaya-local-SAS.Constellation
        11-14-2009 00:26:34 --  info: DEBUG - VMDK(s):
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Constellation|http://himalaya-local-SAS.Constellation] vMA-2/vMA-2.vmdk
        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO vMA-2 ----------

        11-14-2009 00:26:37 --  info: ============================== ghettoVCBg2 LOG END ==============================


FAQ

0Q: I'm having some issues or would like to provide feedback

0A: Please join the ghettoVCBg2 Group to post your comments/questions

1Q: I'm using ESXi and free licensed version ( I did not pay for anyting ), will this script still work?

1A: No, you need to have a licensed version of ESXi to use this script.

2Q: I have a VM listed in my backup file but no backup of that VM  took place. The log stated that the VM could not be found. How could  this be? I know that it’s running on host X…

2A: Ensure the ESX/ESXi host is being managed by VMware VIMA/vMA (sudo vifp listservers to verify)

3Q: Can I schedule backups to take place hourly, daily, monthly, yearly?

3A: Yes, do a search online for crontab.

4Q: I want to schedule my backup on Windows, how do I do this?

4A: Do a search for

plink. Make sure you have paired SSH keys setup between your Windows system and VIMA/vMA.

5Q: Do I have to use VMware VIMA/vMA to initiate the backup?

5A: Yes, vi-fastpass is being used and only VIMA/vMA provides  this capability. However, you can kickoff the process from another host  (*nix/win)

4Q: I only have a single ESXi host. I want to take backups and  store them somewhere else. The problem is: I don't have NFS, iSCSI nor  FC SAN. What can I do?

4A: You can use local storage to store your backups assuming that  you have enough space on the destination datastore.  Afterwards, you  can use scp (WinSCP/FastSCP) or vifs (vCLI) to transfer the backups from the ESXi host to your local desktop.

5Q: I’m pissed; the backup is taking too long. My datastore is of type X?

5A: YMMV, take a look at your storage configuration and make sure it is optimized.

6Q: I noticed that the backup rotation is occurring after a  backup. I don't have enough local storage space, can the process be  changed?

6A: This is primarily done to ensure that you have at least one  good backup in case the new backup fails. If you would like to modify  the script, you're more than welcome to do so.

7Q: What is the best storage configuration for datastore type X?

7A: Search the VMTN forums; there are various configurations for the different type of storage/etc.

8Q: I want to setup NFS Server to run my backups. Which is the best and should it be virtual or physical?7

8A: Please refer to answer 7A. From experience, we’ve seen  physical instances of NFS servers to be faster than their virtual  counterparts.  As always, YMMV.

9Q: When I try to use the script I get an error stating: "restricted version". What the heck is this?

9A: Please take a look at the ESX/ESXi Version Support Table in this document. The reason is explained here.

10Q: I'm getting error X when using the script or I'm not getting  any errors, the backup didn’t even take place. Oh what could be wrong?

10A: First, you'll want to check the log (default  /tmp/ghettoVCBg2) and see if there are any obvious errors. If not,  please provide the entire output of the log, a short description of your  environment and the type of VM(s) you're trying to backup—I will try to  provide some assistance.

11Q: I have VMs that have snapshots. I want to back these things up but the script doesn’t let me do it. How do I fix that?

11A: VM snapshots are not meant to be kept for long durations.  When backing up a VM that contains a snapshot, you should ensure all  snapshots have been committed prior to running a backup. No exceptions  will be made…ever.

12Q: I would like to restore from backup, what is the best method?

12A: The restore process will be unique for each environment and  should be determined by your backup/recovery plans. At a high level you  have the option of mounting the backup datastore and registering the VM  in question or copy the VM from the backup datastore to the ESX/ESXi  host. The latter is recommended so that you're not running a VM living  on the backup datastore or inadvertently modifying your backup VM(s).

13Q: When using the experimental feature: Individual VM backup policy, if I don't have a policy will it use the default configurations defined in the script?

13A: No, this use case will not be supported, ever. If you decide  to use the per VM policy, then the global configuration in the script  is ignored. If a VM policy config is not found, the VM will not be  backed up even if it's in the –-vmlist.

14Q: In vMA 4.0, vCenter target will be supported, can I just add vCenter target and assume this will work?

14A: At the moment, this is not a supported feature. You will  need to add each ESX/ESXi host for the vi-fastpass library to be able to  locate the host(s).

15Q: When I try to run the script I get: "-bash: ./ghettoVCBg2.pl: Permission denied", what is wrong?

15A: You need to change the permission on the script to be executable, chmod +x ghettoVCBg2.pl

16Q: Where can I download the latest version of the script?

16A: The latest version is available on this page at the bottom. To get the version of the script run

./ghettoVCBg2.pl --version

17Q: I'm running into an issue when I backup my VMs using NFS on  Linux, I'm able to either get 1 successful backup and all subsequent  backups fail OR I'm having issues where snapshots are not being removed  upon completition and future backups fail.

17A: This problem/issue has been reported by quite a few users in  the community and only affects a very small percentage of users overall  under a specific configuration. The problem as I understand it today  which is still an on-going investigation by myself and the community is  as follows:

The Setup

Setup: NFS on Linux where the underlying volume is EXT3 filesystem

and backups are failing primarily around larger sized VMs ( > 150GB).

The Observed Problem

The issue is during the final phase of a VM being backed-up in the  rotation function where a request is sent out to the NFS server to  either A) rotate/rename a directory B) delete an old backup. There is  either a disconnect or delay that severely affects the backup process  that may lead to a whole slew of "unexpected" behavior from having only 1  backup complete successful and all subsquent backups failing/etc.

First off, there is nothing wrong with the script from what we've found  it and has to do with NFS server and specifically the volume that the  backup datastore is hosted on. Thanks to community user

ralfoertner who had a simliar problem using

ghettoVCBg2 using Debian with EXT3 base filesystem for his NFS volume. He decided  to do some additional troubleshooting to further isolate the problem and  actually found by changing the underlying filesystem from EXT3 to XFS  that the issue went away and resolve all issues he had ran into  regarding an NFS on Linux backup solution.

The Theory

Now, we're still 100% sure why the above solution has resolved the  problem or what the actual problem is but I'll explain the current  theory:

EXT3 is a journaled filesystem, you can do a search online for more  information. There's a few advantages with EXT3 from it's predcessor  EXT2, but one difference is in the approach it takes with journaling as  explained here in greater detail:

http://www.ibm.com/developerworks/library/l-fs7.html

Basically EXT2 does only "metadata journling" whereas EXT3 does both  data and metadata journaling, this would ensure that all changes to the  filesystem where it's data or metadata is consistent in the event of an  unexpected system crash. With EXT3, there are 3 journaling options: (

data=writeback mode,data=ordered mode,data=journal mode ) and by default it uses

data=ordered mode to ensure the consistency as described from above.

One easy method to check what mount options are configured, on a Linux system you can check /proc/mounts:

[vi-admin@scofield ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/VolGroup00/var /var/log ext3 rw,data=ordered 0 0
/dev/sda1 /boot ext3 rw,data=ordered 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0

So now what does this have to do with XFS and EXT3? With the defaults of  EXT3, it may not always give you the best performance due to the extra  overhead of the way it does its journaling, if you were to change the  options to

data=writeback mode, you'll notice that this is simliar journaling method found in

XFS,JFS and ResierFS as described here:

http://www.ibm.com/developerworks/library/l-fs8.html

The Conclusion

I believe that there are some additional overheads that maybe caused by  using NFS on EXT3, which may explain the issue that users are seeing and  that moving from EXT3 to XFS has "seemed" to solve the issues for those  who have implemented the change. Though, before you go out and make  this change, you need to understand your environment and what this  change would exactly mean, please refer to differences in EXT2/XFS and  EXT3, especially around a power interruption or system crash.

In conclusion, I'm not recommending one way or the other, it really  depends on your environment. Though this has shown to resolve the issue  as users have implemented to get around this problem, but it's  definitely worth while to really understand the problem in your  environment and figure out why and decide on the best solution. I would  also recommend users perform their own tests to further isolate the  problem in their environment and try local storage, using iSCSI to see  if they can pin-point the problem.


Useful Links:

If you have found this script or other scripts to be useful and would like to contribute back, please click

here to donate! Thank You


Change Log

07/20/10 - build 4.3

Enhancements:

  • Support for vSphere 4.1 (ESX and ESXi)

Bug Fixes:

  • Due to changes in the VIFP Library introduced with vMA 4.1, the  script has been updated to work on both vMA 4.0 and vMA 4.1. Only  upgrade to vMA 4.1 if you're managing ESX(i) 4.1 hosts


03/27/10 - build 4.2

Enhancements:

  • Added a check to see if ESX(i) host is in a connected state
  • Added a check to see if ESXi host is using the free license, if so, skip host
  • Added FAQ #0

03/27/10 - build 4.1

Enhancements:

  • Updated FAQ #17 regarding "NFS" issue


02/24/09 - build 4.1

Bug Fixes:

  • There was a typo in handling the case of a VM's VMDK(s) being  distributed across multiple datastores in which the backup was not  creating the directory to reflect this layout, thanks to briandatamatters for finding this bug.

05/26/09 - build 2.5.1

Enhancements:

  • N/A

Bug Fixes:

  • Implemented shorterm solution fix for bug in VMware vCLI's vifs
  • Updated logic to allow virtual RDM to be backed up


04/25/09 - build 2.5

Enhancements:

  • Added experimental feature Individual VM backup policy
  • Added VM snapshot memory & quiesce options

Bug Fixes:

  • N/A


02/21/09 - build 2.0

Initial release

Attachments
Comments

Nice Script and Nice work !

I had trouble with previous version.

Now it's works fine ! I can backup all my vm and i receive a report status in my mailbox everytime !

Thanks a lot William.

Yea that was changed in this release with just having two logging options. Glad to see you got it working

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Awesome to hear!

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

hi all ,

I just installed this fine tool to evaluate is as main backup tool

(after being disapointed of other commercial tools)

i was wondering if it would be possible to add a feature to execute a script add the end of each VM backup

something like

POST_BACKUP_SCRIPT=/home/vi-admin/myscript.sh

POST_BACKUP_SCRIPT_ONESX=true

this would copy the script to the ESX/ESXi server and execute it after each backup.

i would provide 2 parameters :

- path of backup location

- name of VM

we could use this to compress the files in a rsync friendly way

so we can transfer the "big" files to a offsite storage for save keeping

Like the idea of post script. Another idea might be to add another option --vm vmname so i could create a backup script like:

./ghettoVCBg2.pl --vm vm_to_backup

my_post_script vm_to_backup

./ghettoVCBg2.pl --vm next_vm_to_backup

my_post_script next_vm_to_backup

....

so one could then choose if he wants to use --vmlist or just --vm

This kind of defeats the purpose of the script not requiring any SSH connections to either the classic ESX or unsupported busybox console on ESXi, else a user could just use [ghettoVCB.sh| which does support compression.

If you're looking for such functionality, I would recommend you checking out VMware's vDR or Veeams Backup/Recovery (which I've heard great things about, especially supporting change block tracking). There are no compression support within the VMware APIs ... minus change block tracking which I'm sure is not a trivial task to implement into a backup solution.

Having said that, you're more than welcome to modify/tweak the script to fit your environment but this is something that will not be supported.

Thanks for the comments.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

You also fail to understand the point of having either a global configuration for all VMs to be backed up or individual VM policy ...by having specifying individual VM to be backed up on the command line, you've introduced additional overhead which can be solved by using individual VM backup policy.

If the suggestion of allowing some type of post script to execute, this config would be stored in the VM policy file and you would still have 1 global vm list to backup but the specific params to each VM would be store in a configuration file which is why -c option is for.

Again, you can take a look at my previous reply, this defeats the purpose of this specific script not utilizing SSH console on either ESX(i) and you're more than welcome to modify/tweak the script to fit your environment but this is something that will not be supported.

Thanks for the commands.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

It was not my intention to break the "no SSH" requirements. i'm not yet on speed on what is possible with all

API's but my tough where going in the way of coping the script to the ESX(i) server and execute it there because

there is access to the backup files .

thinking longer on the subject , i guess that it would be hard to create scripts that can work in ESX/ESXi environment.

especially because ESXi is a very stripped busybox environment and VMWare would like it to be misused.

luc

What i fail to understand is what is option -c for, can't find a description for it.

My problem is that at the moment i have to maintain a configuration in the config dir and a vmlist file for each vm. My recomendation would just avoid this:

"Note: --vmlist is still a required parameter with the exception that backup policy files must exist for each VM in the list else the VM will be ignored in the backup process."

What i would like to achieve is that we just need to maintain one configuration file per backup policy (not per vm, thats to much work) and then call ghettoVCBg2 with the right policy for the right vm.

The postscripts im using do not connect to the ESXi host, they do manipulations in the NFS filer which is the datastore.

If you don't understand the -c which specifies a configuration directory for your individual VM backup policy, you may want to go through the documentation carefully and I've provided some example output to show you each of the options.

With your logic, you're effectively duplicating the number of files that need to be created and maintain to accomplish the exact same task that is already available in the script today.

There are two core use cases that I've designed the script to handle:

1) Global configuration where all VMs are backed up in the same manner (1 vmlist, no need for -c option)

2) Individual VM backup policy, where you MUST create individual policy on a per VM basis, as each VM differs from one another (1 vmlist, -c option required)

What i would like to achieve is that we just need to maintain one configuration file per backup policy (not per vm, thats to much work) and then call ghettoVCBg2 with the right policy for the right vm.

That is effectively option #2 and again running any type of post/pre scripts is not supported as this defeats the purpose of the script not requiring any type of SSH connectively to the ESX(i) host as mentioned in the last replies.

Again, you're more than welcome to add/modify the script to fit your needs, its definitely doable but will not be supported.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Hello.

I have a few machines which cannot be backuped by ghettovcbg2. After starting the script only the "vmx" file will be copied. No vmdk files ?! (snapshot is created and deleted) - I doesn't make any difference by using the old or new version of the script.

Any ideas ???

(As you can see -> 0 min to copy the vmdk!)

The debug log from the email:

11-30-2009 16:25:50 -- info: ============================== ghettoVCBg2 LOG START ==============================

11-30-2009 16:25:50 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

11-30-2009 16:25:50 -- info: CONFIG - VM_BACKUP_DATASTORE = nfs-storage

11-30-2009 16:25:50 -- info: CONFIG - VM_BACKUP_DIRECTORY = backup

11-30-2009 16:25:50 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

11-30-2009 16:25:50 -- info: CONFIG - ADAPTER_FORMAT = buslogic

11-30-2009 16:25:50 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

11-30-2009 16:25:50 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

11-30-2009 16:25:50 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

11-30-2009 16:25:50 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-30

11-30-2009 16:25:50 -- info: CONFIG - VM_VMDK_FILES = all

11-30-2009 16:25:50 -- debug: Main: Login by vi-fastpass to: esx01.short

11-30-2009 16:25:50 -- debug: copyTask: Task START

11-30-2009 16:25:50 -- debug: copyTask: waiting for next job and sleep ...

11-30-2009 16:25:52 -- info: Initiate backup for SH28S found on esx01.short

11-30-2009 16:25:59 -- debug: SH28S original powerState: poweredOff

11-30-2009 16:25:59 -- info: SH28S has 1 VMDK(s)

11-30-2009 16:25:59 -- debug: backupVMDK: Backing up "[msa-storage] SH28S/SH28S.vmdk" to "[nfs-storage] backup/SH28S/SH28S-2009-11-30/SH28S.vmdk"

11-30-2009 16:25:59 -- debug: backupVMDK: Signal copyThread to start

11-30-2009 16:25:59 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

11-30-2009 16:25:59 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

11-30-2009 16:25:59 -- debug: CopyThread: Start backing up VMDK(s) ...

11-30-2009 16:26:01 -- debug: copyTask: send copySuccess message ...

11-30-2009 16:26:01 -- debug: copyTask: waiting for next job and sleep ...

11-30-2009 16:27:00 -- debug: backupVMDK: Successfully completed backup for SH28S/SH28S.vmdk Elapsed time: 1 min

11-30-2009 16:27:01 -- debug: checkVMBackupRotation: Starting ...

11-30-2009 16:27:01 -- debug: Purging backup/SH28S/SH28S-2009-11-30--1 due to rotation max

11-30-2009 16:27:02 -- info: Backup completed for SH28S!

11-30-2009 16:27:02 -- debug: Main: Disconnect from: esx01.short

11-30-2009 16:27:02 -- debug: Main: Login by vi-fastpass to: esx02.short

11-30-2009 16:27:03 -- debug: Main: Disconnect from: esx02.short

11-30-2009 16:27:03 -- debug: Main: Calling final clean up

11-30-2009 16:27:03 -- debug: cleanUP: Thread clean up starting ...

11-30-2009 16:27:03 -- debug: cleanUp: Send exit to copyThread

11-30-2009 16:27:03 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

11-30-2009 16:27:03 -- debug: copyTask: die ...

11-30-2009 16:27:03 -- debug: cleanUp: Join passed

11-30-2009 16:27:03 -- info: ============================== ghettoVCBg2 LOG END ==============================

It looks like it might not have been able to locate the specific VMDK in question, right after locating the # of VMDK(s), you should see a line like this:

11-14-2009 00:19:45 --  debug: findVMDKFile: Found VMDK File: VIMA_1.vmdk

Can you paste the output of dryrun mode, please take a look at the documentation on how to enable dryrun and please wrap your output using tags for readability

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

11-30-2009 17:02:23 -- info: ============================== ghettoVCBg2 LOG START ==============================

11-30-2009 17:02:23 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

11-30-2009 17:02:23 -- info: CONFIG - VM_BACKUP_DATASTORE = nfs-storage

11-30-2009 17:02:23 -- info: CONFIG - VM_BACKUP_DIRECTORY = backup

11-30-2009 17:02:23 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

11-30-2009 17:02:23 -- info: CONFIG - ADAPTER_FORMAT = buslogic

11-30-2009 17:02:23 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

11-30-2009 17:02:23 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

11-30-2009 17:02:23 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

11-30-2009 17:02:23 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-30

11-30-2009 17:02:23 -- info: CONFIG - VM_VMDK_FILES = all

11-30-2009 17:02:23 -- debug: Main: Login by vi-fastpass to: esx01.short

11-30-2009 17:02:23 -- debug: copyTask: Task START

11-30-2009 17:02:23 -- debug: copyTask: waiting for next job and sleep ...

11-30-2009 17:02:25 -- info: -


DRYRUN DEBUG INFO SH28S -


11-30-2009 17:02:25 -- info: DEBUG - Host Build: VMware ESX Server 3i 3.5.0 build-123629

11-30-2009 17:02:25 -- info: DEBUG - Host: esx01.short

11-30-2009 17:02:25 -- info: DEBUG - Virtual Machine: SH28S

11-30-2009 17:02:25 -- info: DEBUG - VM ConfigPath: SH28S/SH28S.vmx

11-30-2009 17:02:25 -- info: DEBUG - VMX File: SH28S.vmx

11-30-2009 17:02:25 -- info: DEBUG - BackupConfigPath: backup/SH28S/SH28S-2009-11-30/SH28S.vmx

11-30-2009 17:02:25 -- info: DEBUG - BackupPath: backup/SH28S/SH28S-2009-11-30

11-30-2009 17:02:25 -- info: DEBUG - VM Datastore: msa-storage

11-30-2009 17:02:25 -- info: DEBUG - VMDK(s):

11-30-2009 17:02:25 -- info: DEBUG - SH28S/SH28S.vmdk

11-30-2009 17:02:25 -- info: -


DRYRUN DEBUG INFO SH28S -


11-30-2009 17:02:25 -- debug: Main: Disconnect from: esx01.short

11-30-2009 17:02:25 -- debug: Main: Login by vi-fastpass to: ESX02.short

11-30-2009 17:02:26 -- debug: Main: Disconnect from: ESX02.short

11-30-2009 17:02:26 -- debug: Main: Calling final clean up

11-30-2009 17:02:26 -- debug: cleanUP: Thread clean up starting ...

11-30-2009 17:02:26 -- debug: cleanUp: CopyTask was never started, send copyTaskStart

11-30-2009 17:02:26 -- debug: cleanUp: Send exit to copyThread

11-30-2009 17:02:26 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

11-30-2009 17:02:26 -- debug: copyTask: die ...

11-30-2009 17:02:26 -- debug: cleanUp: Join passed

11-30-2009 17:02:26 -- info: ============================== ghettoVCBg2 LOG END ==============================

I see that you're using ESXi 3.5, do you know which Update this is? and are you using the free version of ESXi?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Version 3.5 U3 and license is free! Please keep in mind that 90 percent of the other machines are working ...

Interesting, I can't see anything wrong from the logs and it seems like the VMDK passes the important checks such as not being physical RDM nor independent disk ... but it's only backing up the .vmx file ... usually this occurs if you're using the free version of ESXi and vmkfstools is not allowing you to do the copy of the VMDK.

Does this path still exists on your ESXi host:

\[nfs-storage\] backup/SH28S/SH28S-2009-11-30

If so, can we try manually calling vmkfstools and see if you are in fact using U3 which has the API bug?

vmkfstools --server <server_name> --username <username> -i "\[msa-storage\] SH28S/SH28S.vmdk" "\[nfs-storage\] backup/SH28S/SH28S-2009-11-30/SH28S.vmdk"

Please fill in the fields that match your system and we'll see if you get access denied OR if its actually issuing the backup.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

the path still exists

after entering your command the following message appears on the vima cli: "Unable to clone virtual disk : A general system error occurred: Internal error"

I assume you took a snapshot prior to executing the command? else it will not work

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

As I see in the GUI there is no open snapshot ...

For the command to work, you must have a snapshot else the disk is locked. Hopefully you understand why ...

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

no open or existing snapshot !

"vmkfstools -i" from the console on the esx works fine ! (probably the command syntax from vima is not correct)

...my problem is that there is no failure message ...

If the VM is powered on, then you need a snapshot, hopefully you understand this point. If the VM is powered off, then you don't need a snapshot

Try this ( for vmA, you'll need to specify both adapter/disktype from what I remember):

vmkfstools --server <server_name> --username <username> -i "\[msa-storage\] SH28S/SH28S.vmdk" -a buslogic -d thin "\[nfs-storage\ backup/SH28S/SH28S-2009-11-30/SH28S.vmdk"

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

instead of using "[msa-storage..." i've used the full qualified path "/vmfs...." for both (source and destination" and now it works !!

and what does this mean for my backup problem ??

Are you running this on VIMA 1.0 or vMA 4.0?

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

VIMA 1.0

I know there were some vmkfstools syntax issues on VIMA 1.0, can you deploy vMA 4.0 and try the backup on it? It should still work with ESXi 3.5u3 as the API is exposed from the host side. Let me know if that resolves the problem

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

ok I've done the update to vMa and the backup still doesn't work ... but ... I've got now an new failure message in my task pane in the infrastructure client gui

Task: "Copy Virtual Disk - General fault caused by file. One of the parameters supplied is invalid"

ok now i've found the problem with changing the script to give me an output when starting the copy command (like descriped before in the comments)

the problem was the diskformat ..like you wrote me I have to use "thin" but in the script was written "zeroedthick". At the moment I don't understand the reason why 95 percent of my VM work with zero and some only with thin ???

p.s. sorry for using your time to search a failure which was mine.....

New features/enhancements request:

eMail -> Possibility to change the subject (at the moment I do it manually Smiley Wink

Output -> Possibility to get an output on the shell what the script is doing at moment (like adding the print command)

Thankz

No worries, glad we were able to get to the bottom of this since it did sound like a very 'odd' issue.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Hey lamw,

nice scripts. it works fine for me so far. unless there is something wrong with the vmfs share.

if it doesn't exist or is not writeable, the script doesn't complain. it rather says the backup was ok and continuous. vms on the host, that have not been backed up yet, are then mentioned as "Unable to locate".

is this a known bug? let me know if u need any outputs of the logfiles.

i did not look at the source so far. just let me know if i should try any changes or whatever.

Kind regards

Julian

Can you define "something wrong with vmfs share" ? I'll have to go with the assumption that your destination volume is valid and available ... if it's not then that should be up to the administrator to ensure it is and if not, properly alert on it.

There are different issues that could occur when a volume is not available, either its not mounted like an NFS volume or a volume that has been dropped (FC/iSCSI LUN)...etc. Currently the script does not run any type of verifications on the destination and it'll try to create directories/etc and prep for the backup and if it's unable to copy, you'll get an error in the logs.

If it's still occurring, please run the script with debugging flag on and if you have questions about this, please refer to the documentation and remember to wrap your output in tags for readability.

Thanks

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

the definition for "something wrong" is: it doesn't exist or is not writeable

here is the log output when running a backup on a datastore which does not exist (same thing when the datastore is not writeable):

	12-03-2009 12:10:57 --  info: ============================== ghettoVCBg2 LOG START ==============================
	12-03-2009 12:11:08 --  info: CONFIG - USING CONFIGURATION FILE = necksolvminfra
	12-03-2009 12:11:08 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
	12-03-2009 12:11:08 --  info: CONFIG - VM_BACKUP_DATASTORE = netapp_b02
	12-03-2009 12:11:08 --  info: CONFIG - VM_BACKUP_DIRECTORY = ghettoVCB
	12-03-2009 12:11:08 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
	12-03-2009 12:11:08 --  info: CONFIG - ADAPTER_FORMAT = lsilogic
	12-03-2009 12:11:08 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
	12-03-2009 12:11:08 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
	12-03-2009 12:11:08 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
	12-03-2009 12:11:08 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-12-03
	12-03-2009 12:11:08 --  info: CONFIG - VM_VMDK_FILES = all

	12-03-2009 12:11:08 --  info: Initiate backup for necksolvminfra found on necksrvviesx02.comany.int
	12-03-2009 12:11:18 --  debug: necksolvminfra original powerState: poweredOn
	12-03-2009 12:11:18 --  debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-12-03" for necksolvminfra
	12-03-2009 12:11:28 --  info: necksolvminfra has 1 VMDK(s)
	12-03-2009 12:11:28 --  debug: backupVMDK: Backing up "[MR1_V00_P0A0_A] necksolvminfra/necksolvminfra.vmdk" to "[netapp_b02] ghettoVCB/necksolvminfra/necksolvminfra-2009-12-03/necksolvminfra.vmdk"
	12-03-2009 12:11:28 --  debug: backupVMDK: Signal copyThread to start
	12-03-2009 12:11:28 --  debug: backupVMDK: Backup progress: Elapsed time 0 min
	12-03-2009 12:11:28 --  debug: copyTask: Wake up and follow the white rabbit, with status: doCopy
	12-03-2009 12:11:28 --  debug: CopyThread: Start backing up VMDK(s) ...
	12-03-2009 12:11:35 --  debug: copyTask: send copySuccess message ...
	12-03-2009 12:11:35 --  debug: copyTask: waiting for next job and sleep ...
	12-03-2009 12:12:29 --  debug: backupVMDK: Successfully completed backup for [MR1_V00_P0A0_A] necksolvminfra/necksolvminfra.vmdk Elapsed time: 1 min
	12-03-2009 12:12:32 --  debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-12-03" for necksolvminfra
	12-03-2009 12:12:34 --  debug: checkVMBackupRotation: Starting ...
	12-03-2009 12:12:34 --  info: Backup completed for necksolvminfra!

	12-03-2009 12:12:34 --  debug: Main: Disconnect from: necksrvviesx02.comany.int

	12-03-2009 12:12:34 --  debug: Main: Login by vi-fastpass to: necksrvviesx03.comany.int
	12-03-2009 12:12:39 --  debug: Main: Disconnect from: necksrvviesx03.comany.int

	12-03-2009 12:12:39 --  debug: Main: Login by vi-fastpass to: necksrvviesx04.comany.int
	12-03-2009 12:12:43 --  debug: reConfigureBackupParams: VM - NeckSRVTermPST
	12-03-2009 12:12:43 --  info: CONFIG - USING CONFIGURATION FILE = NeckSRVTermPST
	12-03-2009 12:12:43 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
	12-03-2009 12:12:43 --  info: CONFIG - VM_BACKUP_DATASTORE = netapp_b02
	12-03-2009 12:12:43 --  info: CONFIG - VM_BACKUP_DIRECTORY = ghettoVCB
	12-03-2009 12:12:43 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
	12-03-2009 12:12:43 --  info: CONFIG - ADAPTER_FORMAT = buslogic
	12-03-2009 12:12:43 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
	12-03-2009 12:12:43 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
	12-03-2009 12:12:43 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = YES
	12-03-2009 12:12:43 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-12-03
	12-03-2009 12:12:43 --  info: CONFIG - VM_VMDK_FILES = all

	12-03-2009 12:12:43 --  info: Initiate backup for NeckSRVTermPST found on necksrvviesx04.comany.int
	12-03-2009 12:12:53 --  debug: NeckSRVTermPST original powerState: poweredOn
	12-03-2009 12:12:53 --  debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-12-03" for NeckSRVTermPST
	12-03-2009 12:13:03 --  info: NeckSRVTermPST has 1 VMDK(s)
	12-03-2009 12:13:03 --  debug: backupVMDK: Backing up "[MR2_V03_P1S1_A] NeckSRVTermPST/NeckSRVTermPST.vmdk" to "[netapp_b02] ghettoVCB/NeckSRVTermPST/NeckSRVTermPST-2009-12-03/NeckSRVTermPST.vmdk"
	12-03-2009 12:13:03 --  debug: backupVMDK: Signal copyThread to start
	12-03-2009 12:13:03 --  debug: backupVMDK: Backup progress: Elapsed time 0 min
	12-03-2009 12:13:03 --  debug: copyTask: Wake up and follow the white rabbit, with status: doCopy
	12-03-2009 12:13:03 --  debug: CopyThread: Start backing up VMDK(s) ...
	12-03-2009 12:13:10 --  debug: copyTask: send copySuccess message ...
	12-03-2009 12:13:10 --  debug: copyTask: waiting for next job and sleep ...
	12-03-2009 12:14:04 --  debug: backupVMDK: Successfully completed backup for [MR2_V03_P1S1_A] NeckSRVTermPST/NeckSRVTermPST.vmdk Elapsed time: 1 min
	12-03-2009 12:14:06 --  debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-12-03" for NeckSRVTermPST
	12-03-2009 12:14:08 --  debug: checkVMBackupRotation: Starting ...
	12-03-2009 12:14:08 --  info: Backup completed for NeckSRVTermPST!

	12-03-2009 12:14:08 --  debug: Main: Disconnect from: necksrvviesx04.comany.int

	12-03-2009 12:14:08 --  error: getFinalList: ERROR - Unable to locate VM: NeckLNXVMvma01

	12-03-2009 12:14:08 --  debug: Main: Calling final clean up
	12-03-2009 12:14:08 --  debug: cleanUP: Thread clean up starting ...
	12-03-2009 12:14:08 --  debug: cleanUp: Send exit to copyThread
	12-03-2009 12:14:08 --  debug: copyTask: Wake up and follow the white rabbit, with status: exit
	12-03-2009 12:14:08 --  debug: copyTask: die ...
	12-03-2009 12:14:08 --  debug: cleanUp: Join passed
	12-03-2009 12:14:08 --  info: ============================== ghettoVCBg2 LOG END ==============================

i cannot find an error in the log, or am i just blind?

And the machine NeckLNXVMvma01 is there. After a job on a host "fails", the loop which goes through the vms on a host seems not to finish.

I'll see if I can do a check on the destination datastore, there is an API property that checks to see if its accessible...but that may just mean if the ESX(i) host can see the datastore, it may not check to see if its actually mounted or if there are issues.

Again, I would probably say this is a corner case as I would hope that this type an issue where a volume is not available or having issue is properly monitored by the infrastructure VI team ... the expectation/assumption is that your environment has no issues when trying to run the backup

I'll let you know if I update the script, need to find some free time

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Hey,

u are right. The datastore should be monitored by the vi team. but maybe there is a typo in the backup configuration.

I don't know much about threads in perl, but can't u just check for the exit code of the vmkfstools?

[vi-admin@NeckLNXVMvma01 ~]$ vmkfstools --server necksrvviesx01.company.int --username root --password xxx -i "[MR2_V03_P1S1_A] NeckSRVVMPrt03/NeckSRVVMPrt03.vmdk"  -a lsilogic -d zerodthick  "[netapp_b02] ghettoVCB/NeckSRVVMPrt03/NeckSRVVMPrt03-2009-12-03/NeckSRVVMPrt03.vmdk"
Unable to clone virtual disk : vim.fault.InvalidDatastore
[vi-admin@NeckLNXVMvma01 ~]$ echo $?
1

the exit code is also != 0 when the store is not writeable.

something like that:

                if ($copyThreadStatus eq "doCopy"){
                        &log("debug", "CopyThread: Start backing up VMDK(s) ...");
                        eval {
                                my $vmkfstools_cpy = `$copyCmd` ;
                                if ($?) {
                                    die "Could not backup VMDK(s): $?, $vmkfstools_cpy";
                                }
                                &log("debug", "copyTask: send copySuccess message ...");
                                $copyThreadStatus= "copySuccess";
                        } or do {
                                &log("error", "CopyTask: ". $@ );
                                &log("error", "CopyTask: Cmd " .$copyCmd);
                                $copyThreadStatus= "copyFail";
                        }
                }

as i said, i don't know nothing about threads. so i have no clue if the example would work. at least it drops an error in my test. but the remaining machines on the host are still skiped. i dind't look into that yet.

Regarding the eMail log...

When the backup didn't completed well could this also be reflected in the subject ??? e.g "Incompete"

Hi,

I tried the new exciting script, but without success. Both powerOn and powerOff is not functional. Did I make a configuration mistake? The script connects to the ESXi, creates the user but make no snapshot or create the folders in the destination drive (NFS network storage, it's included at the source and vMA ESXi). Any idea? Thanks a lot.

12-09-2009 11:16:04 -- info: ============================== ghettoVCBg2 LOG START ==============================

12-09-2009 11:16:04 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

12-09-2009 11:16:04 -- info: CONFIG - VM_BACKUP_DATASTORE = nfs_storage_backup_35

12-09-2009 11:16:04 -- info: CONFIG - VM_BACKUP_DIRECTORY = vmbackups

12-09-2009 11:16:04 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

12-09-2009 11:16:04 -- info: CONFIG - ADAPTER_FORMAT = lsilogic

12-09-2009 11:16:04 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

12-09-2009 11:16:04 -- info: CONFIG - VM_SNAPSHOT_MEMORY = YES

12-09-2009 11:16:04 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = YES

12-09-2009 11:16:04 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-12-09

12-09-2009 11:16:04 -- info: CONFIG - VM_VMDK_FILES = all

12-09-2009 11:16:04 -- debug: Main: Login by vi-fastpass to: 192.168.3.94

12-09-2009 11:16:04 -- debug: copyTask: Task START

12-09-2009 11:16:04 -- debug: copyTask: waiting for next job and sleep ...

12-09-2009 11:16:07 -- debug: Main: Disconnect from: 192.168.3.94

12-09-2009 11:16:07 -- debug: Main: Login by vi-fastpass to: 192.168.3.38

12-09-2009 11:16:09 -- info: Initiate backup for ASP-VM-ADMIN found on 192.168.3.38

12-09-2009 11:16:16 -- debug: ASP-VM-ADMIN original powerState: poweredOn

12-09-2009 11:16:16 -- info: ASP-VM-ADMIN has 1 VMDK(s)

12-09-2009 11:16:16 -- debug: backupVMDK: Backing up "[datastore1] Test4Bkp/Test4Bkp.vmdk" to "[nfs_storage_backup_35] vmbackups/ASP-VM-ADMIN/ASP-VM-ADMIN-2009-12-09/Test4Bkp.vmdk"

12-09-2009 11:16:16 -- debug: backupVMDK: Signal copyThread to start

12-09-2009 11:16:16 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

12-09-2009 11:16:16 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

12-09-2009 11:16:16 -- debug: CopyThread: Start backing up VMDK(s) ...

12-09-2009 11:16:19 -- debug: copyTask: send copySuccess message ...

12-09-2009 11:16:19 -- debug: copyTask: waiting for next job and sleep ...

12-09-2009 11:17:16 -- debug: backupVMDK: Successfully completed backup for Test4Bkp/Test4Bkp.vmdk Elapsed time: 1 min

12-09-2009 11:17:16 -- warn: WARNING: Snapshot "ghettoVCBg2-snapshot-2009-12-09" not found

12-09-2009 11:17:18 -- debug: checkVMBackupRotation: Starting ...

12-09-2009 11:17:18 -- debug: Unable to purge vmbackups/ASP-VM-ADMIN/GET https:192.168.3.38foldervmbackupsASP-VM-ADMIN?dsName=nfs_storage_backup_35 unsuccessful : 404 Not Found due to rotation max

12-09-2009 11:17:18 -- info: Backup completed for ASP-VM-ADMIN!

12-09-2009 11:17:18 -- debug: Main: Disconnect from: 192.168.3.38

12-09-2009 11:17:18 -- debug: Main: Login by vi-fastpass to: 192.168.3.41

12-09-2009 11:17:19 -- debug: Main: Disconnect from: 192.168.3.41

12-09-2009 11:17:19 -- debug: Main: Calling final clean up

12-09-2009 11:17:19 -- debug: cleanUP: Thread clean up starting ...

12-09-2009 11:17:19 -- debug: cleanUp: Send exit to copyThread

12-09-2009 11:17:19 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

12-09-2009 11:17:19 -- debug: copyTask: die ...

12-09-2009 11:17:19 -- debug: cleanUp: Join passed

12-09-2009 11:17:19 -- info: ============================== ghettoVCBg2 LOG END ==============================

EDIT:

Found my mistake, the license is missing... But foundation license from vCenter? My ESXi doesn't know, if I have a licensed vCenter, does it? confused

Please refer to the documentation, your ESX(i) host must be fully licensed, you can NOT use a free version. The host can be a standalone or managed by vCenter, but if it's managed by vCenter, then it should also be properly licensed else you will not be able to use this script.

In a future update, I'll probably implement this validation check at the very beginning, I find that not everyone always goes through the documentation on first try.

Thanks

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

hi,

i'm seems to have some "logout" errors after a failed purge of rotated directory.

After backup is finished , directories are rotated and old are purged .

the purge of oldes directory seems to fail, which would lead to a logout of the VIM library.

I have a list of VM's and it happens on different places :

12-18-2009 12:16:05 -- info: ============================== ghettoVCBg2 LOG START ==============================

12-18-2009 12:16:09 -- info: CONFIG - USING CONFIGURATION FILE = ECL110

12-18-2009 12:16:09 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

12-18-2009 12:16:09 -- info: CONFIG - VM_BACKUP_DATASTORE = backup

12-18-2009 12:16:09 -- info: CONFIG - VM_BACKUP_DIRECTORY = ESX

12-18-2009 12:16:09 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

12-18-2009 12:16:09 -- info: CONFIG - ADAPTER_FORMAT = buslogic

12-18-2009 12:16:09 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

12-18-2009 12:16:09 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

12-18-2009 12:16:09 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

12-18-2009 12:16:09 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = WEEKLY

12-18-2009 12:16:09 -- info: CONFIG - VM_VMDK_FILES = all

12-18-2009 12:16:09 -- info: Initiate backup for ECL110 found on esx104.eclipseinternational.com

12-18-2009 12:16:27 -- debug: ECL110 original powerState: poweredOn

12-18-2009 12:16:27 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-12-18" for ECL110

12-18-2009 12:16:37 -- info: ECL110 has 1 VMDK(s)

12-18-2009 12:16:37 -- debug: backupVMDK: Backing up "[Storage1] ECL110/ECL110 - DC3.vmdk" to "[backup] ESX/ECL110/ECL110-WEEKLY/ECL110 - DC3.vmdk"

12-18-2009 12:16:37 -- debug: backupVMDK: Signal copyThread to start

12-18-2009 12:16:37 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

12-18-2009 12:16:37 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

12-18-2009 12:16:37 -- debug: CopyThread: Start backing up VMDK(s) ...

12-18-2009 12:31:39 -- debug: backupVMDK: Backup progress: Elapsed time 15 min

12-18-2009 12:46:42 -- debug: backupVMDK: Backup progress: Elapsed time 30 min

12-18-2009 13:01:47 -- debug: backupVMDK: Backup progress: Elapsed time 45 min

12-18-2009 13:16:49 -- debug: backupVMDK: Backup progress: Elapsed time 60 min

12-18-2009 13:31:51 -- debug: backupVMDK: Backup progress: Elapsed time 75 min

12-18-2009 13:46:54 -- debug: backupVMDK: Backup progress: Elapsed time 90 min

12-18-2009 14:01:56 -- debug: backupVMDK: Backup progress: Elapsed time 105 min

12-18-2009 14:16:58 -- debug: backupVMDK: Backup progress: Elapsed time 120 min

12-18-2009 14:32:01 -- debug: backupVMDK: Backup progress: Elapsed time 135 min

12-18-2009 14:47:03 -- debug: backupVMDK: Backup progress: Elapsed time 150 min

12-18-2009 15:02:06 -- debug: backupVMDK: Backup progress: Elapsed time 165 min

12-18-2009 15:17:08 -- debug: backupVMDK: Backup progress: Elapsed time 180 min

12-18-2009 15:32:10 -- debug: backupVMDK: Backup progress: Elapsed time 195 min

12-18-2009 15:44:37 -- debug: copyTask: send copySuccess message ...

12-18-2009 15:44:37 -- debug: copyTask: waiting for next job and sleep ...

12-18-2009 15:45:12 -- debug: backupVMDK: Successfully completed backup for ECL110/ECL110 - DC3.vmdk Elapsed time: 208 min

12-18-2009 15:45:35 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-12-18" for ECL110

12-18-2009 15:45:42 -- debug: checkVMBackupRotation: Starting ...

12-18-2009 15:46:32 -- info: Backup completed for ECL110!

12-18-2009 15:46:32 -- debug: Main: Disconnect from: esx104.eclipseinternational.com

12-18-2009 15:46:32 -- error: getFinalList: ERROR - Unable to locate VM: ECL131

... a list of

the output during backup shows this error :

>> ERROR on delete of a old directory

RESPONSE: $VAR1 = ' $vm_name});

this lines seems to fail after the logout.

the storage used for backups is a NFS mounted ReadyNas server.

luc

The reason it fails after this disconnect as you know, you're disconnecting from the server, so that line will error out since it can't locate the VM.

I've not seen this before, but it looks like it had issues trying to purge the old files. I know that uses of ghettoVCB see this time to time and generally it's been isolated to their NFS server timing out or having some input/output errors and is generally an issue with the NFS server. You may want to review the vmkernel logs during the time of the backup to get further insight on what happened.

To further isolate this, you can backup w/o rotating to see if you hit this error again and if you don't then its most likely the delete operation thats taking awhile or even commits of snapshots that's causing issues with your NFS. You may also want to review your NFS server logs to ensure you're not saturating the link or resources.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Hi William ,

i'll check my ESX server logs & NFS server log to see i can found some more info.

as far i can see , the rotation seems successful . files are moved correctly and 2 rotated backups exist .

luc

Hello William,

we are recently discovered your 4 version of ghettoVCBg2 and we are trying to use it from Vma, exists another more recent version than 4? if exists, where can we download it?

We have 3 Esxi Update 3 free version servers and one Esxi update 2 Foundation server, all with Vm´s on the local storage and with a shared NFS datastore in a Windows 2003 Server. We don´t have virtual center.

Our intentions are copy the local storage Vm´s of each server to NFS datastore. First we added all servers to Vma without problems with sudo vifp addserver IP and the password... when we execute the script from Vma "./ghettoVCBg2.pl --vmlist todas", the file todas contain the name of the Vm that we are using to test this proccess (owa in this case).. we obtain this log:

12-16-2009 07:05:22 -- info: ============================== ghettoVCBg2 LOG START ==============================

12-16-2009 07:05:22 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

12-16-2009 07:05:22 -- info: CONFIG - VM_BACKUP_DATASTORE = sntbck

12-16-2009 07:05:22 -- info: CONFIG - VM_BACKUP_DIRECTORY = SntwebservicesTEST

12-16-2009 07:05:22 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

12-16-2009 07:05:22 -- info: CONFIG - ADAPTER_FORMAT = lsilogic

12-16-2009 07:05:22 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

12-16-2009 07:05:22 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

12-16-2009 07:05:22 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

12-16-2009 07:05:22 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-12-16

12-16-2009 07:05:22 -- info: CONFIG - VM_VMDK_FILES = all

12-16-2009 07:05:22 -- debug: Main: Login by vi-fastpass to: 134.134.20.202

12-16-2009 07:05:22 -- debug: copyTask: Task START

12-16-2009 07:05:22 -- debug: copyTask: waiting for next job and sleep ...

12-16-2009 07:05:25 -- debug: Main: Disconnect from: 134.134.20.202

12-16-2009 07:05:25 -- debug: Main: Login by vi-fastpass to: 10.20.222.3

12-16-2009 07:05:27 -- debug: Main: Disconnect from: 10.20.222.3

12-16-2009 07:05:27 -- debug: Main: Login by vi-fastpass to: 10.20.222.2

12-16-2009 07:05:29 -- info: Initiate backup for owa found on 10.20.222.2

12-16-2009 07:05:45 -- debug: owa original powerState: poweredOn

12-16-2009 07:05:45 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-12-16" for owa

12-16-2009 07:05:56 -- info: owa has 1 VMDK(s)

12-16-2009 07:05:56 -- debug: backupVMDK: Backing up "[datastore1] owa/owa.fm.es.vmdk" to "[sntbck] SntwebservicesTEST/owa/owa-2009-12-16/owa.fm.es.vmdk"

12-16-2009 07:05:56 -- debug: backupVMDK: Signal copyThread to start

12-16-2009 07:05:56 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

12-16-2009 07:05:56 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

12-16-2009 07:05:56 -- debug: CopyThread: Start backing up VMDK(s) ...

12-16-2009 07:06:01 -- debug: copyTask: send copySuccess message ...

12-16-2009 07:06:01 -- debug: copyTask: waiting for next job and sleep ...

12-16-2009 07:06:56 -- debug: backupVMDK: Successfully completed backup for owa/owa.fm.es.vmdk Elapsed time: 1 min

12-16-2009 07:06:59 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-12-16" for owa

12-16-2009 07:07:02 -- debug: checkVMBackupRotation: Starting ...

12-16-2009 07:07:07 -- info: Backup completed for owa!

12-16-2009 07:07:07 -- debug: Main: Disconnect from: 10.20.222.2

12-16-2009 07:07:07 -- debug: Main: Calling final clean up

12-16-2009 07:07:07 -- debug: cleanUP: Thread clean up starting ...

12-16-2009 07:07:07 -- debug: cleanUp: Send exit to copyThread

12-16-2009 07:07:07 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

12-16-2009 07:07:07 -- debug: copyTask: die ...

12-16-2009 07:07:07 -- debug: cleanUp: Join passed

12-16-2009 07:07:07 -- info: ============================== ghettoVCBg2 LOG END ==============================

Everything seems to have gone well, but only the .vmx file is copied to NFS datastorage. In the vsphere console we can see that folder of the destination backup is created, snapshot is created end in the copy file proccess fail with "error general fault cause by file. One of the parameters supplied is invalid", in the forum we have seen that this can be an error with the value of variable $DISK_BACKUP_FORMAT, in our case we have test with all values without best results.

Thanks a lot William

Regards

Something caused it to fail the connection. Again, I would recommend isolating it local VMFS backup to ensure you're not having issues with your NFS server and monitor your NFS server during the period of backups and also monitor your ESXi host.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Thanks William,

I'm trying to copy the Vm to another local datastore and I'm getting the same error at the same point, it seems that the problem can not be the NFS server... Create the directories, copy the .vmx file, make snapshot and fail before start to copy..

what can be wrong?

Regards

If you look at my previous reply, the comment was not directed to you.

I'm not sure why it's not working for you .. potentially something might have changed between u2/u3. We'll probably need to manually issue the vmkfstools copy manually from vMA and see if it's actually cloning the disk.

Try this from vMA:

vmkfstools --server <host> --username <username> -i "\[datastore\] vmfolder/source.vmdk" -d zeroedthick -a lsilogic "\[datastore\] vmfolder/destination.vmdk"

This is on the assumption your VM is powered off or if it's still on, you need to issue a snapshot before you can clone the disk and see if this makes it to your destination datastore. This should hopefully help figure out what the issue might be.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

It seems there are some connection errors between ESX server and NFS host.

this what i could find on ESX4 server

vmkernel:Dec 21 12:03:58 esx104 vmkernel: 32:03:43:00.213 cpu6:4112)WARNING: NFS: 277: Lost connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

vmkernel:Dec 21 12:04:10 esx104 vmkernel: 32:03:43:12.678 cpu1:4397)NFS: 286: Restored connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

vmkernel:Dec 21 13:10:43 esx104 vmkernel: 32:04:49:45.530 cpu4:4112)WARNING: NFS: 277: Lost connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

vmkernel:Dec 21 13:13:03 esx104 vmkernel: 32:04:52:05.060 cpu2:4397)NFS: 286: Restored connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

vmkernel:Dec 22 04:37:36 esx104 vmkernel: 32:20:16:36.130 cpu12:4112)WARNING: NFS: 277: Lost connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

vmkernel:Dec 22 04:37:39 esx104 vmkernel: 32:20:16:39.680 cpu7:4383)NFS: 286: Restored connection to server 10.1.0.200 mount point /backupesx4, mounted as b416f3a1-85912734-0000-000000000000 ("backup")

It seems to reconnect , as i can browse the datastore.

The NFS server is a ready nas , no special errors there. maybe a network problem , may just a slow NFS server. it's only used for backup purposes.

Thanks William,

the same error with vmkfstools to a local datastore or a NFS too..

$ vmkfstools --server 134.134.20.2 --username root -i "[datastore1] vMA-ovf-4.0.0-161993/vMA-ovf-4.0.0-161993.vmdk" -d zeroedthick -a LSIlogic "[datastore2] vma/vma.vmdk"

Enter password:

Unable to clone virtual disk : General fault caused by file.

One of the parameters supplied is invalid

Yea if even the connection was lost for few seconds could cause issues with the backup, I would probably look into this before continuing, either the host can't handle it or your NAS server can't handle it. You may want to do a search in the general VMTN communities to see if there's any specific configurations you may need to set. I know for some NAS providers, you may need to increase the timeout value, again a search or new post in the general forum may yield better results. This has nothing to do with the script.

Good Luck

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

I think the error might be that you're using an invalid key for the adapter "LSIlogic" .... it's case sensitive and the supported strings are: buslogic, lsilogic or ide

Can you please re-try the command on your vMA 4.0 host

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at: http://engineering.ucsb.edu/~duonglt/vmware/

Twitter: @lamw

vGhetto Script Repository

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

Thanks William,

wea are using the correct form, i think, the first lines of the script are:

###############################################################

  1. NAME OF THE BACKUP DATASTORE

###############################################################

my $VM_BACKUP_DATASTORE = "Sntback";

###############################################################

  1. BACKUP DIRECTORY NAME ON DATASTORE

###############################################################

my $VM_BACKUP_DIRECTORY = "pruebasvlan1";

####################################################

  1. Number of backups for a given VM before deleting

####################################################

my $VM_BACKUP_ROTATION_COUNT = "3";

###################################################

  1. Supported backup types

  2. 'zeroedthick' 'eagerzeroedthick' 'thin' '2gbsparse'

###################################################

my $DISK_BACKUP_FORMAT = "zeroedthick";

###################################################

  1. Supported adapter types

  2. 'buslogic' 'lsilogic'

###################################################

my $ADAPTER_FORMAT = "buslogic";

###################################################################################################

  1. Shutdown guestOS prior to running backups and power them back on afterwards

  2. This feature assumes VMware Tools are installed, else hard power down will be initiated

  3. 1=enable, 0=disable (disable by default)

###################################################################################################

my $POWER_VM_DOWN_BEFORE_BACKUP = "0";

##############################################################

  1. VM BACKUP DIRECTORY NAMING CONVENTION (default -YYYY-MM-DD)

##############################################################

my $VM_BACKUP_DIR_NAMING_CONVENTION = timeStamp('YMD');

###################################################

  1. VM Snapshot Memory & Quiesce

  2. 1=enable, 0=disable (disable by default)

###################################################

my $VM_SNAPSHOT_MEMORY = "0";

my $VM_SNAPSHOT_QUIESCE = "0";

################################################

  1. LOG LEVEL VERBOSITY : "debug" or "info"

################################################

my $LOG_LEVEL = "debug";

                                                    1. DO NOT MODIFY PAST THIS LINE ##########################

We have tested on NFS datastores and local, with all vmdk format...

Any idea?

Version history
Revision #:
1 of 1
Last update:
‎04-22-2009 09:36 PM
Updated by: