ghettoVCBg2 - Free alternative for backing up VMs in ESX(i) 3.5 and 4.x (no SSH console required!)

ghettoVCBg2 - Free alternative for backing up VMs in ESX(i) 3.5 and 4.x (no SSH console required!)

Table of Contents:

    • Description
    • Features
    • Requirements
    • ESX/ESXi Version Support Table
    • Setup
    • Addings ESX/ESXi host(s) to VIMA/vMA
    • Configurations
    • Usage
    • Sample Execution 
      • Backup VMs located in a list
      • Backup VMs using individual backup policies
      • Dryrun Mode
    • Experimental Features
    • FAQ
    • Useful Links
    • Change Log

Description:

This tool is the follow up release of the ghettoVCB

backup utility which allows users to perform backups of virtual machines residing on ESX(i) 3.5+/4.x+

servers using methodology similar to VMware's VCB

tool. By incorporating highly constructive feedback from the VMware  community and utilizing the existing VI API, ghettoVCB’s framework was  completely rewritten to be harder, better, faster, stronger

.

The primary motivation for ghettoVCBg2 was to provide ESXi users with  access to the utility without relying on unlocking and utilizing the  unsupported console. To satisfy this requirement, the rebuilt framework  takes advantage of the

VI Perl Toolkit/vSphere SDK for Perl which is present in the

VIMA/vMA virtual appliance provided by VMware. As a result, ghettoVCBg2 provides  a more proper backup solution that administrators can utilize in their  virtual infrastructure.

As it stands, it is a requirement that ghettoVCB be invoked directly on  each of the ESX servers hosting virtual machines in need of a backup. By  taking advantage of VIMA/vMA, the entire backup process becomes  centralized in turn eliminating the abovementioned requirement.  Additionally, by leveraging the vi-fastpass library, unattended  authentication between VIMA/vMA and target ESX servers is made possible.  This alleviates the need to continually pass in credentials for each  ESX server.

By centralizing the backup process, individual virtual machine backup  lists pertaining to their respective ESX(i) servers are no longer  required. ghettoVCBg2 has the ability to identify virtual machines and  their respective hosts, potentially eliminating any past confusion  associated with maintaining multiple lists.

In its current configuration, the script will allow up to 3 unique  backups of the Virtual Machine before it will overwrite the previous  backups; this however, can be modified to fit procedures if need be.  Please be diligent in running the script in a test or staging  environment before using it on production live Virtual Machines; this  script functions well within our environment but there is a chance that  it may not fit well into other environments.

Features

  • Support for logging (normal & verbose)
  • Single VM backup list across multiple ESX/ESXi host(s)
  • Credential-less backups (so long as host(s) are being managed by VIMA/vMA)
  • Online back up of VM(s)
  • Only valid VMDKs presented to the VM will be backed up
  • Support for multiple VMDK backup per VM
  • Preserve original powerState of VM(s)
  • Ability to shutdown guestOS, initiate backup process and power on VM afterwards with the option of hardpower timeout
  • Ensure that snapshot removal process completes prior to continuing the backup process
  • VM(s) that initially contain snapshots and Physical RDMs (raw device mapppings) will not be backed up
  • Support for VM(s) with Virtual RDMs
  • Ability to specify the number of backup rotations per VM
  • Output back up VMDKs in ZEROEDTHICK (default behavior), EAGERZEROEDTHICK, 2GB SPARSE or THIN format
  • Ouput backup VMDKs using either BUSLOGIC or LSILOGIC adapter type
  • Fully support VMDKs stored across multiple datastores
  • VM snapshot memory and quiesce options
  • Individual VM backup policy (supported on ESX(i) 3.5u2+ & ESX(i) 4.x+)
  • Email logs (Experimental Support) NEW!
  • Ability to include/exclude specific VMDK(s) per VM (requires individual VM backup policy setup) NEW!
  • Independent disk aware (will ignore VMDK) NEW!
  • Additional debugging information including dry run execution NEW!


Requirements:

VMs running on ESX(i) 3.5u2+ or ESX(i) 4.x+

VMware VIMA 1.0 or VMware vMA 4.x


ESX/ESXi Version Support Table

VersionSupportAlternative
ESX 3.5u2Requires at least Foundation/Essentials licenseghettoVCB\
ESX 3.5u3Requires at least Foundation/Essentials licenseghettoVCB\
ESX 3.5u4Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 3.5u2Works for freeghettoVCB\
ESXi 3.5u3Works for freeghettoVCB\
ESXi 3.5u4Requires at least Foundation/Essentials licenseghettoVCB\
ESX 4.0Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 4.0Requires at least Foundation/Essentials licenseghettoVCB\
ESX 4.1Requires at least Foundation/Essentials licenseghettoVCB\
ESXi 4.1Requires at least Foundation/Essentials licenseghettoVCB\



Setup:

1) Download ghettoVCBg2.tar.gz

to your VIMA 1.0/vMA 4.x system

2) Extract the contents of the tarball

[vi-admin@scofield ~]$ tar -zxvf ghettoVCBg2.tar.gz
ghettoVCBg2/
ghettoVCBg2/ghettoVCBg2-vm_backup_configuration_template
ghettoVCBg2/ghettoVCBg2.pl

3) The script should be in a directory called

ghettoVCBg2


Addings ESX/ESXi host(s) to VIMA/vMA

Prior to starting, ensure that the VMware VIMA/vMA host is managing the appropriate ESX(i) 3.5u2/4.x+ host(s)

1. Add ESXi hosts to VIMA/vMA managment using vifp

A password prompt for the vi-admin account and root password to the pertinent ESXi host(s) will be presented.

[vi-admin@vima-primp-industries ~]$ sudo vifp addserver olga.resnet.ucsb.edu
root@olga.resnet.ucsb.edu's password:
[vi-admin@vima-primp-industries ~]$

2. Verify that the appropriate hosts are being managed by VIMA/vMA

[vi-admin@vima-primp-industries ~]$ sudo vifp listservers
olga.resnet.ucsb.edu
himalaya.primp-industries.com


Configurations

The following variables (similar to that of ghettoVCB) need to be defined within the script prior to execution.

Defining the datastore in which the backups are stored:

my $VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups";

Defining the name of folder that will contain backups (if folder does not exist, it will automatically be created):

my $VM_BACKUP_DIRECTORY = "WILLIAM_BACKUPS";

Defining the backup rotation per VM:

my $VM_BACKUP_ROTATION_COUNT = "3";

Defining the backup disk format (zeroedthick, eagerzeroedthick, thin, and 2gbsparse are available):

my $DISK_BACKUP_FORMAT = "zeroedthick";

Defining the adapter type for backed up VMDK (buslogic, and lsilogic are available):

my $ADAPTER_FORMAT = "buslogic";

Defining whether the VM is powered down or not prior to backup (1 = enable, 0 = disable):

Note: VM(s) that are powered off will not require snapshoting

my $POWER_VM_DOWN_BEFORE_BACKUP = "0";

Defining whether virtual machine memory is snapped and if quiescing is enabled (1 = enable, 0 = disable):

Note: By default both are disabled

my $VM_SNAPSHOT_MEMORY = "0";
my $VM_SNAPSHOT_QUIESCE = "0";

Defining the detail of logging (verbose or normal):

my $LOG_LEVEL = "normal";

*Defining VMDK(s) to backup from a particular VM either a list of vmdks or "all"

VM_VMDK_FILES="myvmdk.vmdk"

Experimental Support Configs:

Note: Port 25 by default

my $SEND_MAIL = "no";
my $EMAIL_HOST = "emailserver";
my $EMAIL_DOMAIN = "localhost.localdomain";
my $EMAIL_TO = 'William Lam <william@primp-industries.com.com>';
my $EMAIL_FROM = 'ghettoVCBg2 <ghettoVCBg2@primp-industries.com.com>';

Ensure that you do not edit past this section:

########################## DO NOT MODIFY PAST THIS LINE ##########################


Usage:

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl
Required command option 'vmlist' not specified.

Synopsis: ./ghettoVCBg2.pl OPTIONS


Command-specific options:
   --config_dir
      Name of directory containing VM(s) backup configurations
   --dryrun (default '0')
      Set to 1 to enable dryrun mode (default 0)
   --output (default '/tmp/ghettoVCBg2.log')
      Full path to output log (default /tmp/ghettoVCBg2.log)
   --vmlist (required)
      A file containing a list of virtual machine(s) to be backed up on host

Common VI options:
   --config (variable VI_CONFIG)
      Location of the VI Perl configuration file
   --credstore (variable VI_CREDSTORE)
      Name of the credential store file defaults to <HOME>/.vmware/credstore/vicredentials.xml on Linux and <APPDATA>/VMware/credstore/vicredentials.xml on Windows
   --encoding (variable VI_ENCODING, default 'utf8')
      Encoding: utf8, cp936 (Simplified Chinese), iso-8859-1 (German), shiftjis (Japanese)
   --help
      Display usage information for the script
   --passthroughauth (variable VI_PASSTHROUGHAUTH)
      Attempt to use pass-through authentication
   --passthroughauthpackage (variable VI_PASSTHROUGHAUTHPACKAGE, default 'Negotiate')
      Pass-through authentication negotiation package
   --password (variable VI_PASSWORD)
      Password
   --portnumber (variable VI_PORTNUMBER)
      Port used to connect to server
   --protocol (variable VI_PROTOCOL, default 'https')
      Protocol used to connect to server
   --savesessionfile (variable VI_SAVESESSIONFILE)
      File to save session ID/cookie to utilize
   --server (variable VI_SERVER, default 'localhost')
      VI server to connect to. Required if url is not present
   --servicepath (variable VI_SERVICEPATH, default '/sdk/webService')
      Service path used to connect to server
   --sessionfile (variable VI_SESSIONFILE)
      File containing session ID/cookie to utilize
   --url (variable VI_URL)
      VI SDK URL to connect to. Required if server is not present
   --username (variable VI_USERNAME)
      Username
   --verbose (variable VI_VERBOSE)
      Display additional debugging information
   --version
      Display version information for the script

Note: There are some additional command line arguments that can be provided upon runtime: --debug, --output and --vmlist. --vmlist is the only parameter required. The rest will have defaults if not explicitly defined.

--dryrun

  • Dry run information will be written to the log and no backup will take place

--output

  • Name of the output log, if the same log is used, it will  automatically append (ensure you provide full path e.g.  /tmp/ghettoVCBg2.log)

--config_dir

  • This directory will contain the individual VM backup policy files.  If a virtual machine has an associated policy file, the global  configuration will be ignored for that backup instance.

The input to this script is a file that contains the display name of the  virtual machine(s) separated by a newline. When creating this file on a  non-Linux/UNIX system, you may introduce a ^M character which may cause  the script to misbehave. To ensure that this does not occur, please  create the file on the ESX/ESXi host.

Provided is a sample of what the file should contain:

[vi-admin@scofield ~]$ cat virtual_machine_backup_list
VIMA
vMA-2


Sample Execution

  • Backup VMs located in a list
  • Backup VMs based on individual VM backup policies
  • Debug Mode
  • Dry run Mode

Backup VMs located in a list

  • Log verbosity: info (default)
  • Log output: /tmp/ghettoVCBg2.log (default)

DISK_FORMAT2gbsparse
ADAPTER_FORMATbuslogic
LOG_LEVELinfo
VM_SNAPSHOT_MEMORY0
VM_SNAPSHOT_QUIESCE0
VMS_TO_BACKUP(VIMA,vMA-2)


[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --vmlist virtual_machine_backup_list

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-13-2009 23:34:53 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-13-2009 23:34:53 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-13-2009 23:34:53 --  info: CONFIG - DISK_BACKUP_FORMAT = 2gbsparse
        11-13-2009 23:34:53 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-13-2009 23:34:53 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-13-2009 23:34:53 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-13
        11-13-2009 23:34:53 --  info: CONFIG - VM_VMDK_FILES = all

        11-13-2009 23:34:58 --  info: Initiate backup for VIMA found on himalaya.primp-industries.com
        11-13-2009 23:35:18 --  info: VIMA has 1 VMDK(s)
        11-13-2009 23:37:37 --  info: Backup completed for VIMA!

        11-13-2009 23:37:37 --  info: Initiate backup for vMA-2 found on himalaya.primp-industries.com
        11-13-2009 23:37:47 --  info: vMA-2 has 1 VMDK(s)
        11-13-2009 23:40:07 --  info: Backup completed for vMA-2!

        11-13-2009 23:40:10 --  info: ============================== ghettoVCBg2 LOG END ==============================

Backup VMs based on individual VM backup policies and log output to /tmp/ghettoVCB.log

  • Log verbosity: info (default)
  • Log output: /tmp/ghettoVCB.log (default)

VIMA

VM_BACKUP_DATASTOREdlgCore-NFS-bigboi.VM-Backups
VM_BACKUP_DIRECTORYWILLIAM_BACKUPS
DISK_FORMATzeroedthick
ADAPTER_FORMATbuslogic
LOG_LEVELdebug
VM_SNAPSHOT_MEMORY1
VM_SNAPSHOT_QUIESCE1
VM_VMDK_FILESVIMA_1.vmdk


vMA-2

VM_BACKUP_DATASTOREdlgCore-NFS-bigboi.VM-Backups
VM_BACKUP_DIRECTORYvGHETTO_BACKUPS
DISK_FORMATthin
ADAPTER_FORMATlsilogic
LOG_LEVELinfo
VM_SNAPSHOT_MEMORY0
VM_SNAPSHOT_QUIESCE0
VM_VMDK_FILESall


1. Create folder to hold individual VM backup policies (can be named anything):

[vi-admin@scofield ~]$ mkdir backup_config

2. Create individual VM backup policies for each VM and ensure each file  is named exactly as the display name of the VM being backed up (use  provided template to create duplicates):

[vi-admin@scofield backup_config]$ cp ghettoVCBg2-vm_backup_configuration_template VIMA
[vi-admin@scofield backup_config]$ cp ghettoVCBg2-vm_backup_configuration_template vMA-2

4. Using a preferred editor (nano or vi), edit each VM backup policy file

Listing of VM backup policy within backup configuration directory

[vi-admin@scofield backup_config]$ ls
ghettoVCBg2-vm_backup_configuration_template  VIMA  vMA-2

5. Pass in the value, backup_config, into the --config_dir parameter

(e.g.)
--config_dir backupConfigs

Example of two distinct VM backup policies will be applied to VIMA and vMA-2

Backup policy for "VIMA" (backup only 1 specific VMDKs)

[vi-admin@scofield backup_config]$ cat VIMA
VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups"
VM_BACKUP_DIRECTORY = "WILLIAM_BACKUPS"
VM_BACKUP_ROTATION_COUNT = "3"
DISK_BACKUP_FORMAT = "zeroedthick"
ADAPTER_FORMAT = "buslogic"
POWER_VM_DOWN_BEFORE_BACKUP = "0"
VM_SNAPSHOT_MEMORY = "1"
VM_SNAPSHOT_QUIESCE = "1"
LOG_LEVEL = "debug"
VM_VMDK_FILES = "VIMA_1.vmdk"

Backup policy for VM "vMA-2" (backup all VMDKs found)

VM_BACKUP_DATASTORE = "dlgCore-NFS-bigboi.VM-Backups"
VM_BACKUP_DIRECTORY = "vGHETTO_BACKUPS"
VM_BACKUP_ROTATION_COUNT = "3"
DISK_BACKUP_FORMAT = "thin"
ADAPTER_FORMAT = "lsilogic"
POWER_VM_DOWN_BEFORE_BACKUP = "0"
VM_SNAPSHOT_MEMORY = "0"
VM_SNAPSHOT_QUIESCE = "0"
LOG_LEVEL = "info"
VM_VMDK_FILES = "all"

The following is an output of the script when utilizing the new --config_dir parameter with debug mode enabled.

Note: --vmlist is still a required  parameter with the exception that backup policy files must exist for  each VM in the list else the VM will be ignored in the backup process.

Execution of backup

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --config_dir backup_config --vmlist virtual_machine_backup_list

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-14-2009 00:18:41 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-14-2009 00:18:45 --  info: CONFIG - USING CONFIGURATION FILE = VIMA
        11-14-2009 00:18:45 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-14-2009 00:18:45 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
        11-14-2009 00:18:45 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-14-2009 00:18:45 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:18:45 --  info: CONFIG - VM_SNAPSHOT_MEMORY = YES
        11-14-2009 00:18:45 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = YES
        11-14-2009 00:18:45 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:18:45 --  info: CONFIG - VM_VMDK_FILES = VIMA_1.vmdk

        11-14-2009 00:18:45 --  info: Initiate backup for VIMA found on himalaya.primp-industries.com
        11-14-2009 00:18:55 --  debug: VIMA original powerState: poweredOn
        11-14-2009 00:18:55 --  debug: Creating Snapshot "ghettoVCBg2-snapshot-2009-11-14" for VIMA
        11-14-2009 00:19:45 --  info: VIMA has 2 VMDK(s)
        11-14-2009 00:19:45 --  debug: findVMDKFile: Found VMDK File: VIMA_1.vmdk
        11-14-2009 00:19:45 --  debug: backupVMDK: Backing up "[http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk" to "[http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14/VIMA_1.vmdk"
        11-14-2009 00:19:45 --  debug: backupVMDK: Signal copyThread to start
        11-14-2009 00:19:45 --  debug: backupVMDK: Backup progress: Elapsed time 0 min
        11-14-2009 00:19:45 --  debug: copyTask: Wake up and follow the white rabbit, with status: doCopy
        11-14-2009 00:19:45 --  debug: CopyThread: Start backing up VMDK(s) ...
        11-14-2009 00:19:52 --  debug: copyTask: send copySuccess message ...
        11-14-2009 00:19:52 --  debug: copyTask: waiting for next job and sleep ...
        11-14-2009 00:20:46 --  debug: backupVMDK: Successfully completed backup for [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk Elapsed time: 1 min
        11-14-2009 00:20:49 --  debug: Removing Snapshot "ghettoVCBg2-snapshot-2009-11-14" for VIMA
        11-14-2009 00:20:51 --  debug: checkVMBackupRotation: Starting ...
        11-14-2009 00:20:57 --  debug: Purging [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-13--3 due to rotation max
        11-14-2009 00:21:05 --  info: Backup completed for VIMA!

        11-14-2009 00:21:05 --  debug: reConfigureBackupParams: VM - vMA-2
        11-14-2009 00:21:05 --  info: CONFIG - USING CONFIGURATION FILE = vMA-2
        11-14-2009 00:21:05 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DIRECTORY = vGHETTO_BACKUPS
        11-14-2009 00:21:05 --  info: CONFIG - DISK_BACKUP_FORMAT = thin
        11-14-2009 00:21:05 --  info: CONFIG - ADAPTER_FORMAT = lsilogic
        11-14-2009 00:21:05 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-14-2009 00:21:05 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:21:05 --  info: CONFIG - VM_VMDK_FILES = all

        11-14-2009 00:21:05 --  info: Initiate backup for vMA-2 found on himalaya.primp-industries.com
        11-14-2009 00:21:15 --  info: vMA-2 has 1 VMDK(s)
        11-14-2009 00:23:22 --  info: Backup completed for vMA-2!

        11-14-2009 00:23:25 --  info: ============================== ghettoVCBg2 LOG END ==============================

Dry run Mode (no backup will take place)

  • Log verbosity: drymode
  • Log output: /tmp/ghettoVCB.log (default)

[vi-admin@scofield ~]$ ./ghettoVCBg2.pl --config_dir backup_config --vmlist virtual_machine_backup_list --dryrun 1

[vi-admin@scofield ~]$ cat /tmp/ghettoVCBg2.log
        11-14-2009 00:26:29 --  info: ============================== ghettoVCBg2 LOG START ==============================
        11-14-2009 00:26:34 --  info: CONFIG - USING CONFIGURATION FILE = VIMA
        11-14-2009 00:26:34 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIRECTORY = WILLIAM_BACKUPS
        11-14-2009 00:26:34 --  info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick
        11-14-2009 00:26:34 --  info: CONFIG - ADAPTER_FORMAT = buslogic
        11-14-2009 00:26:34 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_MEMORY = YES
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = YES
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:26:34 --  info: CONFIG - VM_VMDK_FILES = VIMA_1.vmdk

        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO VIMA ----------
        11-14-2009 00:26:34 --  info: DEBUG - Host Build: VMware ESX 4.0.0 build-164009
        11-14-2009 00:26:34 --  info: DEBUG - Host: himalaya.primp-industries.com
        11-14-2009 00:26:34 --  info: DEBUG - Virtual Machine: VIMA
        11-14-2009 00:26:34 --  info: DEBUG - VM ConfigPath: [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - VMX File: VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupConfigPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14/VIMA.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] WILLIAM_BACKUPS/VIMA/VIMA-2009-11-14
        11-14-2009 00:26:34 --  info: DEBUG - VM Datastore: himalaya-local-SAS.Savvio
        11-14-2009 00:26:34 --  info: DEBUG - VMDK(s):
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA.vmdk
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Savvio|http://himalaya-local-SAS.Savvio] VIMA/VIMA_1.vmdk
        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO VIMA ----------

        11-14-2009 00:26:34 --  debug: reConfigureBackupParams: VM - vMA-2
        11-14-2009 00:26:34 --  info: CONFIG - USING CONFIGURATION FILE = vMA-2
        11-14-2009 00:26:34 --  info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DATASTORE = dlgCore-NFS-bigboi.VM-Backups
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIRECTORY = vGHETTO_BACKUPS
        11-14-2009 00:26:34 --  info: CONFIG - DISK_BACKUP_FORMAT = thin
        11-14-2009 00:26:34 --  info: CONFIG - ADAPTER_FORMAT = lsilogic
        11-14-2009 00:26:34 --  info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_MEMORY = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_SNAPSHOT_QUIESCE = NO
        11-14-2009 00:26:34 --  info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2009-11-14
        11-14-2009 00:26:34 --  info: CONFIG - VM_VMDK_FILES = all

        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO vMA-2 ----------
        11-14-2009 00:26:34 --  info: DEBUG - Host Build: VMware ESX 4.0.0 build-164009
        11-14-2009 00:26:34 --  info: DEBUG - Host: himalaya.primp-industries.com
        11-14-2009 00:26:34 --  info: DEBUG - Virtual Machine: vMA-2
        11-14-2009 00:26:34 --  info: DEBUG - VM ConfigPath: [http://himalaya-local-SAS.Constellation|http://himalaya-local-SAS.Constellation] vMA-2/vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - VMX File: vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupConfigPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] vGHETTO_BACKUPS/vMA-2/vMA-2-2009-11-14/vMA-2.vmx
        11-14-2009 00:26:34 --  info: DEBUG - BackupPath: [http://dlgCore-NFS-bigboi.VM-Backups|http://dlgCore-NFS-bigboi.VM-Backups] vGHETTO_BACKUPS/vMA-2/vMA-2-2009-11-14
        11-14-2009 00:26:34 --  info: DEBUG - VM Datastore: himalaya-local-SAS.Constellation
        11-14-2009 00:26:34 --  info: DEBUG - VMDK(s):
        11-14-2009 00:26:34 --  info: DEBUG - [http://himalaya-local-SAS.Constellation|http://himalaya-local-SAS.Constellation] vMA-2/vMA-2.vmdk
        11-14-2009 00:26:34 --  info: ---------- DRYRUN DEBUG INFO vMA-2 ----------

        11-14-2009 00:26:37 --  info: ============================== ghettoVCBg2 LOG END ==============================


FAQ

0Q: I'm having some issues or would like to provide feedback

0A: Please join the ghettoVCBg2 Group to post your comments/questions

1Q: I'm using ESXi and free licensed version ( I did not pay for anyting ), will this script still work?

1A: No, you need to have a licensed version of ESXi to use this script.

2Q: I have a VM listed in my backup file but no backup of that VM  took place. The log stated that the VM could not be found. How could  this be? I know that it’s running on host X…

2A: Ensure the ESX/ESXi host is being managed by VMware VIMA/vMA (sudo vifp listservers to verify)

3Q: Can I schedule backups to take place hourly, daily, monthly, yearly?

3A: Yes, do a search online for crontab.

4Q: I want to schedule my backup on Windows, how do I do this?

4A: Do a search for

plink. Make sure you have paired SSH keys setup between your Windows system and VIMA/vMA.

5Q: Do I have to use VMware VIMA/vMA to initiate the backup?

5A: Yes, vi-fastpass is being used and only VIMA/vMA provides  this capability. However, you can kickoff the process from another host  (*nix/win)

4Q: I only have a single ESXi host. I want to take backups and  store them somewhere else. The problem is: I don't have NFS, iSCSI nor  FC SAN. What can I do?

4A: You can use local storage to store your backups assuming that  you have enough space on the destination datastore.  Afterwards, you  can use scp (WinSCP/FastSCP) or vifs (vCLI) to transfer the backups from the ESXi host to your local desktop.

5Q: I’m pissed; the backup is taking too long. My datastore is of type X?

5A: YMMV, take a look at your storage configuration and make sure it is optimized.

6Q: I noticed that the backup rotation is occurring after a  backup. I don't have enough local storage space, can the process be  changed?

6A: This is primarily done to ensure that you have at least one  good backup in case the new backup fails. If you would like to modify  the script, you're more than welcome to do so.

7Q: What is the best storage configuration for datastore type X?

7A: Search the VMTN forums; there are various configurations for the different type of storage/etc.

8Q: I want to setup NFS Server to run my backups. Which is the best and should it be virtual or physical?7

8A: Please refer to answer 7A. From experience, we’ve seen  physical instances of NFS servers to be faster than their virtual  counterparts.  As always, YMMV.

9Q: When I try to use the script I get an error stating: "restricted version". What the heck is this?

9A: Please take a look at the ESX/ESXi Version Support Table in this document. The reason is explained here.

10Q: I'm getting error X when using the script or I'm not getting  any errors, the backup didn’t even take place. Oh what could be wrong?

10A: First, you'll want to check the log (default  /tmp/ghettoVCBg2) and see if there are any obvious errors. If not,  please provide the entire output of the log, a short description of your  environment and the type of VM(s) you're trying to backup—I will try to  provide some assistance.

11Q: I have VMs that have snapshots. I want to back these things up but the script doesn’t let me do it. How do I fix that?

11A: VM snapshots are not meant to be kept for long durations.  When backing up a VM that contains a snapshot, you should ensure all  snapshots have been committed prior to running a backup. No exceptions  will be made…ever.

12Q: I would like to restore from backup, what is the best method?

12A: The restore process will be unique for each environment and  should be determined by your backup/recovery plans. At a high level you  have the option of mounting the backup datastore and registering the VM  in question or copy the VM from the backup datastore to the ESX/ESXi  host. The latter is recommended so that you're not running a VM living  on the backup datastore or inadvertently modifying your backup VM(s).

13Q: When using the experimental feature: Individual VM backup policy, if I don't have a policy will it use the default configurations defined in the script?

13A: No, this use case will not be supported, ever. If you decide  to use the per VM policy, then the global configuration in the script  is ignored. If a VM policy config is not found, the VM will not be  backed up even if it's in the –-vmlist.

14Q: In vMA 4.0, vCenter target will be supported, can I just add vCenter target and assume this will work?

14A: At the moment, this is not a supported feature. You will  need to add each ESX/ESXi host for the vi-fastpass library to be able to  locate the host(s).

15Q: When I try to run the script I get: "-bash: ./ghettoVCBg2.pl: Permission denied", what is wrong?

15A: You need to change the permission on the script to be executable, chmod +x ghettoVCBg2.pl

16Q: Where can I download the latest version of the script?

16A: The latest version is available on this page at the bottom. To get the version of the script run

./ghettoVCBg2.pl --version

17Q: I'm running into an issue when I backup my VMs using NFS on  Linux, I'm able to either get 1 successful backup and all subsequent  backups fail OR I'm having issues where snapshots are not being removed  upon completition and future backups fail.

17A: This problem/issue has been reported by quite a few users in  the community and only affects a very small percentage of users overall  under a specific configuration. The problem as I understand it today  which is still an on-going investigation by myself and the community is  as follows:

The Setup

Setup: NFS on Linux where the underlying volume is EXT3 filesystem

and backups are failing primarily around larger sized VMs ( > 150GB).

The Observed Problem

The issue is during the final phase of a VM being backed-up in the  rotation function where a request is sent out to the NFS server to  either A) rotate/rename a directory B) delete an old backup. There is  either a disconnect or delay that severely affects the backup process  that may lead to a whole slew of "unexpected" behavior from having only 1  backup complete successful and all subsquent backups failing/etc.

First off, there is nothing wrong with the script from what we've found  it and has to do with NFS server and specifically the volume that the  backup datastore is hosted on. Thanks to community user

ralfoertner who had a simliar problem using

ghettoVCBg2 using Debian with EXT3 base filesystem for his NFS volume. He decided  to do some additional troubleshooting to further isolate the problem and  actually found by changing the underlying filesystem from EXT3 to XFS  that the issue went away and resolve all issues he had ran into  regarding an NFS on Linux backup solution.

The Theory

Now, we're still 100% sure why the above solution has resolved the  problem or what the actual problem is but I'll explain the current  theory:

EXT3 is a journaled filesystem, you can do a search online for more  information. There's a few advantages with EXT3 from it's predcessor  EXT2, but one difference is in the approach it takes with journaling as  explained here in greater detail:

http://www.ibm.com/developerworks/library/l-fs7.html

Basically EXT2 does only "metadata journling" whereas EXT3 does both  data and metadata journaling, this would ensure that all changes to the  filesystem where it's data or metadata is consistent in the event of an  unexpected system crash. With EXT3, there are 3 journaling options: (

data=writeback mode,data=ordered mode,data=journal mode ) and by default it uses

data=ordered mode to ensure the consistency as described from above.

One easy method to check what mount options are configured, on a Linux system you can check /proc/mounts:

[vi-admin@scofield ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/VolGroup00/var /var/log ext3 rw,data=ordered 0 0
/dev/sda1 /boot ext3 rw,data=ordered 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0

So now what does this have to do with XFS and EXT3? With the defaults of  EXT3, it may not always give you the best performance due to the extra  overhead of the way it does its journaling, if you were to change the  options to

data=writeback mode, you'll notice that this is simliar journaling method found in

XFS,JFS and ResierFS as described here:

http://www.ibm.com/developerworks/library/l-fs8.html

The Conclusion

I believe that there are some additional overheads that maybe caused by  using NFS on EXT3, which may explain the issue that users are seeing and  that moving from EXT3 to XFS has "seemed" to solve the issues for those  who have implemented the change. Though, before you go out and make  this change, you need to understand your environment and what this  change would exactly mean, please refer to differences in EXT2/XFS and  EXT3, especially around a power interruption or system crash.

In conclusion, I'm not recommending one way or the other, it really  depends on your environment. Though this has shown to resolve the issue  as users have implemented to get around this problem, but it's  definitely worth while to really understand the problem in your  environment and figure out why and decide on the best solution. I would  also recommend users perform their own tests to further isolate the  problem in their environment and try local storage, using iSCSI to see  if they can pin-point the problem.


Useful Links:

If you have found this script or other scripts to be useful and would like to contribute back, please click

here to donate! Thank You


Change Log

07/20/10 - build 4.3

Enhancements:

  • Support for vSphere 4.1 (ESX and ESXi)

Bug Fixes:

  • Due to changes in the VIFP Library introduced with vMA 4.1, the  script has been updated to work on both vMA 4.0 and vMA 4.1. Only  upgrade to vMA 4.1 if you're managing ESX(i) 4.1 hosts


03/27/10 - build 4.2

Enhancements:

  • Added a check to see if ESX(i) host is in a connected state
  • Added a check to see if ESXi host is using the free license, if so, skip host
  • Added FAQ #0

03/27/10 - build 4.1

Enhancements:

  • Updated FAQ #17 regarding "NFS" issue


02/24/09 - build 4.1

Bug Fixes:

  • There was a typo in handling the case of a VM's VMDK(s) being  distributed across multiple datastores in which the backup was not  creating the directory to reflect this layout, thanks to briandatamatters for finding this bug.

05/26/09 - build 2.5.1

Enhancements:

  • N/A

Bug Fixes:

  • Implemented shorterm solution fix for bug in VMware vCLI's vifs
  • Updated logic to allow virtual RDM to be backed up


04/25/09 - build 2.5

Enhancements:

  • Added experimental feature Individual VM backup policy
  • Added VM snapshot memory & quiesce options

Bug Fixes:

  • N/A


02/21/09 - build 2.0

Initial release

Attachments
Comments

HI William,

I'm using your script (and it is beautiful), but i'm running into a problem that I can't figure out...

I've created a backup list with all of the vmware machines that need to be backed up (about 15). The backup starts normally and files are being copied.

After some random time, the script indicates that several virtual machines can't be found (always random). When I create a new backup list with these virtual machines (same names) and run that, the backup script can find them and starts with the backup.

This second run also usually stops with the message that one or more virtual machines can't be found. Then i have to adjust the list again until all of the virtual machines have been backed up.

I'm using 3 vmware machines, all of them are licensed properly (vmware foundation). They can all access the backup medium (nfs share).

It is not a host related problem, when i create a list of 3 virtual machines that are all on the same server, the same thing happens.

This is quite annoying because now I have to keep an eye on the backup log the entire weekend to ensure it has done it's job..

Got any advice for me?

Thanks in advance! I truly love your work!

update

I've found something interesting.. I do a 3 VM (test) job (first located on host1, second located on host2 and third located on host1).

(host1: 10.10.1.51, host2: 10.10.1.52, host3: 10.10.1.54)

1. The script starts looking at host1 and finds the first vm, backup starts and finishes. The script searches for the second VM, doesn't find it on host1

2. The script goes on to host2, finds the second vm, backup starts and finishes. The script searches for the third VM, doesn't find it on host2, goes onto host 3. Doesn't find it on host 3 and says: error: getFinalList: ERROR - Unable to locate VM. Then the script stops

So the script isn't searching for the VM on the entire vmware farm, only on the same host and the next. Not the previous hosts.

Is this behaviour normal? In a non-vmotion environment, it's just a case of putting all of the backups in the right order. But when you do Vmotion, you never know where your VM's are at a specific time.

Below you can see the two runs that i did. First was with all three of the VM's in the backup list, second is with the one that didn't go at the first run.

FIRST RUN

06-23-2010 12:50:13 -- info: CONFIG - ADAPTER_FORMAT = buslogic

06-23-2010 12:50:13 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

06-23-2010 12:50:13 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

06-23-2010 12:50:13 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

06-23-2010 12:50:13 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2010-06-23

06-23-2010 12:50:13 -- info: CONFIG - VM_VMDK_FILES = all

06-23-2010 12:50:13 -- debug: Main: Login by vi-fastpass to: 10.10.1.51

06-23-2010 12:50:13 -- debug: copyTask: Task START

06-23-2010 12:50:13 -- debug: copyTask: waiting for next job and sleep ...

06-23-2010 12:50:16 -- info: Initiate backup for Topcon found on 10.10.1.51

06-23-2010 12:50:25 -- debug: Topcon original powerState: poweredOn

06-23-2010 12:50:25 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Topcon

06-23-2010 12:50:35 -- info: Topcon has 1 VMDK(s)

06-23-2010 12:50:35 -- debug: backupVMDK: Backing up "[OpenFiler 1] Topcon/Topcon.vmdk" to "[synw1] BACKUPS/Topcon/Topcon-2010-06-23/Topcon.vmdk"

06-23-2010 12:50:35 -- debug: backupVMDK: Signal copyThread to start

06-23-2010 12:50:35 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

06-23-2010 12:50:35 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

06-23-2010 12:50:35 -- debug: CopyThread: Start backing up VMDK(s) ...

06-23-2010 13:05:52 -- debug: backupVMDK: Backup progress: Elapsed time 15 min

06-23-2010 13:06:21 -- debug: copyTask: send copySuccess message ...

06-23-2010 13:06:21 -- debug: copyTask: waiting for next job and sleep ...

06-23-2010 13:06:53 -- debug: backupVMDK: Successfully completed backup for Topcon/Topcon.vmdk Elapsed time: 16 min

06-23-2010 13:06:56 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Topcon

06-23-2010 13:06:57 -- debug: checkVMBackupRotation: Starting ...

06-23-2010 13:08:01 -- info: Backup completed for Topcon!

06-23-2010 13:08:01 -- debug: Main: Disconnect from: 10.10.1.51

06-23-2010 13:08:01 -- debug: Main: Login by vi-fastpass to: 10.10.1.52

06-23-2010 13:08:05 -- info: Initiate backup for Monitoring Proxy found on 10.10.1.52

06-23-2010 13:08:13 -- debug: Monitoring Proxy original powerState: poweredOn

06-23-2010 13:08:13 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Monitoring Proxy

06-23-2010 13:08:23 -- info: Monitoring Proxy has 1 VMDK(s)

06-23-2010 13:08:23 -- debug: backupVMDK: Backing up "[OpenFiler 2] Monitoring Proxy/Monitoring Proxy.vmdk" to "[synw1] BACKUPS/Monitoring Proxy/Monitoring Proxy-2010-06-23/Monitoring Proxy.vmdk"

06-23-2010 13:08:23 -- debug: backupVMDK: Signal copyThread to start

06-23-2010 13:08:23 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

06-23-2010 13:08:23 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

06-23-2010 13:08:23 -- debug: CopyThread: Start backing up VMDK(s) ...

06-23-2010 13:14:59 -- debug: copyTask: send copySuccess message ...

06-23-2010 13:14:59 -- debug: copyTask: waiting for next job and sleep ...

06-23-2010 13:15:31 -- debug: backupVMDK: Successfully completed backup for Monitoring Proxy/Monitoring Proxy.vmdk Elapsed time: 7 min

06-23-2010 13:15:36 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Monitoring Proxy

06-23-2010 13:15:37 -- debug: checkVMBackupRotation: Starting ...

06-23-2010 13:15:59 -- info: Backup completed for Monitoring Proxy!

06-23-2010 13:15:59 -- debug: Main: Disconnect from: 10.10.1.52

06-23-2010 13:15:59 -- debug: Main: Login by vi-fastpass to: 10.10.1.54

06-23-2010 13:16:01 -- debug: Main: Disconnect from: 10.10.1.54

06-23-2010 13:16:01 -- error: getFinalList: ERROR - Unable to locate VM: Wiki

06-23-2010 13:16:01 -- debug: Main: Calling final clean up

06-23-2010 13:16:01 -- debug: cleanUP: Thread clean up starting ...

06-23-2010 13:16:01 -- debug: cleanUp: Send exit to copyThread

06-23-2010 13:16:01 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

06-23-2010 13:16:01 -- debug: copyTask: die ...

06-23-2010 13:16:01 -- debug: cleanUp: Join passed

06-23-2010 13:16:01 -- info: ghettoVCBg2 LOG END

SECOND RUN

06-23-2010 15:24:20 -- debug: Main: Login by vi-fastpass to: 10.10.1.51

06-23-2010 15:24:24 -- info: Initiate backup for Wiki found on 10.10.1.51

06-23-2010 15:24:47 -- debug: Wiki original powerState: poweredOn

06-23-2010 15:24:47 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Wiki

06-23-2010 15:24:57 -- info: Wiki has 1 VMDK(s)

06-23-2010 15:24:57 -- debug: backupVMDK: Backing up "[OpenFiler 2] Confluence/Confluence.vmdk" to "[synw1] BACKUPS/Wiki/Wiki-2010-06-23/Confluence.vmdk"

06-23-2010 15:24:57 -- debug: backupVMDK: Signal copyThread to start

06-23-2010 15:24:57 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

06-23-2010 15:24:57 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

06-23-2010 15:24:57 -- debug: CopyThread: Start backing up VMDK(s) ...

06-23-2010 15:40:08 -- debug: copyTask: send copySuccess message ...

06-23-2010 15:40:08 -- debug: copyTask: waiting for next job and sleep ...

06-23-2010 15:40:16 -- debug: backupVMDK: Successfully completed backup for Confluence/Confluence.vmdk Elapsed time: 15 min

06-23-2010 15:40:20 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2010-06-23" for Wiki

06-23-2010 15:40:22 -- debug: checkVMBackupRotation: Starting ...

06-23-2010 15:41:12 -- info: Backup completed for Wiki!

06-23-2010 15:41:12 -- debug: Main: Disconnect from: 10.10.1.51

06-23-2010 15:41:12 -- debug: Main: Login by vi-fastpass to: 10.10.1.52

06-23-2010 15:41:14 -- debug: Main: Disconnect from: 10.10.1.52

06-23-2010 15:41:14 -- debug: Main: Login by vi-fastpass to: 10.10.1.54

06-23-2010 15:41:15 -- debug: Main: Disconnect from: 10.10.1.54

06-23-2010 15:41:15 -- debug: Main: Calling final clean up

06-23-2010 15:41:15 -- debug: cleanUP: Thread clean up starting ...

06-23-2010 15:41:15 -- debug: cleanUp: Send exit to copyThread

06-23-2010 15:41:15 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

06-23-2010 15:41:15 -- debug: copyTask: die ...

06-23-2010 15:41:15 -- debug: cleanUp: Join passed

Hi Joriz, this seems to be about the ext3 issue that William documented above, do you have a nas with ext3 or ext4 file system?

Regards

Christian

Hi Christian,

I do have a NAS with ext3... I also have the async option enabled.. What is still don't understand is, what is the downside of using XFS as a storage filesystem?

Regards,

Joris

Jorizzzzzz we have the same problem (i described it at the head of topic) and find solution:

Run backup only for one vm (in vmlist) = "one vm per backup session".

For many vm`s we write simple shell script, that sequential run backup job for each vm in text file.

Switched over from the ghettoVCB to this g2 and nice, like the email feature.

Wondering...

1. Will the new Paravirtual SCSI controller type be added to .pl config file someday?

2. any ETA on FAQ #14 for vCenter support? without having to add each ESXi host?

3. forgot...can we have option to use an exclude list instead of include list?

4. like ghettoVCB, can this option offer compression and NON-PERSISTANT NFS?

Overall, looking good.

Joris, we use XFS with Openfiler and no issues.

Also...make sure ASYNC is enabled on the share for better performance.

Not a script expert so...

Trying to automate...anyone know how to create a sh script on the VMA to:

1. first gather VM names from all the ESXi hosts configured, output to a file with just the true VM name, nothing else, (drop the .vmx extension and volume path)

2. then parse through that file and remove VM's from a skip file (like good ole' ghettoVCB), that would then create a finished file for use in the ghettoVCBg2 command line option --vmlist

Thinking....

vmware-cmd -H server1 -U username -P password >> vm_to_backup_listing

vmware-cmd -H server2 -U username -P password >> vm_to_backup_listing

etc...

...which would append output to the vm_to_backup_listing file.

find VM's from skip.txt, script could then:

A. remove VM names from original "vm_to_backup_listing"

B. or create new polishedlist file.

then we'd run:

$ ./ghettoVCBg2.pl --vmlist polishedlist

Done!

If anyone can figure out, I'll place you on our Christmas list.

You're better off writing your own script querying exactly what you need vs trying to hack around existing scripts Smiley Wink

Luckily, this is a pretty simple task and I already have snippets of code that does these basic operations.

Take a look at: generateVMList.pl which can be run on vMA and connects to an ESX or ESXi host.

To execute you'll need to provide 2 inputs: --vm_exclusionfile (which is list of VMs to exclude) and --vm_outputfile (which is the name of the output file)

FYI - Donations are always welcome - http://www.virtuallyghetto.com/p/how-you-can-help.html

Thanks

BTW - Due to limited amount of free cycles and multiple projects I have going on both on and off work, I may or may not get to all the other questions and provide answers. Do know, I am very active on all 80+ scripts that I maintain, may take some time for me to get around Smiley Happy

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

Awesome! All Hail William!

Just so you know I was "trying" on my own script (for loop, strings, awk, etc....) but no expert.

You got all my requests, won't add anymore, promise.

Thanks again!

For those who would want a quick way to scan for all virtual machine names from all your ESXi hosts registered with VMA and place into 1 file then backup them up, see below, thanks to lamw (get his pl script above first). Make sure you place VM's to skip in exclusion file.

call this script "runbackup.sh"

#!/bin/bash

  1. scans all registered ESXi hosts in VMA for virtual machine names to backup

#VARIABLES FOR ESXi hosts

ESXHOSTS="server1 server2 server3 server4 server5"

USER=user

PASSWORD=password

#REMOVE OLD OUTPUTFILE to REFRESH LIST

rm vm_outputfile

  1. Getting VM info per esx host

for esxhost in $; do

./generateVMList.pl --vm_exclusionfile vm_exclusionfile --vm_outputfile vm_outputfile --server $ --username $ --password $

done

#just remove "--dryrun 1" to run the backup live!

./ghettoVCBg2.pl --vmlist vm_outputfile --dryrun 1

Hello,

i've exactly the same problem as Jorizzzzzz.

I also tested to mount my ext3 NFS share with "data=writeback" mode, but that is more worse then before.

Backup storage is a Openfiler NAS system with ext3+NFS. The VMs are at 15 - 40 GBytes.

If i remember right the problem exist since the update from ESXi 4.0 to 4.0u1.

Ditto that. Compression on the radar for g2 in 2010? NON-PERSISTANT NFS too? please... : )

Hi William, is there a chance to get a ghettoVCBg2 script for vMA 4.1 and ESXi 4.1?

Thank you.

It should work for vSphere 4.1 (ESX,ESXi and vMA 4.1) but will require a quick update to the script to allow for the new build number as it checks for 4.0.0 explicitly. I've not seen any changes in the API during the beta to cause the script to break but I've just not had the time to validate these scripts. Been having a lot of other things to take care of on my plate. If you want, you can try editing the script and just add and "OR" statement to allow for 4.1.0 and that should work, definitely test before rolling it out. I plan on pushing updates to both ghettoVCB and ghettoVCBg2 but not sure when that might be because I'm pretty booked for the next 3months with various projects.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

I tested it in vMA 4.1 and it doesn't work, there was some API functions (about creating user or user information) that raise an error. I'm sorry that I haven't more information now, but my vMA was in production at the time I test it and I rolled back to a 4.0 snapshot.

In vMA 4.0 and esxi 4.1 servers works fine

Did you perform an upgrade from vMA 4.0 -> 4.1 using vima-update?

afaik, it should migrate your exiting credstore to the new format and the hosts within the vi-fastpass should still work via the API calls.

If you can test with a dev environment, I would like to see what errors you're seeing. It may take me some time to upgrade our environment to vSphere 4.1, not rushing it since there are quite a few issues out in the while.

Thanks for your patience.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

Yes, I performed the upgrade using vima-update

After the upgrade I tested the script and it raised the error about the version so I fixed it.

After that, it raised an error with a api function, if I'm not wrong the error was in line 286 with the function vifplib_perl::CreateVIUserInfo()

To stay sure, I unregistered and registered again my esxi servers in vma and I tested vi-fastpass but nothing change (the error still there)

Yea, I actually had a moment to take a look and looks like VMware updated the vi-fastpass client libs and did not mention anything in their release notes. They did not even provide proper backwards compatibility mappings to their methods which is why you're seeing an error with that function call. I'll need to spend some time updating the script but it's not looking too pretty at the moment =/

Worse case, I'll have to put up a 4.1 specific version which is not ideal from a code management perspective. I'll see if I can get something out this weekend, but for now I would recommend staying on vMA 4.0. With respect to ghettoVCBg2 and vMA 4.1, there are 0 updates ... so there's no benefits in updating if you're just using it for this script.

This vSphere 4.1 release in general was not a smooth one by far .... there's been so many bugs, issues incompatibilities/etc. been a pain in the ass

Thanks for reporting the issue early and stay tuned.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

Firstly, thanks for what looks to be a very useful tool, and particularly for a speedy update to make it work with vMA4.1, which I installed yesterday and then cursed because the script didn't work... Smiley Happy

I have one problem so far and a couple of questions:

As with another user above, I'm finding that if I configure the script to use either zeroedthickor eagerzeroedthick, it doesn't back up the vmdk file, but if I configure the same VM to back up using 'thin', it works as expected. I have no problem with leaving it like that, as long as there are no disadvantages to doing so, or any problems that will crop up later? (haven't got as far as testing restores yet) A colleague has also been having exactly the same problem with ghettoVCB version 1 with a standalone ESXi box.

My hardware is a pair of ESX 3.5 hosts, with vCenter 2.5 - all licensed, and running on vMA 4.1. VMs are all stored on an iSCSI array, and using an NFS datastore for backups. The test VM has no snapshots and was powered off.

Questions next (I've tried searching but haven't found specific answers):

Is there actually a difference with setting lsilogic vs buslogic in the configuration? I always thought that a virtual disk was a virtual disk, and that just denoted what virtual SCSI card would be emulated in the guest? What does changing the controller type actually do?

As well as my main ESX boxes, I also have an ESXi box for testing. The vMA machine is running on the licensed ESX cluster, and configured to authenticate and talk to the ESXi machine. Am I right in assuming that as the ESXi box does not specifically have a license, I can't backup VMs on it, even though the ESX hosts running the vMA machine are licensed? (I've tried, and the backups fail with 'Can't get vim_service') The FAQ only talks about licensed vs unlicensed, not a mix of the two. It's not an issue if that's so, but if you think it should work I'll need to dig further into the problem..

Again, many thanks for developing this tool..

Good points on the 2 below:

>zeroedthickor eagerzeroedthick

Maybe someday we can have an auto-detect, pull from the VM setings, vmx file maybe? which one would yield faster backups?

>Is there actually a difference with setting lsilogic vs buslogic in the configuration? I always thought that a virtual disk was a virtual disk, and that just denoted what virtual SCSI card would be emulated in the guest? What does changing the controller type actually do?

again, could script detect from VMX file or other? we have a mix of lsi and bus, maybe others in the future.

Too many variables. LOL, other then that, ghettoVCBg2 is awesome!

lamw,

Problem:

Been using script for over a week now, with a 1.36TB NFS, we use a rotation of 1 copy - each VM since we can't compress.

Today I had our file server VM complete it's backup within 2 minutes, all that is left is the vmx file, there was prior copy on NFS before backup started. I had info and not debug configured in script, that has now been changed, ; ( , but... wondering if script is checking to make sure backup is 100% completed successfully before wiping out the older copy, still using the ESXi 4.0 script, not the latest one. Guessing the snapshot quiece failed mid-stream, there was no prior snapshot but your script checks that and works fine. Script does not log anything on ESXi host or vCenter when it does the snapshot.

Besides configuring "debug" and "keeping" 2 copies, any other ideas?

FYI:

Script needs a slight tweak.

Here is debug output. No failure message in email but it failed. The real issue is we did not have enough space to backup the VM, so, script needs to tell user "Failed to start backup due to not enough space to backup VM", deleted some VM's and it finally backed up. The white rabbit must have ate the message cause I did not get it. LOL.

07-23-2010 08:47:20 -- info: ============================== ghettoVCBg2 LOG START ==============================

07-23-2010 08:47:20 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

07-23-2010 08:47:20 -- info: CONFIG - VM_BACKUP_DATASTORE = VDRBACKUP

07-23-2010 08:47:20 -- info: CONFIG - VM_BACKUP_DIRECTORY =

07-23-2010 08:47:20 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

07-23-2010 08:47:20 -- info: CONFIG - ADAPTER_FORMAT = lsilogic

07-23-2010 08:47:20 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

07-23-2010 08:47:20 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

07-23-2010 08:47:20 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = YES

07-23-2010 08:47:20 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2010-07-23

07-23-2010 08:47:20 -- info: CONFIG - VM_VMDK_FILES = all

07-23-2010 08:47:20 -- debug: Main: Login by vi-fastpass to: server1

07-23-2010 08:47:20 -- debug: copyTask: Task START

07-23-2010 08:47:20 -- debug: copyTask: waiting for next job and sleep ...

07-23-2010 08:47:23 -- debug: Main: Disconnect from: server1

07-23-2010 08:47:23 -- debug: Main: Login by vi-fastpass to: server2

07-23-2010 08:47:25 -- debug: Main: Disconnect from: server2

07-23-2010 08:47:25 -- debug: Main: Login by vi-fastpass to: server3

07-23-2010 08:47:26 -- info: Initiate backup for PC601 FILE SERVER found on server3

07-23-2010 08:47:34 -- debug: PC601 FILE SERVER original powerState: poweredOn

07-23-2010 08:47:34 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2010-07-23" for PC601 FILE SERVER

07-23-2010 08:47:59 -- info: PC601 FILE SERVER has 1 VMDK(s)

07-23-2010 08:47:59 -- debug: backupVMDK: Backing up "[DataStore3] File Server/File Server.vmdk" to "[VDRBACKUP] /PC601 FILE SERVER/PC601 FILE SERVER-2010-07-23/File Server.vmdk"

07-23-2010 08:47:59 -- debug: backupVMDK: Signal copyThread to start

07-23-2010 08:47:59 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

07-23-2010 08:47:59 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

07-23-2010 08:47:59 -- debug: CopyThread: Start backing up VMDK(s) ...

07-23-2010 08:48:04 -- debug: copyTask: send copySuccess message ...

07-23-2010 08:48:04 -- debug: copyTask: waiting for next job and sleep ...

07-23-2010 08:48:59 -- debug: backupVMDK: Successfully completed backup for File Server/File Server.vmdk Elapsed time: 1 min

07-23-2010 08:49:08 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2010-07-23" for PC601 FILE SERVER

07-23-2010 08:49:09 -- debug: checkVMBackupRotation: Starting ...

07-23-2010 08:49:12 -- info: Backup completed for PC601 FILE SERVER!

07-23-2010 08:49:12 -- debug: Main: Disconnect from: server3

07-23-2010 08:49:12 -- debug: Main: Login by vi-fastpass to: server4

07-23-2010 08:49:14 -- debug: Main: Disconnect from: server4

07-23-2010 08:49:14 -- debug: Main: Login by vi-fastpass to: server5

07-23-2010 08:49:15 -- debug: Main: Disconnect from: server5

07-23-2010 08:49:15 -- debug: Main: Calling final clean up

07-23-2010 08:49:15 -- debug: cleanUP: Thread clean up starting ...

07-23-2010 08:49:15 -- debug: cleanUp: Send exit to copyThread

07-23-2010 08:49:15 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

07-23-2010 08:49:15 -- debug: copyTask: die ...

07-23-2010 08:49:15 -- debug: cleanUp: Join passed

07-23-2010 08:49:15 -- info: ============================== ghettoVCBg2 LOG END ==============================

Disk capacity and management is something I'm going to assume users are taking into consideration. There's nothing stopping you from creating a snapshot that fills up a datastore volume while the backup in place. I'll consider adding some checks but this can be difficult. I suggest you ensure that you actually have sufficient space for backups.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

...actually, your script MUST BE checking the space "needed" cause it goes to the next VM within minutes. I think it is an easy fix, the script just needs to say "failed" not "successful" backup when that occurs.

also...we really need the VMware vMA appliance to do some sort of CBT, inline compression, (maybe dedupe but that may be asking too much) in the future.

Again, I'll take your feedback into consideration and I'll see what I can do as time permits. Though you as an administrator should ensure that you have sufficient space for backups and setup proper alerts in case you hit thresholds before the volume fills up.

also...we really need the VMware vMA appliance to do some sort of CBT

Note, this is not a function of vMA but a function of vSphere. If you're looking for CBT, you need to look at true enterprise backup solution Smiley Wink Check out vDR, Veeam, etc. this is not an easy functionality to implement and would be a nice to have but for a free solution, it is out of the question.

Remember it's important to note that I manage more than 100+ scripts and I have requests and Q/A everyday, so I try to split my "free" time on the various projects to provide scripts and utilities to the community free of charge. Hopefully you understand and if you have found any of my scripts useful in your day to day life, feel free to support us - http://www.virtuallyghetto.com/p/how-you-can-help.html

Thanks

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

When executing ghettoVCBg2.pl by hand on vMA 4.1 it works like a charm, but when setting up a cronjob via crontab -e i get some strange errors, even when using /etc/crontab edited by sudo vi.

the error reads as follows:

Can't load '/usr/lib/perl5/site_perl/5.8.8/libvmatargetlib_perl.so' for module vmatargetlib_perl: libtypes.so: cannot open shared object file: No such file or directory at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.

at /usr/lib/perl5/site_perl/5.8.8/VMware/VmaTargetLib.pm line 10

Compilation failed in require at /usr/lib/perl5/site_perl/5.8.8/VMware/VIFPLib.pm line 10.

BEGIN failed--compilation aborted at /usr/lib/perl5/site_perl/5.8.8/VMware/VIFPLib.pm line 10.

Compilation failed in require at /DATEN/ghettoVCBg2/ghettoVCBg2.pl line 16.

BEGIN failed--compilation aborted at /DATEN/ghettoVCBg2/ghettoVCBg2.pl line 16.

i already made a sh-script where the path is set that is also used in the vi-admin-bash but it still does not work...

anyone any idea ?

What's most likely happening is the path to the Perl modules that are being used is not defined in the cron entry, remember by default cron will have a very limited default path defined.

Do a search online for setting the PATH for cron job and take a look at the existing PATH and you should be good once that is set

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

I added the SHELL and LD_LIBRARY_PATH entries found in vi-admin's /usr/bin/env to the cron job and it seems to have solved the problem.

How exactly did you do ?

adding this per crontab -e or did you add this to /etc/crontab ?

i tried to call this .sh file per crontab -e, but it still does not work Smiley Sad

SHELL=/bin/bash

PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/java/jre-vmware/bin:/sbin:/usr/sbin:/home/vi-admin/bin

LD_LIBRARY_PATH=:/opt/vmware/vma/lib64:/opt/vmware/vma/lib

/ghettoVCBg2/ghettoVCBg2.pl --dryrun --verbose --vmlist /ghettoVCBg2/virtual_machine_back_list&> /vcbg2.log

"#!/bin/bash -l" as shebang makes it work "-l"

makes bash act as if it had been invoked as a login shell and it reads the .profile

Smiley Happy

We seem to have a problem with the latest version. Everything seems to be working properly, the backups are being written to the designated location. But once every single backup is finished the backup files get deleted. The folders are still there, but it contents have disappeared. I changed the destination from iSCSI to NFS and back, and also the backup format (from thin to zeroedthick) to no avail. I also thought it to be possible that there might be something wrong with the number of backups that are saved but changing that number does not help either. Is there anyone that can shed a light on this, or maybe has the same issue?

See also the log below where we do a backup of the vMA itself.

JW

08-12-2010 18:13:09 -- info: ============================== ghettoVCBg2 LOG START ==============================

08-12-2010 18:13:09 -- debug: copyTask: Task START

08-12-2010 18:13:09 -- debug: copyTask: waiting for next job and sleep ...

08-12-2010 18:13:09 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCBg2.log

08-12-2010 18:13:09 -- info: CONFIG - VM_BACKUP_DATASTORE = sun01-nfs-share01

08-12-2010 18:13:09 -- info: CONFIG - VM_BACKUP_DIRECTORY = ghettoVCB

08-12-2010 18:13:09 -- info: CONFIG - DISK_BACKUP_FORMAT = zeroedthick

08-12-2010 18:13:09 -- info: CONFIG - ADAPTER_FORMAT = buslogic

08-12-2010 18:13:09 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = NO

08-12-2010 18:13:09 -- info: CONFIG - VM_SNAPSHOT_MEMORY = NO

08-12-2010 18:13:09 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = NO

08-12-2010 18:13:09 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2010-08-12

08-12-2010 18:13:09 -- info: CONFIG - VM_VMDK_FILES = all

08-12-2010 18:13:09 -- debug: Main: Login by vi-fastpass to: ndims-beh02.****.****

08-12-2010 18:13:10 -- warn: Unable to login to: ndims-beh02.****.**** - maybe offline or unreachable - skipping

08-12-2010 18:13:10 -- debug: Main: Login by vi-fastpass to: vmhost14.****.****

08-12-2010 18:13:14 -- debug: Main: Disconnect from: vmhost14.****.****

08-12-2010 18:13:14 -- debug: Main: Login by vi-fastpass to: vmhost13.****.****

08-12-2010 18:13:20 -- debug: Main: Disconnect from: vmhost13.****.****

08-12-2010 18:13:20 -- debug: Main: Login by vi-fastpass to: vmhost12.****.****

08-12-2010 18:13:26 -- debug: Main: Disconnect from: vmhost12.****.****

08-12-2010 18:13:26 -- debug: Main: Login by vi-fastpass to: vmhost11.****.****

08-12-2010 18:13:29 -- info: Initiate backup for vSphere Management Assistant (vMA) found on vmhost11.****.****

08-12-2010 18:13:33 -- debug: vSphere Management Assistant (vMA) original powerState: poweredOn

08-12-2010 18:13:33 -- debug: Creating Snapshot "ghettoVCBg2-snapshot-2010-08-12" for vSphere Management Assistant (vMA)

08-12-2010 18:13:43 -- info: vSphere Management Assistant (vMA) has 1 VMDK(s)

08-12-2010 18:13:43 -- debug: backupVMDK: Backing up "[sun01-nfs-share01] vSphere Management Assistant (vM/vSphere Management Assistant (vM.vmdk" to "[sun01-nfs-share01] ghettoVCB/vSphere Management Assistant (vMA)/vSphere Management Assistant (vMA)-2010-08-12/vSphere Management Assistant (vM.vmdk"

08-12-2010 18:13:43 -- debug: backupVMDK: Signal copyThread to start

08-12-2010 18:13:43 -- debug: backupVMDK: Backup progress: Elapsed time 0 min

08-12-2010 18:13:43 -- debug: copyTask: Wake up and follow the white rabbit, with status: doCopy

08-12-2010 18:13:43 -- debug: CopyThread: Start backing up VMDK(s) ...

08-12-2010 18:14:57 -- debug: copyTask: send copySuccess message ...

08-12-2010 18:14:57 -- debug: copyTask: waiting for next job and sleep ...

08-12-2010 18:15:44 -- debug: backupVMDK: Successfully completed backup for vSphere Management Assistant (vM/vSphere Management Assistant (vM.vmdk Elapsed time: 2 min

08-12-2010 18:15:46 -- debug: Removing Snapshot "ghettoVCBg2-snapshot-2010-08-12" for vSphere Management Assistant (vMA)

08-12-2010 18:15:47 -- debug: checkVMBackupRotation: Starting ...

08-12-2010 18:15:54 -- info: Backup completed for vSphere Management Assistant (vMA)!

08-12-2010 18:15:54 -- debug: Main: Disconnect from: vmhost11.****.****

08-12-2010 18:15:54 -- debug: Main: Login by vi-fastpass to: vmhost10.****.****

08-12-2010 18:15:57 -- debug: Main: Disconnect from: vmhost10.****.****

08-12-2010 18:15:57 -- debug: Main: Calling final clean up

08-12-2010 18:15:57 -- debug: cleanUP: Thread clean up starting ...

08-12-2010 18:15:57 -- debug: cleanUp: Send exit to copyThread

08-12-2010 18:15:57 -- debug: copyTask: Wake up and follow the white rabbit, with status: exit

08-12-2010 18:15:57 -- debug: copyTask: die ...

08-12-2010 18:15:57 -- debug: cleanUp: Join passed

08-12-2010 18:15:57 -- info: ============================== ghettoVCBg2 LOG END ==============================

As it appears the folder in which the backup first appears is renamed after the backup. A dash and a number are being added to the end of the name. When one refreshes the vSphere client that I used to browse the datastore it shows an empty folder, where it should be saying that the folder does not exist anymore. When one goes up one level, refreshes and drills down one level, the backup files are there. A whole lot less complicated than I thought it was, but still rather confusing.

JW

I am running ESXi 4.1 booting to a USB. I have no native OS running on the bare metal. Do I need ESXi running on a host OS in order to use ghettoVCBg2?

I've not tested it with ESXi running off a USB ... though the backup does not rely on where ESXi image is running off of, it's more of where the backups will be stored. As long as a valid datastore is seen by the host and you're using a licensed version of ESXi, this script should function. Please take a look at the documentation for more details around the backup

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

Can test ghettovcb with an unlicensed version? or will it not even function in anyway?

Please take a look at the documentation.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

Hello!

First of all tanks for the great script!

I'am using the script for quite a while and I was really happy with it.

Then the system disk of my nfs-server crashed. I had no backup, so I had to reinstall the whole server sytem.

After I was finished and made all the connections to the esxi server the backup failed to backup the second VM.

I changed to the newest version of the script but nothing changed. The log always says that the the second VM could not be found.

I changed the destination datastore - no change.

I changed the first vm to be the second in the backup - no change.

Also the performance of the backup is much more slow than before.

Thank you for your help!

Thank you for making this script. We appreciate having an affordable option for backing up VMware guests. Like several other users we are also experiencing the "Unable to locate VM:" errors. Our datastore for backups is also an NFS share with ext3 filesystem. Any plans for a fix to the ext3/NFS issue in the near future? Will switching to xfs definitely fix the problem?

Hi, sean.carolan, in my case I'm still having that issue also with XFS. The only way I found to definitely solve the issue is to do a bash script where I run the Getto script once for each VM.

Wow...that's sort of...non-ideal. I will consider taking the one-script-instance per VM route if there are no other options.

Great script, as usual, but can I run the script against a single host and not all hosts registered in the vMA? I didn't find an option to do this (--server doesn't).

lamw,

Any tips for us on killing the PID ghettoVCBg2 creates? I know you have solution for the "other" ghettoVCB (storing PID in log)

I want to stop ghettoVCBg2 at 6am on Mondays thru Sat. Don't want backups running during business hours.

Thinking, we could edit your script and add (somewhat, not sure perl code method):

echo $$ > pid.pid

I could then read this file "pid.pid" (containing PID) in my taskkill.sh script inside our crontab, then stop the backup.

Any thoughts?

MKguy,

Per the documentation, the script does not run against any specific ESX(i) host, it relies on vi-fastpass targets being added into vMA and based on the VM backup list, it'll traverse through all hosts locating the VMs for you. If it finds all the VMs in the first say 5 out of 10 hosts, it'll disconnect and quit after the 5th and not try to go on. If it doesn't find the VM, it will state a warning at the very end.

Hopefully this makes more sense

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

vmbru,

This is something I could look into adding, though it's pretty trivial to get the PID but you will not be using any type of "echo" statement as that is not how you do it in Perl. You'll use the same syntax, the "$$" will give you the PID of the script which can then be printed in the logs, similar to how is doing it today.

Though the real issue looks like you're trying to "kill" the script during a certain period of time? Why not just use cron to schedule your backups so that it does not conflict with your business hours? I think that is the right way of doing it versus trying to kill the script, the latter should only be used if an issue occurs or you need to stop the script. In either case, you should be able to look up the PID by running "ps -ef" and locating the ghettoVCBg2 script being executed.

=========================================================================

William Lam

VMware vExpert 2009,2010

VMware scripts and resources at:

Twitter: @lamw

vGhetto Script Repository

Getting Started with the vMA (tips/tricks)

Getting Started with the vSphere SDK for Perl

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

lamw,

Trust me, I'd rather not try to kill the ghetto script/process, I just have many VM's and TB's of data to backup and it could run thru business hours. I also use VDR and it shuts down further backups when the backup window is exceeded, just trying to add that feature to your script. My crontab is setup to only start your script at 6:35pm M-F and 1:35pm on Sat.

I was able to edit your script right before the EMAIL CONF section and added:

open (FILE, ">>/home/vi-admin/pid.pid");

print FILE "$$\n";

close (FILE);

...this creates file called pid.pid, with the initial PID (there are other PIDS, calls to esxi hosts, etc...) but figure safer to kill the original 1st generated one), that my taskkill.sh script (below) reads:

taskkill.sh

arr=(`cat pid.pid`)

kill -15 ${arr[*]}

exit 0

I have a kickoff script that generates the list of VM's to backup that clears this PID file too.

Anyway, this seems to work for me, it kills the remainder of process and allows the current backup to finish (which is ok)...as far as I can tell from testing for past 2 days.

So...guess I'm good for now.

Well, my problem is that I want to use the local datastore of all my hosts to store the VM backups on, which are named differently for each host.

So what I'm doing currently looks like this:

- prior to the backups, run a custom script which connects to each host and outputs which VMs run on this hostX (of which I want to backup) into textfileX

- have a ghettovcbg2.pl scripts for each host, only difference is the $VM_BACKUP_DATASTORE variable

- run ghettovcbg2-hostX.pl --vmlist textfileX for each host respectively

I know this system isn't perfect, even if I could specify hosts; in fact it may seem even dumb, but do you know a better way to achieve this?

Hi all,

I have a problem running g2. I currently use ghettoVCB on my esxi servers with no problem, but I would like to upgrade to g2 and found a bizzarre problem... Here what I did:

- downloaded and installed vma 4.1 via ovf

- setup vma

- run vifp to add a server of mine, and it correctly shows in vifp listservers

- created a file with only one line containing "XP" without quotes (an xp pro test machine)

- downloaded g2 and untarred in its default directory

- modified variables to my needs

- run ghettoVCBg2.pl --vmlist mylist

The problem is that is does nothing. It only passes to the next line and the shell prompt returns. On my ESXi 4.1 (licensed) nothing runs.

Also the original script, with default variables, does nothing. No errors. Loglevel is set to Debug, but nothing happens.

If I run the script with no options it gives me the man page. If I put a wrong machine name in the list-file it does nothing, if I put the wrong name for the list file it complains about it.

I'm probably missing something very big, but I read all the FAQs and posts and I'm stuck here. Simply nothing happens.

Specifying a server with --server does not help. If I put a dummy server name I obtain no error.

--version is

vSphere SDK for Perl version: 4.1

Script 'ghettoVCBg2.pl' version: 4.3

A thread exited while 2 threads were running

Any ideas?

Any idea on how to use this with VMA4.1? I am getting an error: Use of inherited AUTOLOAD for non-method vifplib_perl::CreateVIUserInfo() is depreciated at ./ghettoVCBg2.pl line 305

Thanks!

nevermind - found the problem with the .pl script. Since I did an in-place upgrade of vMA both the vima-release and vma-release files existed in the /etc directory. I modified the script to look for vma-release first and the above error went away.

William, Thanks for all that you do!

Hi tschneider,

I'm having the same issue and would be grateful if you could let me know what changes you made to the script to get this to work.

Thanks

Paul

Version history
Revision #:
1 of 1
Last update:
‎04-22-2009 09:36 PM
Updated by: