How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud

[#1]

Summary

An interesting question was posed on this thread: Re: Backup to Amazon S3 on whether or not the Amazon s3cmd Utility would install and function within the unsupported ESXi Busybox console.  As you know, users should not install any applications within this  limited console, it's generally used when troubleshooting with VMware  Support and that's when you'll unlock this console.

I decided to take on the challenge and see if it would be possible to install the s3cmd on ESXi, the installation of the utility is basically a python script  though it requires additional python libraries that just don't exists  with the ESXi Busybox console. The work around is to use an  Ubuntu VM to install the s3cmd utility and copy over the necessary libraries to get it to function on ESXi. To install s3cmd on classic ESX is even eaiser, the additional python libraries already exists.

Requirements

• Ubuntu 9.04 64bit VM w/apt-get enabled and running (only for ESXi)
• Amazon s3cmd utility ( s3cmd-0.9.9.91.tar.gz )
• ESXi 4.0 with unsupported Busybox console unlocked & SSH access enabled OR ESX 4.0

Install Amazon s3cmd utility on ESXi 4.0

1. Setup Ubuntu 9.04 64bit VM that has the package manager apt-get enabled and running. (If you don't know how to set this up, search on Google)

root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 9.04
Release:        9.04
Codename:       jaunty

root@ubuntu:~# uname -a
Linux ubuntu 2.6.28-11-server #42-Ubuntu SMP Fri Apr 17 02:45:36 UTC 2009 x86_64 GNU/Linux

2. Download s3cmd-0.9.9.91.tar.gz and upload to your Ubuntu VM

3. Install python2.5 interpreter on your Ubuntu VM since python2.5 is the version ESXi Busybox is running

root@ubuntu:~# apt-get install python2.5

4. Extract the contents of s3cmd-0.9.9.91.tar.gz:

root@ubuntu:~# tar -zxvf s3cmd-0.9.9.91.tar.gz:
s3cmd-0.9.9.91/
s3cmd-0.9.9.91/S3/
s3cmd-0.9.9.91/S3/PkgInfo.py
s3cmd-0.9.9.91/S3/S3.py
s3cmd-0.9.9.91/S3/S3Uri.py
s3cmd-0.9.9.91/S3/ACL.py
s3cmd-0.9.9.91/S3/Progress.py
s3cmd-0.9.9.91/S3/Exceptions.py
s3cmd-0.9.9.91/S3/SimpleDB.py
s3cmd-0.9.9.91/S3/Config.py
s3cmd-0.9.9.91/S3/Utils.py
s3cmd-0.9.9.91/S3/SortedDict.py
s3cmd-0.9.9.91/S3/__init__.py
s3cmd-0.9.9.91/S3/BidirMap.py
s3cmd-0.9.9.91/S3/CloudFront.py
s3cmd-0.9.9.91/INSTALL
s3cmd-0.9.9.91/s3cmd.1
s3cmd-0.9.9.91/setup.cfg
s3cmd-0.9.9.91/NEWS
s3cmd-0.9.9.91/PKG-INFO
s3cmd-0.9.9.91/setup.py
s3cmd-0.9.9.91/README
s3cmd-0.9.9.91/s3cmd

5. Change into the extract directory of s3cmd-0.9.9.91:

root@ubuntu:~# cd s3cmd-0.9.9.91

6. Install s3cmd utility on your Ubuntu VM:

root@ubuntu:~/s3cmd-0.9.9.91# python setup.py install

7. Verify that s3cmd utility is installed:

root@ubuntu:~# which s3cmd
/usr/local/bin/s3cmd

8. You'll now need copy over the required files to get s3cmd to work on your ESXi host

9. First you'll need to set an environmental variable to either the hostname/IP of your ESXi host:

root@ubuntu:~/s3cmd-0.9.9.91# export ESXI_HOSTNAME=172.30.0.183

10. Now you'll want to run the following commands which will upload the  required files to your ESXi host and you'll be prompted to end your ESXi  host credentials:

scp /usr/local/bin/s3cmd root@${ESXI_HOSTNAME}:/bin
scp /usr/bin/gpg root@${ESXI_HOSTNAME}:/bin
scp /usr/lib/python2.5/htmlentitydefs.py root@${ESXI_HOSTNAME}:/lib/python2.5-visor
scp /usr/lib/python2.5/hmac.py root@${ESXI_HOSTNAME}:/lib/python2.5-visor/
scp /usr/lib/python2.5/_strptime.py root@${ESXI_HOSTNAME}:/lib/python2.5-visor/
scp /usr/lib/python2.5/calendar.py root@${ESXI_HOSTNAME}:/lib/python2.5-visor/
scp /usr/lib/python2.5/__future__.py root@${ESXI_HOSTNAME}:/lib/python2.5-visor/
scp -r /usr/lib/python2.5/distutils/ root@${ESXI_HOSTNAME}:/lib/python2.5-visor
scp -r /usr/local/lib/python2.6/dist-packages/S3 root@${ESXI_HOSTNAME}:/lib/python2.5-visor

As you can see we'll be uploading s3cmd and gpg utility to /bin on the  ESXi host which is one of the default search paths and the remainder are  python packages required to run the s3cmd and we'll be storing that in  the default PYTHONPATH which includes /lib/python2.5-visor

11. Now login to your ESXi host and try to run s3cmd and you should see something like the following:

~ # ./s3cmd
ERROR: /.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

You're now ready to setup the s3cmd configuration by specifying --configure and with some luck you might be able to upload data including a VM into Amazon's S3!

Please provide any feedback for those who care to try that have S3  account, would be interesting to see if this would actually work and  also ensure you take the necessary measures in locking down your ESXi  host as it'll need access out into the wild wild west that we call the  Internet.

Install Amazon s3cmd utility on ESX 4.0

1. Download s3cmd-0.9.9.91.tar.gz and upload to your classic ESX host

2. Extract the contents of s3cmd-0.9.9.91.tar.gz:

[root@himalaya ~]# tar -zxvf s3cmd-0.9.9.91.tar.gz
s3cmd-0.9.9.91/
s3cmd-0.9.9.91/S3/
s3cmd-0.9.9.91/S3/PkgInfo.py
s3cmd-0.9.9.91/S3/S3.py
s3cmd-0.9.9.91/S3/S3Uri.py
s3cmd-0.9.9.91/S3/ACL.py
s3cmd-0.9.9.91/S3/Progress.py
s3cmd-0.9.9.91/S3/Exceptions.py
s3cmd-0.9.9.91/S3/SimpleDB.py
s3cmd-0.9.9.91/S3/Config.py
s3cmd-0.9.9.91/S3/Utils.py
s3cmd-0.9.9.91/S3/SortedDict.py
s3cmd-0.9.9.91/S3/__init__.py
s3cmd-0.9.9.91/S3/BidirMap.py
s3cmd-0.9.9.91/S3/CloudFront.py
s3cmd-0.9.9.91/INSTALL
s3cmd-0.9.9.91/s3cmd.1
s3cmd-0.9.9.91/setup.cfg
s3cmd-0.9.9.91/NEWS
s3cmd-0.9.9.91/PKG-INFO
s3cmd-0.9.9.91/setup.py
s3cmd-0.9.9.91/README
s3cmd-0.9.9.91/s3cmd

3. Change into the extract directory of s3cmd-0.9.9.91:

[root@himalaya ~]# cd s3cmd-0.9.9.91

6. Install s3cmd utility on your Ubuntu VM:

[root@himalaya s3cmd-0.9.9.91]# python setup.py install

7. Verify that s3cmd utility is installed:

[root@himalaya s3cmd-0.9.9.91]# s3cmd
ERROR: /root/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

Configure Amazon s3 utility to backup a VM to the cloud

1. Configure s3cmd with your S3 account:

~ # s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key  .......
.........

2. Let's say you have the following VM that you would like to backup

~ # ls /vmfs/volumes/esxi4-1-local-storage/veryImportantVM/
veryImportantVM-flat.vmdk  veryImportantVM.vmdk       veryImportantVM.vmsd       veryImportantVM.vmx        veryImportantVM.vmxf

3. You use your favorite VM backup solution ghettoVCB and you compress the backup:

~ # ls -la /vmfs/volumes/esxi4-1-local-storage/WILLIAM_BACKUPS/veryImportantVM/
drwxr-xr-x    1 root     root                420 Jan 10 17:56 .
drwxr-xr-x    1 root     root                420 Jan 10 17:55 ..
-rw-r--r--    1 root     root              26705 Jan 10 17:55 veryImportantVM-2010-01-10--1.gz

4. You'll first need to create s3 bucket

~ # s3cmd mb s3://primp-industries.com-ghettoBucket
Bucket 's3://primp-industries.com-ghettoBucket/' created

5. Verify the bucket creation by listing all buckets:

~ # s3cmd ls
2010-01-10 18:21  s3://primp-industries.com-ghettoBucket

6. Upload the file to s3 bucket:

~ # s3cmd put /vmfs/volumes/esxi4-1-local-storage/WILLIAM_BACKUPS/veryImportantVM/veryImportantVM-2010-01-10--1.gz s3://primp-industries.com-ghettoBucket
/vmfs/volumes/esxi4-1-local-storage/WILLIAM_BACKUPS/veryImportantVM/veryImportantVM-2010-01-10--1.gz -> s3://primp-industries.com-ghettoBucket/veryImportantVM-2010-01-10--1.gz  [1 of 1]
 26705 of 26705   100% in    0s    52.60 kB/s  done

5. To verify the upload, you can list the contents of your s3 bucket:

~ # s3cmd ls s3://primp-industries.com-ghettoBucket
2010-01-10 18:22     26705   s3://primp-industries.com-ghettoBucket/veryImportantVM-2010-01-10--1.gz

6. Let's say you've deleted your VM on your ESXi host and would like to download the VM you've uploaded to your s3 bucket:

/vmfs/volumes/4b22c223-9e9930ba-aa45-00505692697a/WILLIAM_BACKUPS/veryImportantVM # s3cmd get s3://primp-industries.com-ghettoBucket/veryImportantVM-2010-01-10--1.gz
s3://primp-industries.com-ghettoBucket/veryImportantVM-2010-01-10--1.gz -> ./veryImportantVM-2010-01-10--1.gz  [1 of 1]
 26705 of 26705   100% in    0s   143.93 kB/s  done

Known Issues

1. When configuring your s3 access credentials and you run into the following error when validating your passphrase using GPG:

Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
ERROR: Test failed: unknown encoding: string-escape

You may need to disable GPG either by specifying an invalid path to gpg or renaming /bin/gpg to /bin/gpg.tmp as a work around