VMware Cloud Community
lasinl
Enthusiast
Enthusiast

Backup to Amazon S3

I've got ESXi 3.5u4. Is it possible to backup the vmdks to Amazon S3?

0 Kudos
14 Replies
DSTAVERT
Immortal
Immortal

There isn't anything that would prevent you from doing it. You would need something to stage the transfer and I can imagine unless you have a very, very, very large Internet connection that it would take a while. You would need to use something to copy off the vmdks to a staging server since you wouldn't be able to install anything directly on ESXi (you also wouldn't want to install anything on ESXi for stability sake). Use a tool like ghettoVCB to clone the disks to an NFS server. There are several tools that will enable you to connect an Amazon S3 account as a disk and mount it as a normal drive. I use Jungledisk to store some things to amazon. Once you have your disks cloned to the NFS server you could just copy them to the mounted amazon drive.

-- David -- VMware Communities Moderator
0 Kudos
lasinl
Enthusiast
Enthusiast

would installing java cause ESXi to be unstable? I'm thinking of writing a shell script to stop the desired VM, push the vmdk to my S3 account with js3tream and start the VM again.

0 Kudos
lamw
Community Manager
Community Manager

You can't install any additional software on ESXi, it does not have a Service Console like classic ESX and even if it did, you would not want or could not install something as heavy as Java.

If you want to use export the image via Java, depending if you're using the licensed version or not, you could just hook into the VI Java API to export the image and then upload to Amazon S3. If you're using the free version, then your options of exporting the initial image is slightly limited and as mentioned, you can take a look at my 100% free scripted solution to export the image. If you have NFS datastore in which the backup can be dumped, then you can another system or VM that has Java installed and can communicate with the NFS datastore in which the backup lives and you can zip/tar up the image and upload.

Hopefully this clears things up, but you won't be installing any type of software on ESXi

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
lasinl
Enthusiast
Enthusiast

I was tip'd to your script a while ago, but I couldn't/can't use it because disk space is an issue (hence me wanting to stream it to S3).

What about installing s3cmd (yes, I have SSH access to the ESXi server)? Its not as hungry as java, its just a little proggie. Will a lil' binary ladybug make ESXi unstable?

0 Kudos
lamw
Community Manager
Community Manager

Well either way you'll still need some spare space for the snapshotted VM to grow while you backup the VM. Regarding your s3cmd utility, you can try but I highly doubt it'll work, the unsupported Busybox console is very limited and you should minimize any type of applications running within that shell.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
lamw
Community Manager
Community Manager

Take a look at this document:

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
lamw
Community Manager
Community Manager

Looks like the s3 account I had just signed up may take up to 72+hrs before it becomes active =/

I'm seeing Authorizing your account to access this web service under my Account Activity

~ # s3cmd mb s3://primp-industries.com-ghettoBucket
ERROR: S3 error: 403 (NotSignedUp): Your account is not signed up for the S3 service. You must sign up before you can use S3.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
DSTAVERT
Immortal
Immortal

I don't know that I would want to send off my vmdks to Amazon but the configs maybe not so bad. I'll give it a try.

-- David -- VMware Communities Moderator
0 Kudos
lamw
Community Manager
Community Manager

Agreed, but it was an interesting question for sure. You can always encrypt VMDKs before shipping them off, from the looks of it, it should work but again my account isn't quite 'activated' from Amazon.

If anyone else has an active an account and verify that would be great.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
lamw
Community Manager
Community Manager

Looks like Amazon s3 support got back to me and it can take up to 24hrs to verify your credit card information.

I've just verified uploading a dummy VM to s3 and it works! I had to update few additional python libraries and everything has been documented and updated here: How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud

Let me know if you run into any issues but this should definitely work on ESXi and should be even more straight forward on classic ESX but I've not had the time to verify yet.

Update: Also updated the document for classic ESX, it's much easier since you have all the required python libraries

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
DSTAVERT
Immortal
Immortal

William

Came down with something so no stomach for technology today but how big was your VM and how long did it take?

-- David -- VMware Communities Moderator
0 Kudos
lamw
Community Manager
Community Manager

It was a dummy VM that was < 1MB. I'm sure the speeds will vary based on connection type/location, I just verified that I could upload/download files. I'll leave it to the users to experiment with a full upload/download.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Comuunity

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
DSTAVERT
Immortal
Immortal

I was going to calculate what that would have added to the bill. At 15 cents per GB and 18 cents and so much per . . . . Can't be more than .001 cents.

-- David -- VMware Communities Moderator
0 Kudos
lasinl
Enthusiast
Enthusiast

Out friggin' standing! I love when can't/shouldn't/don't turns into 'oh snap it works!', especially when I'm the reason behind it (lasinl is dusting his shoulders off, because he feels like a pimp).

Anywho, there's one potential problem: I've read somewhere S3 doesn't support files over 5 GB Smiley Sad I'm sure the majority of people's VMDK files are over 5 GB. I haven't quite figured out how to get around that yet, though I believe using js3tream in combination with tar allows you to split files on the fly when uploading to S3 (or maybe that was a feature in s3napback, I forget). But that's a no-no because it requires JAVA to be installed...

0 Kudos