Definitive answer to backing up live VM's

[prisoner881]

This question has been asked before in different forms but I've yet to see a definitive answer to this thorny subject: is there a free method to backing up live VM's hosted on ESXi that doesn't have a huge downside? And if there isn't a free option, what's the cheapest commercial option? And what is the best option if price isn't an overriding concern?

What we have now is one WIndows 2003 R2 x64 server running VMWare Server 2.0. This box hosts ten VM's. We have Seagate Backup Exec 12.5. We're able to get live backups of running VM's using the VMWare VSS Writer service. Obviously we have no such option under ESX, yet we'd like to move to ESXi if possible for better performance.

According to Seagate's product literature, "the new release of Backup Exec 12.5...will allow users to back up an unlimited number of guest machines within a VMware ESX or Microsoft Hyper-V host environment to disk or tape from a single agent." There is no mention of ESXi, but I'm assuming the great similarity between ESX and ESXi might mean this is still supported. Has anyone tried it? What all is involved?

Last, others have suggested that we just continue to back up our guest OS's the same way we did when they were physical boxes. While this would work it defeats a lot of the convenience of disaster recovery. Backing up the guest OS means a lot of work to restore (reinstall OS, reinstall apps, restore data, etc.) when we could just restore the entire VM image and get the same -- but faster and easier -- results. I'd like to avoid the former backup situation in favor of the latter one.

[khughes]

Right now there aren't that many utilities to backup VM's on an ESXi host. This is due to the lack of service console which VMware took out for security reasons. Unfortunatly just about all of the vendors that provide backup options used the service console for their backups. All of them are in the process of developing new ways to backup VMs on ESXi, but it will most likely be clsoer to the end of the year, early Q1 of 2009 before we see them out on the market. There is a thread where people have been posting the ways they're getting around it here:

http://communities.vmware.com/message/1063420#1063420

++ I wouldn't count on BackupExec working on ESXi unless it directly says works for ESXi. Something to ask Symantec for sure.

• Kyle

[RParker]

>According to Seagate's product literature, "the new release of Backup Exec 12.5...will allow users to back up an unlimited number of guest machines within a VMware ESX or Microsoft Hyper-V host > environment to disk or tape from a single agent. That's because they treat them as physical machines, and not VM's. VCB will work, but it's not part of a 3rd party solution, look at the documentation for setting up VCB, that should work for ESX 3i.

[prisoner881]

<span class="jive-thread-reply-body-container">&gt;According to Seagate's product literature, "the new release of Backup Exec 12.5...will allow users to back up an unlimited number of guest machines within a VMware ESX or Microsoft Hyper-V host > > environment to disk or tape from a single agent. > > > > > > > > > > That's because they treat them as physical machines, and not VM's. VCB will work, but it's not part of a 3rd party solution, look at the documentation for setting up VCB, that should work for ESX 3i. > > > >

I'm not so sure about that. The documentation on BE 12.5 seems to hint rather strongly that it's backing up VM's, not the guest OS and apps. It mentions this in the same paragraph it talks about backing up live VM's (under Hyper-V) using VSS. Admittedly there is no firm evidence either way so only a real test will say. The Symantec rep was completely clueless when I asked. She didn't even understand what a VM was. Pathetic.

[Dave_Mishchenko]

With ESX regular you can install a BE agent in the service console and backup VMs via the host (just as you would with Hyper-V). ESXi lacks the Linux SC that ESX has, thus you can't install agent software on ESXi. http://www.vmware.com/pdf/vi35_backup_guide.pdf

[prisoner881]

Keeping in mind that I've never used ESX before, is the service console the same thing as SSH access? There are several (unsupported) guides to adding SSH functionality back to ESXi. I'm not concerned about support as this is already running on "white box" hardware.

I get the dismal feeling, however, that if it were that easy people would be doing this already.

[Dave_Mishchenko]

Not really. With ESX the service console is a special VM running Linux. With ESXi they have modified Busybox to be able to access the vmkernel but it has no Linux VM you connect to. With ESX you SSH to the VM, with ESXi you SSH to the dropbear SSH server which has been compiled to run on the vmkernel. You then use Busybox to run the Linux like commands.

[ericsl]

prisoner881,

You should check out Veeam Backup 2.0. It is capable of backing up ESXi machines and even include MS VSS functionality. However, isn't there always one, you can't restore to ESXi, only ESX. But from there you can VMmotion to an ESXi host.

For what it's worth.

Eric

[prisoner881]

Not really. With ESX the service console is a special VM running Linux. With ESXi they have modified Busybox to be able to access the vmkernel but it has no Linux VM you connect to. With ESX you SSH to the VM, with ESXi you SSH to the dropbear SSH server which has been compiled to run on the vmkernel. You then use Busybox to run the Linux like commands.

And I suppose there's no way to add this special VM back in, even in an otherwise-unsupported manner? Again, I'm not really concerned about supported vs. non-supported as the hardware isn't on the HCL.

[prisoner881]

prisoner881,

You should check out Veeam Backup 2.0. It is capable of backing up ESXi machines and even include MS VSS functionality. However, isn't there always one, you can't restore to ESXi, only ESX. But from there you can VMmotion to an ESXi host.

For what it's worth.

Eric

I've been to the website for this product and, as would be expected, the product looks good. No pricing, of course. What does this normally run? I've also look at vRanger, but again there's no pricing to be found anywhere.

Like a fancy restaurant, if the prices aren't on the website, it usually means it's pretty darn expensive.

[prisoner881]

I've also read of people enabling SSH on ESXi, then using scp (there's references to a Perl script of some sort sometimes) to copy the VM files to another box. This would actually be ideal for me. The idea is we've got two almost identical servers, one production, one backup, both with direct-attached storage. The production box should do a disk-to-disk copy of the VM's to the backup box via Gigabit Ethernet every night - preferably without taking them down. The backup box will then back those up to tape, and tapes taken offsite regularly.

If I could figure this out it would give us very good disaster recovery. If the production box dies, we fire up the backup box and go with last night's D2D backups. If there's a fire or other disaster in the server room, we get new hardware, restore from tape, and we're good to go. Not as good as VMotion or HA, but then again it's fee.

[s1xth]

Just to throw my two cents in, I have been using Acronis to image all of my VM's, since its technically a physical machine you can restore that image to another phyical machine if you want too, or to another VM via Converter (free) or with Acronis which can also convert to ESX. (although i have had better luck with converter doing this method).

Although I am also still waiting for a true D2D backup from an ESXi to another ESXi server, or some sort of replication program for us free users.

[Guy_Mortlock]

And to add my two cents worth, I have also been looking for a simple, cheap (not necessarily free) backup solution for ESXi that doesn't involve using the unsupported features of ESXi.

I think that the answer will involve VIMA (which has only been released in the last couple of days) but I don't have the Linux skills to convert the scripts etc that have been posted in this newsgroup that rely on using the unsupported features of ESXi to run under the supported method of using VIMA.

The following is a quote from the VIMA community

"the VMware Infrastructure Management Assistant (VIMA), a virtual machine that allows developers and administrators to run agents and scripts to manage ESX and ESXi systems (version 3.5 and later). VIMA can be used to perform many of the tasks commonly performed in the ESX service console."

Guy

[ericsl]

For a production environment I would seriously consider dropping the few dollars for VI3. Then you can easily replicate the native machines to another ESX server. Veeams Backup does it with incremental forever technology so the backups are very small. Plus you can start a machine in the backup ESX server instantly, plus you can extract individual files, etc... There's too many minus' with ESXi for a production environment....

Eric

[prisoner881]

For a production environment I would seriously consider dropping the few dollars for VI3. Then you can easily replicate the native machines to another ESX server. Veeams Backup does it with incremental forever technology so the backups are very small. Plus you can start a machine in the backup ESX server instantly, plus you can extract individual files, etc... There's too many minus' with ESXi for a production environment....

Eric

VI3 is hardly a "few dollars." We've estimated that in order to get two physical boxes (both on the HCL), a SAN, and all the licenses needed for replication and failover between the two (VMotion, etc.) would run us north of $40K. For a company of 35 users this is hardly practical...which is why we're running VMWare Server 2.0 now and only investigating ESXi.

VMWare positions both VMWare Server and ESXi as "entry level" virtualization for small businesses. This is exactly what we are. Small businesses do not spend $40K on IT server licenses annually, but we still need robust backup solutions. What's ironic is my backup options are both numerous and functional on VMWare Server 2.0, but very limited on ESXi -- the product VMWare is positioning as the "gateway" to full ESX. It's not looking like much of a gateway to me right now.

[ericsl]

prisoner881,

We run Intel S5000V series motherboards, single Quad Core, with either LSI or Adaptec RAID cards, 16GB RAM, 6TB SATA for around $4000-$5000. VI 3 Foundation is $995, Veeam Backup is $495. Total: $13,000

IMHO you don't need a SAN, just storage on the primary and secondary ESX box to run your machines. Then, with Veeam Backup, you replicate/backup incrementally to the secondary ESX server and (this is how we do it) replicate to an off-site ESX server (service) for full DR capability. It's not VMotion but you can recover very quickly from a primary server failure. Plus you don't need a SAN that then needs to be duplicated, especially in your environment.

Eric

myManagedBackup.com

[kpc]

Hi Guy

From what I can tell the VIMA appliance is merely a more polished version of the existing linux one, the Remote CLI. I just built my own Linux management server and run my own scripts, much easier to work with but you do need some skills with Linux.

[prisoner881]

prisoner881,

We run Intel S5000V series motherboards, single Quad Core, with either LSI or Adaptec RAID cards, 16GB RAM, 6TB SATA for around $4000-$5000. VI 3 Foundation is $995, Veeam Backup is $495. Total: $13,000

IMHO you don't need a SAN, just storage on the primary and secondary ESX box to run your machines. Then, with Veeam Backup, you replicate/backup incrementally to the secondary ESX server and (this is how we do it) replicate to an off-site ESX server (service) for full DR capability. It's not VMotion but you can recover very quickly from a primary server failure. Plus you don't need a SAN that then needs to be duplicated, especially in your environment.

Eric

myManagedBackup.com

Our server hardware is considerably beefier. The current white boxes are 2x quad core 2.5GHz AMD Opteron, 32GB RAM, 2TB of direct attached storage. We'd be looking at $10K-15K each for similar hardware on the HCL in order to use ESX, so that's $20K-$30K just to start with. Even omitting the SAN, that's a lot of cash to lay out just to get some sort of real-time backup. The white box stuff only ran about $6500 each and they're bought and paid for. There's just no way to make a case to get new hardware just so we can run the full ESX. Performance-wise we've got plenty of headroom on VMWare Server 2.0.

I've read about a couple of people enabling the SSH console for ESXi then using something like rsync to keep things synchronized between two boxes. Has anyone tried this? How did you do it? And is it performing like you want?

[lmhealthcare]

The new VM option is an add-on agent with BackupExec 12.5.

Costs money. Requires VCB, so if you have VCB, you can use it.

If you have VCB though, you can still backup your machines with Backup Exec without the agent. The way I do it is connect to a snapshot and do a full backup Friday, and incrementals M-Th. I also use the FullVM option, and backup the image on Saturdays. With the new agent you can access the files in the image, which I can't do now. The drawback, and the reason I won't buy it, is you can't do an incremental. Every backup is a full backup. I don't have the time or space to store two weeks of backups if every one of them is a full backup.

The link to the manual is helpful,

[Dave_Mishchenko]

Just a couple of things to note:

1) Veeam Backup 2.0 does support ESXi, but it does use VCB which is not licensed as part of ESXi free. VCB with ESXi requires a VI foundation license. Now VCB still appears to work with ESXi free, but I wouldn't count on that for the long term.

2) Likewise with the RCLI and any thing like the appliance mentioned earlier that uses the RCLI or the VI Toolkit, both those items are meant to be read-only with ESXi free. Now the RCLI doesn't seem restricted from my tests, but the documentation says it should be. I brought it up with the ESXi product manager last week and there is the possibility that parts of the RCLI that deal with VMs (i.e. to start / stop / create snapshots) could be read/write for ESXi free. However, at this point I would not rely on the RCLI as the retriction to be read only could be introduced in the future. If you (and I mean you as in anyone reading this) would like to see the RCLI to have less restrictions with ESXi free then it would be good to add a post here explaining what you would like to do with ESXi free and the RCLI - http://communities.vmware.com/community/vmtn/suggest/product.

3) The console is intended for support only and while I'm one of the first to show people how to access it, I don't recommend it for long term / production use. The intent with the RCLI was that it would be a back door for support as you could not do everything with the RCLI or other tools. But in general long term access to the console is not gauranteed. Also, a mistake in the console could toast your install.

4) ESXi is not based on Linux and rsync is not available in the console.