Best backup solution for VMware ESXi?

[Chinabee]

What would be the best backup solution for VMware ESXi? I do not

necessarily want a live replication but I want to have image level

backup for my vms (stored on NAS). In case of a disaster, I can recover

that from the image backup quickly.

I have about 60 vms.

Is Veeam the best choice? They seem to ask $500/socket, which will cause a substantial investment for me.

[marcelo_soares]

If you have your ESX licensed (with other license than the free one), you can use VMware Consolidated Backup, is simple and very useful. Also, here on communities you can find the GhettoVCB (http://communities.vmware.com/docs/DOC-8760) script that also can help you on backups for free.

Marcelo Soares

VMWare Certified Professional 310/410

Technical Support Engineer

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[AntonVZhbankov]

Veeam is the best solution without any doubts.

---

MCSA, MCTS, VCP, VMware vExpert '2009

http://blog.vadmin.ru

[Chinabee]

Veeam licenses by per CPU which gets really expensive after a few servers.

ghettoVCB looks awefully difficult to use.

[IanTech]

Hi,

You could use VMware Data Recovery. Easy backups of vm's and you are able to restore to file level.

Hope this helps.

[AndreTheGiant]

Have a look also at this document:

Andre

[cyberpaul]

Hi all,

I know this is an old thread, but the original question precisely describes some of my customers' environments.

After years of trial and error I started using VMcom (https://vmcom.com) and now I even contribute to it.

If you use a NAS share for your backups, it can store image-level backups in native format (you can see .vmdk and .vmx files on your NAS share). If something goes really bad with your primary storage, you can even attach the NAS to vSphere and run your VMs directly from the backup storage. This is obviously the worst case scenario, but it is nice to have this option.

Furthermore, VMcom can mount those .vmdk files and perform file-level recovery from both windows and linux filesystems.

[IvarHome]

Veeam is not good solution. It is unable to produce application consistent backups. Best solution is to use hardware storage SAN (iSCSI LUN-s) and use SAN integrated snapshot replication.

[daphnissov]

IvarHome​

This is categorically false information. Veeam is most certainly capable of taking application-consistent backups. In fact, it's far, far better at doing so than any other application because they wrote their own VSS provider as opposed to using the buggy VMware Tools provider. You can configure this in a job by turning on application aware image processing.

And in cases where they don't have a provider for a given application, they can execute custom scripts before and after the job to quiesce the given application.

[IvarHome]

You are wrong. VSS is not able to take application consistent backups. What VSS can do? Do you know what VSS does at all? It flushes disks, takes point-in-time snapshot with open-files read-write with VSS writer,  quiescenes windows operating system databases, MS SQL and Exchange. And thats all. But I dont have Exchange or SQL. I have instead some brand server software there. Its even Linux based embedded VM, all-in-one or on-the-box. It donkt know nothing about VSS. It even dont know nothing about VMWare tools - there is no vmware tools at all. Or the tools are outdated and there is no any possibility to insert them, because all stuff in on-the-box and any external action is strongly prohibited. But VSS it dont have anyway, because its not windows. The only possibility to backup it is to take vmware snapshot together with VM memory dump.....Do you know at all that VSS is microsoft windows technology only. Also, some servers are BSD or pure Unix based. VMWare dont have "tools" for them, and if they have, then server is enough protected and dont allow any installation, vmware even dont know where to install and what, because OS is unknown.

.

[daphnissov]

You are wrong. VSS is not able to take application consistent backups. What VSS can do? Do you know what VSS does at all? It flushes disks, takes point-in-time snapshot with open-files read-write with VSS writer,  quiescenes windows operating system databases, MS SQL and Exchange.

Yes, which is why you are wrong, and you've written it here. Quiescence tells applications to write in-memory changes down to disk prior to a snapshot taking place. VSS is a Microsoft technology that enables this on Windows, but there is a sync driver for Linux that does the same thing. It works the same way and clearly you need to go read about it.

I have instead some brand server software there. Its even Linux based embedded VM, all-in-one or on-the-box.

Well, that's a problem you must take up with the vendor of this software and what they recommend as far as backup procedure. Contact them and ask what their recommendations are for backing up. They may offer pre-freeze and post-thaw scripts used for backing up.

It even dont know nothing about VMWare tools - there is no vmware tools at all. Or the tools are outdated and there is no any possibility to insert them, because all stuff in on-the-box and any external action is strongly prohibited. But VSS it dont have anyway, because its not windows.

Again, that's a problem with whatever this software is and the vendor who provides it. You're conflating how quiescence works with not being able to quiesce on this application. Just because this application doesn't seem to have it out-of-the box doesn't mean it doesn't work for other applications.

[IvarHome]

Yes, which is why you are wrong, and you've written it here. Quiescence tells applications to write in-memory changes down to disk prior to a snapshot taking place. VSS is a Microsoft technology that enables this on Windows, but there is a sync driver for Linux that does the same thing. It works the same way and clearly you need to go read about it.

No, quiescence dont tell nothing if vmware tools are not installed and OS is unknown. Hypervisor itself is not able to flust guest memory to guest disk, outside VM. Servers are not only Linux based, they can be also BSD based and VMWare dont know nothing about them. Also, flushing disks is not application consistency backup, its crash consistency. Databases dont hold just disk buffers in memory. They hold databases logs and only this database itself know what to write to where.

Well, that's a problem you must take up with the vendor of this software and what they recommend as far as backup procedure. Contact them and ask what their recommendations are for backing up. They may offer pre-freeze and post-thaw scripts used for backing up.

No, I dont. They dont recommend nothing, I dont have time for this etc. You cant use those pre-post scripts, because you cant take server offline. And usually brand dont give you more specific commands to quiescene. And finally I dont need this crap at all - I can just backup whole VM together with memory dump file and together with snapshots, through iSCSI LUN and thats more faster and easy way to do.

Just because this application doesn't seem to have it out-of-the box doesn't mean it doesn't work for other applications.

Yes, VM embedded servers do only stuff that they declared to do. Do you think I have time to let them make some special version only for me. Today every brand make its own out-the-box stuff. I have many different brands, all embedded VM-s. They are "embedded" just because its easy to use. Without any mails exchange with support. Just read documentation and thats all. Like it - dont like it, take next brand box.

[daphnissov]

You clearly don't understand how quiescence is supposed to work and seem to think it's only a VMware or hypervisor phenemon. It isn't. It's been around for literally years even before virtualization began.

Also, flushing disks is not application consistency backup, its crash consistency. Databases dont hold just disk buffers in memory. They hold databases logs and only this database itself know what to write to where.

You need to read about VSS basic concepts. Quiescence is useless unless there are writers, and this is exactly what database vendors provide. The command to quiesce doesn't just flush files to disk, it informs these applications that provide a writer that a backup is about to happen. The application then handles the logic to write out any outstanding in-memory I/Os to disk. This writer and requestor structure is the whole reason why quiescence does provide application consistency and not crash consistency.

No, I dont. They dont recommend nothing, I dont have time for this etc. You cant use those pre-post scripts, because you cant take server offline. And usually brand dont give you more specific commands to quiescene. And finally I dont need this crap at all - I can just backup whole VM together with memory dump file and together with snapshots, through iSCSI LUN and thats more faster and easy way to do.

So, again, that isn't the fault of the principle of quiescence, that's the fault of whoever this vendor is. If they don't recommend anything then either A.) they don't need their application/system quiesced and crash-consistency is good enough or B.) they're a crap company and don't care about supporting their users in operation. Either one you pick, this doesn't negate the usefulness or operation of quiescence.

Yes, VM embedded servers do only stuff that they declared to do. Do you think I have time to let them make some special version only for me. Today every brand make its own out-the-box stuff. I have many different brands, all embedded VM-s. They are "embedded" just because its easy to use. Without any mails exchange with support. Just read documentation and thats all. Like it - dont like it, take next brand box.

I don't even know what you're talking about here. The brand of physical server makes no difference, it's the operating system and application which determine how data protection must be managed. You don't ask HPE how to take application-consistent backups of Oracle running on a DL380, you ask Oracle how to do that. The physical platform makes no difference.

I recommend you spend some time on Google reading about how quiescence really works and stop making wildly false claims based on incorrect understanding of these principles.

[IvarHome]

You clearly don't understand how quiescence is supposed to work and seem to think it's only a VMware or hypervisor phenemon. It isn't. It's been around for literally years even before virtualization began.

Well, tell me then.

You need to read about VSS basic concepts. Quiescence is useless unless there are writers, and this is exactly what database vendors provide. The command to quiesce doesn't just flush files to disk, it informs these applications that provide a writer that a backup is about to happen. The application then handles the logic to write out any outstanding in-memory I/Os to disk. This writer and requestor structure is the whole reason why quiescence does provide application consistency and not crash consistency.

I tell once more - VSS is windows technology and I dont have windows there in VM. And even when I have windows, VSS cant do nothing with 3rd party servers or databases.

No, database vendors dont provide VSS compatibility. Writers for VSS can made only for windows system. Not for Linux, not for BSD, not for UNIX, only for windows. Secondly, there is no "database vendor" for server software. Usually servers and 3rd party software use their own made unstandard databases. They even dont call them databases, its just software logic, but actually its still database. Thats because they dont have database agents, because its not actually database. The only suggest server brand can give, is to shut server down and make backup. Of course its unacceptable. But I dont need this writer for quiescene. I can just make VM snapshot with memory. Then make storage SAN snapshot for that LUN. Then make storage SAN snapshot replication to another SAN or to different storage pool in the same SAN. This is very fast operation, integrated into storage hardware, uses CBT (changed block tracking). Then I can delete storage SAN snapshot of LUN. Then I can delete vmware VM snapshot. Also, hardware storage SAN have also automation for all this. It have sheduling and also vmware compatible snapshot agent. And I dont need any writer at all.

So, again, that isn't the fault of the principle of quiescence, that's the fault of whoever this vendor is. If they don't recommend anything then either A.) they don't need their application/system quiesced and crash-consistency is good enough or B.) they're a crap company and don't care about supporting their users in operation. Either one you pick, this doesn't negate the usefulness or operation of quiescence.

Yes it is. You dont need to do quiescene through guest writer. You dont need to stop databases, dont need to prepare databases for backup. Its all useless and not possible in most cases.  You can instead backup whole VM together as it is in real time, nothing changing, together with memory. And when you restore, you can get server working and online exactly in the same point. Just first revert SAN snapshot, then revert vmware VM snapshot and your server is in the same time moment.

I don't even know what you're talking about here. The brand of physical server makes no difference, it's the operating system and application which determine how data protection must be managed. You don't ask HPE how to take application-consistent backups of Oracle running on a DL380, you ask Oracle how to do that. The physical platform makes no difference.

I recommend you spend some time on Google reading about how quiescence really works and stop making wildly false claims based on incorrect understanding of these principles.

So, you dont know what is embedded VM? In OVF or OVA format? They are not physical, they are virtual. And yes, any influence to its operating system is prohibited. If you talk to support that you insert something into operating system, then support first request you to delete whole VM and install pure plain one. Because this kind of action is not supported and allowed. Its hacking and against of license. There is no Oracle, most server databases are unique, only made for this software, there is no standard. Only external databases are standards.

[daphnissov]

I tell once more - VSS is windows technology and I dont have windows there in VM. And even when I have windows, VSS cant do nothing with 3rd party servers or databases.

Yes, VSS is a Microsoft technology, but it's only one implementation of quiescence. There are others as I've stated. Also, what do mean by by "3rd party servers or databases"? Databases made by Microsoft? Because that's also not true. Any software vendor can write a writer that responds to VSS commands. The fact that it isn't sold by Microsoft makes no difference.

I can just make VM snapshot with memory. Then make storage SAN snapshot for that LUN. Then make storage SAN snapshot replication to another SAN or to different storage pool in the same SAN. This is very fast operation, integrated into storage hardware, uses CBT (changed block tracking). Then I can delete storage SAN snapshot of LUN. Then I can delete vmware VM snapshot. Also, hardware storage SAN have also automation for all this. It have sheduling and also vmware compatible snapshot agent. And I dont need any writer at all.

Fine, if that works for you because you're using Linux and your software vendor is crap, then great. Glad for you. But have you ever tried restoring a VM to a powered-on state and that VM runs some sort of database application? If not, let me just tell you that most of the time when you restore to this point, you have to reboot the VM anyway because it's unstable. But, as I said, if you are using some obscure software made by a vendor that won't provide proper backup methodologies including quiescing scripts, then more power to you. I wish you luck.

So, you dont know what is embedded VM? In OVF or OVA format? They are not physical, they are virtual. And yes, any influence to its operating system is prohibited. If you talk to support that you insert something into operating system, then support first request you to delete whole VM and install pure plain one. Because this kind of action is not supported and allowed. Its hacking and against of license. There is no Oracle, most server databases are unique, only made for this software, there is no standard. Only external databases are standards.

Once again, this is a problem you're having with whatever software or system vendor you're dealing with who refuse to provide any safe and reliable backup methodologies. You're blaming other vendors for not supporting a method to quiesce these types of applications/systems when the blame is on the vendor who sold it to you. You need to go to them and complain rather than to all other backup application vendors and complain. It's not their responsibility to support obscure, poorly-written apps or operating systems--especially pre-built and packaged virtual appliances.

[IvarHome]

There are others as I've stated. Also, what do mean by by "3rd party servers or databases"? Databases made by Microsoft? Because that's also not true. Any software vendor can write a writer that responds to VSS commands. The fact that it isn't sold by Microsoft makes no differenc

No, not by microsoft and not by any database vendor. And not all brands make writers for their servers, not at all. Examples Ahsay embedded backup server, based on BSD. I have no idea what kind of database it is. Ist Ahsay own made database. Example Palo Alto embedded VM. Check Point embedded VM operating system is GAIA. I havent hear abot any writer existence. And dont expect there isnt databases at all. Databases exist almost in every software. Even home software program can have database and you cant get information what database it is. Support can tell you - Some John Smith, or programmer write this application or server, who the hell know what kind of database this is. John itself writed it, but no writer. And with support you can exchange mails couple of months and still get nothing. Until then still no backups?

Fine, if that works for you because you're using Linux and your software vendor is crap, then great. Glad for you. But have you ever tried restoring a VM to a powered-on state and that VM runs some sort of database application? If not, let me just tell you that most of the time when you restore to this point, you have to reboot the VM anyway because it's unstable. But, as I said, if you are using some obscure software made by a vendor that won't provide proper backup methodologies including quiescing scripts, then more power to you. I wish you luck.

Of course I have done this many times successfully. Yes databases also, but they are databases in the same VM. All is restored. No, its not unstable. I had no any problems, only network sometimes dont start, but its easy to fix. No any reboot wasnt necessary at all......But not only this. Backing up through LUN, through LUN snapshot replication is many times faster than Veeam backup. Storage SAN snapshot replication is hardware specific, very fast. Its surely preferable method than using some backup software with quiescing through writers. Also when you backup in this way, you must after restore boot server. With LUN related backup you dont need boot server.

Once again, this is a problem you're having with whatever software or system vendor you're dealing with who refuse to provide any safe and reliable backup methodologies. You're blaming other vendors for not supporting a method to quiesce these types of applications/systems when the blame is on the vendor who sold it to you. You need to go to them and complain rather than to all other backup application vendors and complain. It's not their responsibility to support obscure, poorly-written apps or operating systems--especially pre-built and packaged virtual appliances.

No, its not problem. Most software vendors dont make writers. Also writers are not stable solution. Because I can change ESXi versions, vCenter versions and also server version of this vendor. All this can stop its compatibility.....And what you mean "poorly written apps". Today, the trend is to use embedded VM-s as servers. They are OVA or OVF files. There is all stuff - operating system, server, apps, configuration, all. You dont need to do anything more than run this VM and start configuring its through web. I have no any interest what operating system or writer it have. I read documentation. When in documetation is written about writer, then there is writer, when not, then I dont care more. Its normal practice. And why I must additionally write some mail to support? Why? I have nothing better to do?

[IvarHome]

Whole Vmware API and VDDK for backups is wrongly implemented. Without VM memory dump file, there is no any application consistent backups. Therefore backup software vendors cant develope normal backup software. Of course backup software can still get all .vmdk, snapshot memory .vmem, snapshot metadata .vmsd files. But then backup software lost CBT (changed block tracking), or backup software must itself read .ctk files and understand vmware standard (for generating deltas). But anyway through SSH it first must download whole file first and this is problem for performance........And second non understandable strategy from Vmware was Data Protection product developing end. Do Vmware want to say Veeam is better. Yes, it have some unique temporary guest client installation feature and able to backup PCI passthrough VM-s, but it still Windows based VM and not embedded OVA VM as Data Protection was. Data Protection was good product, configurable directly from vCenter. Fortunately Vmware Replication is still in production. Vmware Replication is real power, much better than Veeam. It dont need at all any temporary snapshot, backup is very dynamic, uses CBT, uses rotation, incrementals, dont waste resources and is also compactly compressed. Best security is to use both Vmware Replication for everyday backup and hardware storage SAN for less often backups.

[daphnissov]

These are the last points I'm going to make in this thread. I hope you read them carefully:

• Quiescing in ESXi/vSphere is available for both Windows and *nix systems. Windows use the VSS system while *nix uses the vmsync driver.
• Some databases do not need to be quiesced to have consistent backups taken with regular vSphere snapshots. One example is vPostgres where everything is written to a redo log first before being committed to the database. This is why the vCenter Server Appliance, for example, does not need to be quiesced to be successfully backed up.
• Any vendor can write support for quiescing in their app. While VMware provides the VSS writer and vmsync driver, the vendor is responsible for hooking into this system and then implementing whatever is necessary to put their product into a consistent state. Thus, VMware provides the framework necessary to take backups of any system running on top of it, but the responsibility is on the vendor building said system in order to use it.
• If the appliance that you have chosen to deploy on vSphere does not offer a snapshot-based (VADP) method to protect the solution, this is not VMware's fault. You need to speak to the vendor of that appliance. Almost all vendors have some method of protecting their appliances. Sometimes that is through an in-guest method; sometimes that is with built-in scripts; and sometimes that is based on VADP. You complaining about Veeam or VMware's software being poor because your Palo Alto or Checkpoint appliance cannot be backed up with snapshots in a consistent state is absurd. It would be like me complaining to a gasoline company that my Subaru Forester doesn't have power steering.
• If you've found that capturing the memory state of a VM appliance during snapshot, followed by snapshotting the LUN is the only way to achieve a consistent backup on vSphere, then one or more things below apply to you and I don't much care which it is:
  • You haven't read the vendor's documentation or know very much about this appliance. I have absolutely never once heard in all my experience that this is the only or required method to produce a consistent backup of any type of appliance. If you're claiming this is so, provide evidence of it or at least the mythical appliance name in question.
  • You haven't actually tested a restore of your appliance using simple snapshots, or something unusual happened in case that test restore failed to lead you to believe that it won't work.
  • This appliance is not VMware certified
  • This appliance is a piece of garbage

          or lastly

•          You'll have to make do with your backup hack system
• Replication is not backup. Read that very carefully. Replication is not backup. You comparing LUN replication and vSphere Replication with other backup applications shows you don't understand the difference between these two technologies.

[IvarHome]

Quiescing in ESXi/vSphere is available for both Windows and *nix systems. Windows use the VSS system while *nix uses the vmsync driver.

This is not true. I have some BSD VM-s without any possibility to install even vmwate tools. The message from vCenter is something "...vmware tools are not managed by vmware"

And knowledge to know BSD OS is not my business also.

Some databases do not need to be quiesced to have consistent backups taken with regular vSphere snapshots. One example is vPostgres where everything is written to a redo log first before being committed to the database. This is why the vCenter Server Appliance, for example, does not need to be quiesced to be successfully backed up.

Of course, but do vmware know which database need it? Nobady wasnt heared about them before - they are server made vendor own made databases. And when you ask from support, they gallantly say "this information is confidential". And thats it. There is no any vPostgres.

Any vendor can write support for quiescing in their app. While VMware provides the VSS writer and vmsync driver, the vendor is responsible for hooking into this system and then implementing whatever is necessary to put their product into a consistent state. Thus, VMware provides the framework necessary to take backups of any system running on top of it, but the responsibility is on the vendor building said system in order to use it.

Of course they can, but they just dont do it. Im not so important person for them. Im not from General Motors. They are not responsible for anything.

If the appliance that you have chosen to deploy on vSphere does not offer a snapshot-based (VADP) method to protect the solution, this is not VMware's fault. You need to speak to the vendor of that appliance. Almost all vendors have some method of protecting their appliances. Sometimes that is through an in-guest method; sometimes that is with built-in scripts; and sometimes that is based on VADP. You complaining about Veeam or VMware's software being poor because your Palo Alto or Checkpoint appliance cannot be backed up with snapshots in a consistent state is absurd. It would be like me complaining to a gasoline company that my Subaru Forester doesn't have power steering.

I dont care whos fault it is, I just want to backup, thats it.  Im not so important client, they just answer something "shut down server and make backup", they dont care at all. Even more, regular support dont know at all nothing about and if you want to talk with some engineer, then you must wait months. And this "almost all verndor..." is lie.

You haven't read the vendor's documentation or know very much about this appliance. I have absolutely never once heard in all my experience that this is the only or required method to produce a consistent backup of any type of appliance. If you're claiming this is so, provide evidence of it or at least the mythical appliance name in question.

Well, you just have not readed some SAN storage manual. SAN vendors vice-versa suggest backing up VM-s exactly only through LUN-s and through Vmware Snapshot Agent.

You haven't actually tested a restore of your appliance using simple snapshots, or something unusual happened in case that test restore failed to lead you to believe that it won't work.

No, I have tested it and all works perfectly.

This appliance is not VMware certified

This is not my problem. For me its enough when its made for ESXi.

This appliance is a piece of garbage

Maybe Veeam is garbage instead?

You'll have to make do with your backup hack system

Storage SAN is not hack. Just bought some SAN and make normal backups. Instead Veeam is garbage. Its windows based hack. Today its really shame when some vendor dont have on-the-box install-and-use completed and embedded VM with OVA file. I prefer BSD, because BSD takes the least resources. Windows servers are still ok, but not for one simple operation as backup, only for backup.

Replication is not backup. Read that very carefully. Replication is not backup. You comparing LUN replication and vSphere Replication with other backup applications shows you don't understand the difference between these two technologies.

Now I see - you just dont have any experience with Vmware Replication and with LUN snapshot replication. I know very well what is difference between backup and replication. Im used Veritas Volume Manager, Veritas Storage Foundation and Veritas Volume Replicator. They are sector based replications. Also I have used file-based replication software like Retrospect Replicator, Attunity RepliWeb,  PeerSync, GoodSync etc. I have written about this also reviews. The difference is huge between them......But now I tell one strange detail. Both Vmware Replication and LUN snapshot replications arent actually replications. Vmware Replication dont replicate anything, it just makes incremental backups and those backups are even compressed and useless to use without decompressing. They have own files structure. Only after decompressing you see usual VM-s. I dont know why Vmware call it Replication, but its not. Maybe because its so dynamic and "silently always run", its very perfect design, it even have some artificial intelligence for backup strategy and right moment. So, it looks like replication, but actually its not. The same thing is with LUN snapshot replications. There is no any actual replication. It just backing up incrementally all LUN snapshots. And also, you cant directly connect those LUN snapshot replications to LUN target. First you must copy those "replicas" into the same storage SAN and after that you can connect them with iSCSI LUN targets. So, its just name "replication", not the one technologically replication means. They are backups.