Backup AppStack vmdk files

[VArjan]

Well simple question, how can you backup the AppStacks?

The AppStacks are stored on a VMFS volume and not attached to a VM which is part of the backup.

I am kind of supprised this topic is not part of the App Volumes documentation.

[Ray_handels]

Hey VArjan,

We have the exact same issue/question. We now have 2 SAN's and created 1 storagegroup in Appvolumes, this way Appstacks are replicated between SAN's. The only application we could find that was able to backup VMDK's was VEEAM. Issue with this is that it makes the file thick provisioned which is a shame for space because Appstacks are 20GB large in template and most of them are not more than 1 GB.

But the issue gets even bigger if you look at writable volumes. Because these are read/write you cannot sync it between SAN's. VEEAM has the exact same issue as with appstacks, it bloated them to thick provisoned and copies those files to a disk or tape or whatever. Issue with this is that the file should not be in use when backing up (otherwise it is locked) and if you have 200 writable volume each 20GB each (we created larger writable volumes) you need quite the space to facilitate this.

I have no idea if VMWare/Appvolumes will find a good soluiton for that. We are now looking at this issue with our storage vendor being Tintri. They have a SyncVM functionality. We are trying to get them to build a SyncVMDK or SyncDSFolder option. If they are able to create something like that it would be very nice. They even spoke about snapshotting the VMDK. If so it would be the goose with the golden eggs.

[Jason_Marshall]

‌The main reason is that because they are read only, corruption is not an issue so backing them up is as simple as just copping them to another datastore.

ALso what raymond described is also a good way to back them up because they will automatically be replicated to the other datastores. With 2.10 an additional feature will be added to storage groups that will allow you to specify a datastore that will not be used to attach from. This data store can then be he backup/staging/replication point for appstacks across the environment.

[Ray_handels]

We also did a trick with our NFS storage, but I'm not quitte sure if this will work with all storage vendors.

We've just created a folder on the NFS storage and added a datastore to the ESX host. You can then add a storage group into Appvolumes and sync it automatically with just 1 NFS.

Not really backing up but you can actually sync it though.

[anvr]

Sync is not a backup, for example for a few weeks ago an Appstack was gone (status: missing) than it's fine to have a backup instead of a sync process whats sync the missing part also.

[Ray_handels]

Absolutely true. I still would like to see any kind of backup utility.

For now you could just copy the files to a non synced data store and keep it there. Your appstacks won't change that much.

It could off course eventually be done with a copy using VEEAM (I believe it can backup files but takes quite some time) or just download it to a disk.

All still manually. A backup feature would still be nice.

[VArjan]

I guess we all face the same problem…

I expect a solid backup solution in an enterprise product.
As we all know, manual copy actions are doomed to fail.

Only a sync of the volume is not enough, maybe it will help you in case your total storage failure. But if the AppStack is deleted or corrupt the sync will have the same problem.
Since we are investing a lot of time to create and configure all AppStacks I feel not comfortable not having a proper backup solution.

I hope App Volumes 3.x will add a backup feature.

[chriskoch99]

Another large enterprise user here chiming in, disappointed that there isn't better guidance on backing up the files under the [cloudvolumes] directory on the App Volumes datastore(s).  Most enterprise backup tools seem to be focused on backing up VMs and their attached VMDK files rather than on datastore file systems.  I assume this is because the tools don't natively understand how to mount and read VMFS file systems?   VMware should be filling in this gap with product-level backup features, and/or working with the vendors to address the issue.

Deleting the wrong appstack is a scary prospect when there isn't a great way to get it back.  Mitigated somewhat by proper RBAC, but accidents do happen...

[Jason_Marshall]

You have hit the nail on the head. Most backup tools and processes, including VMWare tools, do not understand a single VMDK and instead look at the entire VM. However as stated earlier this tend to be a non-issue with most customers including the very large because it is a single read only file and not prone to corruption, the Storage Groups create "backup" copies across the datastores and if one is deleted by accident it will be replicated back out again from another. We also see that the large enterprise orgs already have some backup process and tool in place. We realized quickly they don't want us to force the use of something specific and would prefer to keep things as simple as possible to allow  flexibility in how they backup. 

Also in 2.10 we will be introducing the concept of a staging or replication datastore that can be used as a hub and spoke replication method for AppStacks. This should help with concerns around backup and accidental deletions as well.

[VArjan]

I agree you do not want to force a backup solution to you customers. But there is nothing wrong of giving them the option.

Each company has his own policies and way of working. I don’t think you will meet all of them.

Earlier this week I was thinking of copying all VMDK files to a server, all of them together are consuming a lot of space. In the end it will use twice the space.

As you said the VMDK files are read only. Why not make an option to mount all the VMDK volumes to the App Volumes server. This way each backup solution can just backup the VMDK files as all other machines. If you have another solution in place you do not have to use the mount option and everybody is happy.

You might run out of drive letters, so it might be a bit more complicate in practice 😉

[Jason_Marshall]

‌Actually you are on to something and in fact we are working to release a fling tool to do something very much along those lines. Once it's out I will post a link on this forum. Stay tuned

[VArjan]

Much appreciated, keep us posted!

[VArjan]

Just curious, any progress on a tool?

[Ray_handels]

We went and tried to do some "funky shit" .

We are using Tintri as our storage which is hybrid NFS storage and created a Windows machine that could attach both Tintri's with a drive letter in Windows thus being able to copy disks around.

We were able to copy VMDK's from one storage to another, import it there and use that one.. All seems to work quite well.. There is just one issue and hopefully someone has some nifty ideas there?

If you copy a writable volume which is in use it cannot be used anymore because it seems to have some sort of "i'm in use" bit in it. Any idea if this can manually be changed?

Also, it copies the disk as Thick (meaning it would need to copy 10GB from 1 storage to another if default size is used) but does store it "Thin" at the other end not consuming the space.

[VArjan]

Is there any progress on a fling?

Belief it or not, I have lost a AppStack

In the mean time I am using Veeam as a backup solution.

This can copy a VMDK which is not attached to a VM.

The downsize is will write the file full size. And I am not able to copy it directly to our deduplication storage backup device. So I don't have enough space to put them somewhere.

It's also a pain to copy the appstack back to it's location, because it's a large file you have to make it smaller again.

Will App Volumes 3.0 add some support for backups?

[sachindsharma]

@VArjun in 3.0, we have a new capability called AppScaling with Mutlizones which allows for automatic replication of AppStacks across multiple datacenters. This might help with what you're looking to do.

[Ray_handels]

My guess is that even this won't be an option. Even if you are able to offsync the appstacks you will still have the writable volumes.

I hope that Appvolumes/VMware comes up with a PowerCLI that can actually copy (or sync or snapshot) a VMDK so you can do this by using scripts on ESX hosts itself.

[VArjan]

A sync/replication is not the same as a backup.

When an AppStack is missing on disk I would like to have a way to get it back. Even after more then 24h.

[anvr]

For now we use 'vmkfstools -i /vmfs/volumes/storage1/cloudvolumes/apps/abcd.vmdk /vmfs/volumes/storage2/backup/cloudvolumes/apps/abcd.vmdk' to backup vmdk's.

[larstr]

Now there's finally a fling here that will help us backup our App Stacks:

Announcing the App Volumes Backup Fling | vDelboy'sView

Lars