VMware Cloud Community
mannyb
Contributor
Contributor
Jump to solution

Snapshots withn ESX3 created by ESXpress

Hello All,

Greeting from the BX, I am in the process of narrowing down why the ESXpress application is skipping delta backups within my environment. I logged into the console and noticed that snapshots are being created but nothing is showing it was created by virtual center snapshot manager. The files created have a delta.vmdk name, which leads me to believe its actually ESXpress. No backups have occurred in quite some time and I am debating if I should just commit these deltas or simply delete them

Manfred

Reply
0 Kudos
1 Solution

Accepted Solutions
kharbin
Commander
Commander
Jump to solution

Manny,

When you commit the snaps, VMware will add an additional snapshot called "Consolidate helper". the idea is that all writes will now occur to this snapshot while the previous snapshots are comitted back into the main VMDK file. This allows very large snaps to be committed without freezing the VM. When the first commit is complete, it then commits the "Consolidate helper" snapshot back into the main VMDK. When complete, VMware does have to freeze the VM for a few milliseconds in order to close the file handle associated with that snap, and assign the file handle of the flat-VMDK back to the VM as the primary.

So no, you should not loose access to your VMs during the commit and should encounter no problems.

Ken Harbin

www.esXpress.com

View solution in original post

Reply
0 Kudos
18 Replies
conradsia
Hot Shot
Hot Shot
Jump to solution

The Xpress snapshots will cleanup the snapshots then commit and they will also be identified as "snapshots created by esXpress". You probably don't want to be running in snapshot mode so unless you don't want your changes I would recommend commiting them.

mannyb
Contributor
Contributor
Jump to solution

Conradsia,

I understand what you are saying, but the backups have been scheduled and skipped on multiple occasions with no cleanup in site. Sure committing them is an option but, one I am unsure how to, secondly, what if it halts my systems while committing? thats not really an option

Reply
0 Kudos
RParker
Immortal
Immortal
Jump to solution

you need to take this up on the esXpress Support forum, not on here.

They can give you an INSTANT response, esXpress doesn't really apply to ESX, even though it's a VM, its a custom made appliance.

Reply
0 Kudos
kharbin
Commander
Commander
Jump to solution

Hi Manny,

Manny, I can find no record of your calling our support techs. Give our support line a call at 717.983.4065. Also, send a copy fo your backup report to supprt at esxpress.com. This report often details exacly why a VM is skipped.

Ken Harbin

www.esXpress.com

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

R Parker,

I do see what you are saying, but the basis of my question implies snapshots and not only if I should commit them but also, how to do it. I did post on the esXpress forum, for the initial part of the question. Thanks for the advice.

Mannyb

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

Kharbin,

I will generate the log internally and contact them directly

Thanks

Mannyb

Reply
0 Kudos
RParker
Immortal
Immortal
Jump to solution

But the snapshots are PART of the esXpress setup, and they only leverage ESX to create them, but this is 100% an esXpress support issue, not an ESX issue.

As Ken said, give them a call, or get on the esXpress Support board for questiions related to esXpress. Most of the people on here use ESX Ranger for backup or some other backup appliance, NOT esXpress.

The snapshots can also be cleaned up by running PHD, and follow the prompts to check for out of sync backups, and the beta version .04 can commit the snapshots BEFORE the next backup.

PHD has a lovely set of tools you can use to diagnose the problem, or even fix them. Either way, this is \*NOT* an ESX problem, it was precipitated by a problem with esXpress, not inherit to ESX 3.0. Therefore THEY need to support your issue, not VM Ware or discussion forums.

mannyb
Contributor
Contributor
Jump to solution

Rparker,

what I have been trying to decipher between was which application was causing the snapshots to appear within the environment as well as how to clean them up.

Perhaps you are correct in that it may require PHD to cleanup the snapshots however, when committing snapshots whether PHD or snapshot manager does that not render the system inaccessible?

In any case I do appreciate your suggestions and trying to point me in the right direction

Thanks

MannyBx

Reply
0 Kudos
RParker
Immortal
Immortal
Jump to solution

"Perhaps you are correct in that it may require PHD to cleanup the snapshots however, when committing snapshots whether PHD or snapshot manager does that not render the system inaccessible?"

No, but you can revert safely (which means going back to the way the VM was before they took a snapshot).

If the VM is running now, and they were created by esXpress, I wouldn't touch it. Everything is fine. The snapshot quiesces the VM in order to make a copy of it (VMDK files are in use). So chances are it can be safely committed (since esXpress commits these between backups anyway).

BTW, I love esXpress, and it's a very good option. I just want you to get the BEST support for this product...

Reply
0 Kudos
kharbin
Commander
Commander
Jump to solution

If the snapshot is named _esxpress then esXpress requested that the VMware snap manger create this snapshot. If it is named consolidate helper, then VMware created this snapshot.

Why are they not clenaed up? Could be anyone of a dozen reasons; the esXpress software is turned off, you have placed an abort/stop lock on the host, the scheduler was stopped, the VMs were vmotioned to a host not esXpress aware, the snap count is inconsistant, file permissions were changed, etc.

But without a shred of information, I can not help you.

And no, snap manager does not render the system inaccessible.

Ken Harbin

www.esXpress.com

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

I agree that touching it with out esXpress support is not an option. I have generated a support script and sending it as we speak( cool option) I have worked with ESx ranger and other products but none offer the scalability and reliability ( yet)

Thanks Again

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

Perhaps I have chosen the wrong set of words.

I meant to state that I loose connectivity to the VM when commiting snapshots, especially when using esXpress. Its not something that I am inclined to do in the middle of the day while in production.

I have generated the support script and transmitted the information,

Thanks for the quick suggestions and help.

Mannyb

Reply
0 Kudos
kharbin
Commander
Commander
Jump to solution

Hi Manny,

OK, see the problem. 3 VMs, each had a snap added, one on 7/13, one 7/16 and one on 7/28. Each snap was requested by esXpress. But esXpress only requests a snap be made, the VMware snap manager actually does the work. In each case snap manager added the snap shot OK, esXpress performed the backup, and esXpress requested the snap manager remove the snapshots. But instead of you removing the snapshots, you experienced the classic snap manager bug where the snap count and the number of actual snapshots become out of sync.

Also, I see you are using vmkernel version 40087. VMware has released updates that have dramatically increased the stability of the snap manager. Make sure you have at least up to May 15 ESX patch set installed on each host.

So, to fix these VMs is fairly simple. For each VM, just add another snapshot, then simply delete all snapshots. Adding a snap will force the snap manager to fix the snap count. Then the delete all will remove all the snaps.

Thanks for the file, definately makes it easier.

Ken Harbin

Reply
0 Kudos
RParker
Immortal
Immortal
Jump to solution

"Each snap was requested by esXpress. But esXpress only requests a snap be made, the VMware snap manager actually does the work"

So what you are saying is that esXpress is the manager on site, and VM ware works for the union, and since they get paid no matter what, they don't have any incentive to get the job done, and since the manager is non-union they won't even talk to him...[/i]

"In each case snap manager added the snap shot OK, esXpress performed the backup, and esXpress requested the snap manager remove the snapshots"

In essence, esXpress hire a consultant to do other work (since hiring more union workers is against the contract) and since it was a non-skilled task, the job wasn't clearly defined...[/i]

"But instead of you removing the snapshots, you experienced the classic snap manager bug where the snap count and the number of actual snapshots become out of sync. "

A spokesperson for the work force at the VM ware center said[/i] "The job entails many hours of dedication, and hard work. We have established the cause to be a known issue, and as such we are working around the clock to finanlize the job, and we should have a resolution soon."[/b]

"Also, I see you are using vmkernel version 40087. VMware has released updates that have dramatically increased the stability of the snap manager. Make sure you have at least up to May 15 ESX patch set installed on each host"

"Due to unforseen events beyond our control, we have asked the management for addtional resources to resolve the issue, and upon extending addtional hours to the crew, the problem will be resolved quickly and accurately once the proper fixes have been put into practice"[/b]

"So, to fix these VMs is fairly simple. For each VM, just add another snapshot, then simply delete all snapshots. Adding a snap will force the snap manager to fix the snap count. Then the delete all will remove all the snaps. "

So there is this guy named "Guido", and he is keeping two sets of books you see...[/i]

\- Tony Soprano

"Thanks for the file, definately makes it easier. "

unions, can't live with 'em . . . . .[/i]

Ken Harbin - da boss

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

Ken,

Thanks for the info, I just have one additional question to this bug, when I add the snapshot and then remove all snapshots, will it cause me to loose accessibilty to the VM?

After looking at posts on both esXpress and Vmware forums there have been instances of loosing access to the Virtual Machines. Have you encountered similar situations?

Thanks

Mannyb

Reply
0 Kudos
kharbin
Commander
Commander
Jump to solution

Manny,

When you commit the snaps, VMware will add an additional snapshot called "Consolidate helper". the idea is that all writes will now occur to this snapshot while the previous snapshots are comitted back into the main VMDK file. This allows very large snaps to be committed without freezing the VM. When the first commit is complete, it then commits the "Consolidate helper" snapshot back into the main VMDK. When complete, VMware does have to freeze the VM for a few milliseconds in order to close the file handle associated with that snap, and assign the file handle of the flat-VMDK back to the VM as the primary.

So no, you should not loose access to your VMs during the commit and should encounter no problems.

Ken Harbin

www.esXpress.com

Reply
0 Kudos
RParker
Immortal
Immortal
Jump to solution

you guys are good.

I would buy your services based upon the support that I have seen and have experienced myself, just on the support alone.

your product is good also.

The biggest problem people have with products, isn't the product, but the people that support the product. It's good to see you are on top of things.

Reply
0 Kudos
mannyb
Contributor
Contributor
Jump to solution

Rparker,

Seriously, I completely agree with what you are saying as far as esXpress. Its good to know the support is there no matter where you post.

I committed snapshots last evening and I did loose access to the system for more than an acceptable threshold. I then had the VC client return a timeout, which I understand as the snapshots were being committed, but it was successful.

Again thanks for all your help

mannybx

Reply
0 Kudos