QueryChangedDiskAreas problem

[eharney]

I'm trying to use QueryChangedDiskAreas on an ESX 4 server through the Perl SDK. The method I'm using works for some VMs, but for others, I get back a FileFault. Looking at the ESX server logs reveals this:

(me) Calling QueryChangedDiskAreas with snapshot: 64-snapshot-35, deviceKey: 2000, changeId: *

2010-04-05 16:57:20.316 F63A7B90 error 'vm:/vmfs/volumes/4b7da659-f2a39f3d-b04d-003048d5dbf0/NewVM/NewVM.vmx' Cannot compute changes for disk /vmfs/volumes/4b7da659-f2a39f3d-b04d-003048d5dbf0/NewVM/NewVM-000007.vmdk: Unknown change epoch.

2010-04-05 16:57:20.316 F63A7B90 info 'App' AdapterServer caught exception: vim.fault.FileFault

2010-04-05 16:57:20.316 F63A7B90 info 'Vmomi' Activation N5Vmomi10ActivationE:0x5dced698 : Invoke done queryChangedDiskAreas on vim.VirtualMachine:64

2010-04-05 16:57:20.316 F63A7B90 verbose 'Vmomi' Arg snapshot:

'vim.vm.Snapshot:64-snapshot-35'

2010-04-05 16:57:20.316 F63A7B90 verbose 'Vmomi' Arg deviceKey:

2000

2010-04-05 16:57:20.316 F63A7B90 verbose 'Vmomi' Arg startOffset:

0

2010-04-05 16:57:20.316 F63A7B90 verbose 'Vmomi' Arg changeId:

"*"

2010-04-05 16:57:20.316 F63A7B90 info 'Vmomi' Throw vim.fault.FileFault

This is on a VM with hardware version 7, and change tracking is enabled as far as I can tell. Any idea about what is causing this error? Is there something that is misconfigured for this particular VM, or am I calling it incorrectly? (I'm not sure what "unknown change epoch" is referring to.)

Thank you.

[lamw]

Generally for API/SDK questions, you'll want to post in the appropriate developer forum - http://communities.vmware.com/community/developer

Regarding the fault that you're seeing, if you take a look at the API reference document (http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.VirtualMachine.html#queryChangedDiskAreas) regarding the potential faults you could receive when issuing this function call, you'll notice one of them is fileFault which is caught as an exception within your error message:

2010-04-05 16:57:20.316 F63A7B90 info 'App' AdapterServer caught exception: vim.fault.FileFault

If you take a look at the fault description, it means - Thrown if the virtual disk files cannot be accessed/queried.

So for whatever reason, it was either unable to query this information OR it could not access the disk files and the "unknown change epoch" is most likely referring to this problem as it's unable to figure out the change for this given point in time.

You may also want to take a look at pg 21 of guide in VDDK if you haven't already for designing backup solutions using VDDK + vSphere API and change block tracking - http://www.vmware.com/support/developer/vddk/vadp_vsphere_backup111.pdf which provides some details in how to query for these change blocks/etc which may be helpful.

=========================================================================

William Lam

VMware vExpert 2009

VMware ESX/ESXi scripts and resources at:

Twitter: @lamw

VMware Code Central - Scripts/Sample code for Developers and Administrators

VMware Developer Community

If you find this information useful, please award points for "correct" or "helpful".

[eharney]

Thanks for the tips. I haven't quite gotten to the bottom of why this specific error actually occurs, but I'll do some more digging on the matter.

(Also, I thought I had gotten into the right forum -- oops.)

[hubs]

Hi eharney

Did you make some progress about this error ?

I have the exact same error, but not on all virtual machines.

[eharney]

hubs: Not significantly. I determined that it was happening to VMs on a particular datastore, moving them to another datastore caused the problem to stop occurring. I don't know what the cause was.

[hubs]

What kind of datastore do you use ?

I've had this problem on ESXi, ESX and vCenter. All were using a datastore on a local disk.

[eharney]

This was all while using local disk datastores on ESX 4.0.

[hubs]

Ok, that might be something else them.

What if you create a VM, make a snapshot, then activate Change Block Tracking, then do a new snapshot and QureyChangedDiskAreas with "*" ?

[eharney]

I just hit a similar problem with some VMs on a different server which is running ESXi 4.0, where the call fails with a FileFault. However, I don't see the "unknown change epoch" error in the logs on the server in this case.

I tried creating a new VM on that server, and used QueryChangedDiskAreas with "*" which worked. (Reported 0 blocks since the VM hadn't been used yet.) I don't yet understand what conditions cause it to fail... each time I've had this problem, it's been on a VM which was working for a while, then began reporting this error.

Message was edited by: eharney

(edited: not the same error)

[hubs]

By following the steps in my previous post, I can each time trigger the problem.

So, I thought it was related to the fact that a snapshot with Change Block Tracking disabled exists.

Not so sure so

By the way, how do you understand the meaning of using "*" as the change id ?

The documentation is not really clear.

It's either:

- all allocated blocks.

- or all changed blocks since the most recent change ID (so the most recent snapshot).

That makes a huge difference for me.

[eharney]

Using "*" as the change id returns all blocks that have ever been changed on the disk (all allocated blocks).

[hubs]

Thanks eharney.

That's the behavior I seemed to observe, while the documentation stated: It only makes sense to use the special ChangeId "*" when no previous ChangeId exists.

[eharney]

I'm still encountering this issue every now and then. Most recently I hit this on ESX 4.1, and was able to work around it by:

1. Power off the VM

2. Disable change tracking

3. Enable change tracking

4. Power on the VM

After doing this, QueryChangedDiskAreas() worked. (Previously it returned FileFault with the ESX server logging "Unknown change epoch".)

I get the impression that there are a lot of common conditions that cause change tracking to fail, I really hope this gets easier in future releases. Previously I had found that disabling and re-enabling change tracking while no snapshots exist is sometimes required, but having to power off the VM for it to work is worse.

[SonalJain]

Hi All,

Observed behavior is known issue and has been documented in the "Designing Backup Solutions for

VMware vSphere" under section "Gathering Changed Block Information".

The following text in this section can be referred for the scenario reported in this case:

"For best results, when using QueryChangedDiskAreas to gather information about snapshots, enable change tracking before taking snapshots. Attempts to collect information about changes to snapshots that occurred before change tracking was enabled result in a FileFault error."

As a workaround for current version, enable "changeTrackingEnabled" before creating any snapshot of VM.

Ref: Page no 23 :  http://www.vmware.com/support/developer/vddk/vadp_vsphere_backup12.pdf

Regards,

Sonal