Requires clarification on accessing snapshot files

[PavanKumarMath]

I have Linux 64 bit VM. I have enable CTK and then taken a snashots.I have taken about 5 snapshots before that and deleted all.

Now by calling QueryDiskChangedArea("*") got a vector of offset to write and i did active blocks written to destination VM.I used base disk to open and write it.

When i power on the vm its showing the old files. New files whatever i copied between the snapshot creation and deletion is not showing.

Siva, you told that once we delete the snashot all the containts will be merged to base disk.If i power off the VM and open the current snapshot file and write it , its seems to be working.

Need a clarification on the steps.Can some one please clarify this.

with thanks,

Santosh

[sivadevel]

Hi,

Refer to the below text taken from VDDK Documentation. If you pass * it will return all the used blocks. Each snapshot object will have a identifier named changeID. You need to store it when you take a full backup and during the next QueryChangedDiskAreas, you need to send that instead of *. Also note, this should be done per disk, changeID will be different for each disk in the same snapshot itself.

Changed Block Tracking on Virtual Disks

On hosts running ESX/ESXi 4.0 and later, virtual machines can keep track of disk sectors that have changed. This is called changed block tracking. Its method in the VMware vSphere API is QueryChangedDiskAreas, which takes the following parameters:

■	_this – Managed object reference to the virtual machine.

■	snapshot – Managed object reference to a Snapshot of the virtual machine.

■	deviceKey – Virtual disk for which to compute the changes.

■	startOffset – Byte offset where to start computing changes to virtual disk. The length of virtual disk sector(s) examined is returned inDiskChangeInfo.

■	changeId – An identifier for the state of a virtual disk at a specific point in time. A new ChangeId results every time someone creates a snapshot. You should retain this value with the version of change data that you extract (using QueryChangedDiskAreas) from the snapshot’s virtual disk.

When you back up a snapshot for the first time, ChangeId should be unset, or unsaved, indicating that a baseline (full) backup is required. If you have a savedChangeId, it identifies the last time a backup was taken, and tells the changed block tracking logic to identify changes that have occurred since the time indicated by the saved ChangeId.

There are two ways to get this baseline backup:

./Siva.

1	Directly save the entire contents of the virtual disk.

2	Provide the special ChangeId "*" (star). The star indicates that QueryChangedDiskAreas should return only active portions of the virtual disk. For both thin provisioned (sparse) virtual disks and for ordinary virtual disks, this causes a substantial reduction in the amount of data to save.

[PavanKumarMath]

Hi Shiva,

Thanks for reply. I have done some progress.Following is the code snippet using for querydiskchangedarea.

  String changeId = "52 a0 df 53 4f 90 0b 19-7b 14 80 28 68 f7 d2 a3/13";

               int diskDeviceKey = 2000;

               DiskChangeInfo changes = null;

               long position = 0, tot_len = 0;

               do {

                   changes = vimPort.queryChangedDiskAreas(vmMor, snapshot, diskDeviceKey, position, changeId);

                  .........................

              }

The issue is that if i pass "*" and write those block , its working fine. If i pass changedId as above (previous snapshot), queryChangedDiskAreas reporting vector of offset and length.But the files copied on source VM is not writing to destination VM. destination VM is on previous state itself. I have not encountered with any error though.

[PavanKumarMath]

I got the change id and device key from the ESXi webservice. As of now its hard coded values.Let me know if i am doing any mistakes here.

[sivadevel]

I am not sure what could be the problem. This part had been pretty straight forward for me though. Say if you are creating a new file of about 2MB after a snapshot backup, and when you trigger the next queryChangedDiskAreas call, did you print the Start and lenght of the DiskChangeExtent structure? It should roughly correspond to the 2MB change you made. Other option to try would be to create a child disk to the pervious parent with the returned DiskChangeExtent and see how big the child disk gets.

[PavanKumarMath]

Hi Shiva thanks for your reply...

Snapshot Name : TESTSNAP

START 526385152  LENGTH 65536

START 529137664  LENGTH 65536

START 529268736  LENGTH 65536

START 529924096  LENGTH 65536

START 530251776  LENGTH 131072

START 531693568  LENGTH 131072

START 794886144  LENGTH 65536

START 822280192  LENGTH 65536

START 914751488  LENGTH 65536

START 2673999872  LENGTH 131072

START 2677014528  LENGTH 65536

START 4821483520  LENGTH 65536

START 4823973888  LENGTH 65536

START 4824956928  LENGTH 65536

START 4826267648  LENGTH 65536

START 4826791936  LENGTH 65536

START 4969267200  LENGTH 983040

START 6969425920  LENGTH 65536

START 6969753600  LENGTH 65536

START 6970736640  LENGTH 65536

START 6971457536  LENGTH 65536

START 6972178432  LENGTH 65536

START 6977159168  LENGTH 65536

START 9116319744  LENGTH 131072

START 9118351360  LENGTH 65536

START 9118744576  LENGTH 65536

START 9121234944  LENGTH 131072

START 9121693696  LENGTH 65536

START 9154658304  LENGTH 65536

START 9253027840  LENGTH 131072

START 9691004928  LENGTH 65536

START 10179706880  LENGTH 131072

Start offset :smileylaugh:  length 16106127360  tot len 3473408

I copied about 2.4MB file, QueryDiskChangedArea reported ~3.4MB. I have written these blocks from source VM to destination VM. Still its showing old files.

[PavanKumarMath]

Shiva,

You mentioned about the creating child disk and parent disk. Can you please tell me the context where it will be used. As a part of CBT is it required to use it in some cases?

with thanks,

Sanotsh

[sivadevel]

You can se VixDiskLib_CreateChild function to create a child disk and write the incremental changes alone to that file. Then you should use VixDiskLib_Attach to parentdisk. Atleast you would know the latest changes are being made to the VMDK

[togtog]

Hi guys,

I have been observing your discussion and I am not quite sure what the overall goal is. To me it seems like PavanKumarMath is trying to write a replication application. At least I understand this from here.

@ PavanKumarMath

Now my question is: Do you want to replicate the VM directly from one server to another - where server can be vCenter, ESX or ESXi?

--

Thomas G.

[PavanKumarMath]

I agree with u. But the thing is that if i pass "*" and write it to destination it is working fine. The only issue is that if i pass the revious snapshot Change_id.Then no change in the state of VM.Doesn't this rule out the above mentioned potential issue?  I understand that CBT will be enabled only once. after that re-enable is not required ?

[sivadevel]

No CBT is enabled only one, just before full backup is taken. I think we are losing track of your actual objective? Thomas, who has answered below has helped me trouble shoot almost all the problems I faced, guess it is time to take his lead and answer his question to get back on track.

[PavanKumarMath]

@Thomas,

You are right. I am writing the VM replication components based on CBT mechanism. Till now what i did that on the same ESXi machine : read the current VM's VMDK and write it to destination VM's VMDK. Copying only active blocks returned by QueryDiskChangedArea("*") API. This is working fine for me. If i pass the change id of previous snapshot and current snapshot MOR to QueryDiskChangedArea reporting the changed blocks.But when i read / write, changes are not replecting on destination VM. Still it is in previous state.

with thanks,

Santosh

[togtog]

Hi Santosh,

Copying only active blocks returned by QueryDiskChangedArea("*") API. This is working fine for me.

OK from now on I assume that this means that your baseline replication, i.e. the first replication run, works and that any following

If i pass the change id of previous snapshot and current snapshot MOR to QueryDiskChangedArea reporting the changed blocks.But when i read / write, changes are not replecting on destination VM. Still it is in previous state.

does not.

From here it seems to me that Siva has already successfully guided you to verify that the changeId based calls to QueryDiskChangedAreas(...) are done properly. So from now on I assume that you actually receive correct DiskChangeInfo objects, i.e. your handling of the source VM is assumed to be correct.

This brings me to destination machine. Following the discussion and also from this discussion's title it is suggested that there is something wrong with snapshot handling. So please, as properly as you can, step by step explain what you do to the destination machine. Especially:

• Is the destination VM powered on or off?
• Do you create any snapshots there?
• Which files do you actually write to?
• Which transport mode is in effect: nbd, nbdssl, hotadd, san?

And please really step by step, otherwise it will be very hard to help you with this problem, if at all 😉

Looking forward to hearing from you ...

--

Thoma G.

[PavanKumarMath]

Thanks a lot for response.So far what i did that :

Create a dummy destination VM with same configuration as of source VM manually.

First pass :-

1. Create a snapshot of source VM.

2. Get the active blocks of Source VM.

3. Read the blocks by using nbd. Here opening the base disk (Not the snapshot disk file). In the vSphere client UI it is showing the snapshot file in the form of vmname/vmname-000001.vmdk is in use.Source VM is in powered on state.

4.Open the destination VM disk by using nbd. Write it to its base disk (vmname/vmname.vmdk) file. Destination VM is powered off.

Note:- For reading/writing using the VixDiskLibSample. Modified to read/write 1024 sectors at a time. This program will be running in the one of the VM on same ESXi host.Call it as a Appliance.

5. Remember the ChangeID of source VM snapshot. Change ID i am getting from the ESXi's Webservice.

6. Delete the Source VM snapshot.

After write is complete Power on the destination VM. It comes up with Source VM replica.

========================================================================================================================

Second pass:-

Write the files to source VM. I have written about 2.7 MB files.

1. Create a snapshot of Source VM.

2. Get the changed blocks by passing ChangeId of previous snapshot.QueryDiskChangedArea function reporting about 3.7 MB changed blocks.

And rest of the steps are same as above.

when i power on the destination VM. whatever files i copied during the second pass is not showing. BUT If follow the steps in the first pass, its shows the files whatever i copied.

==========================================================================================================================

To answer your questions i am not creating any snapshots on destination VM and its powered off.

Please let me know what could be the issue.If you need any clarification let me know.

with thanks,

Sanotsh

[togtog]

Hi Sanotsh,

Hhhhhmmmm your algorithm looks good to me at a frist sight. To be honest I am not sure what the problem is.

Very dumb idea, have you already tried it with a larger change, let's say 1 GB, just want to make sure that there is not any kind of caching problem around.

--

Thomas G.

[sivadevel]

One question. How and when is CBT enabled at the source in the above sequence?

[PavanKumarMath]

Hi Siva,

Thanks for reply..

I enabled it during the first pass and reconfigured the VM. I have verified it on the vSphre client GUI also. Its showing ctkenable &scsi:ctkenable flag as true :

code snippet:

vmConfigSpec.setChangeTrackingEnabled(true);

      ManagedObjectReference tmor

         = vimPort.reconfigVMTask(

              virtualMachine, vmConfigSpec);

with thanks,

Santosh