Can't move nor clone VM (VMDK access denied error)

[citation79]

Hi,

I have been using Veeam to make a backup of all my VMs in my infrastructure.
Now I have a single VM that is not doing its backup. In the logs it says:

Error: VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

For some reason all the other VMs on the server do get copied to their backup locations without issues. They are inside the same backup job.

I tried to vMotion the VM to another storage on the server, it failed because of the VMDK file.

Tried to clone it, it failed as well.

The VM is running OK but it is having performance issues and I need to get it off that server or storage.
What would be the best way to get it off?

Thanks much in advance.

Teo

Error:   VDDK error: 13 (You do not have access rights to this file). Value:   0x000000000000000d Failed to upload disk. Agent failed to process method   {DataTransfer.SyncDisk}.

[a_p_]

Try to contact continuum​. He may be able to help.

André

[citation79]

Thanks!

[continuum]

Hi Andre

he already called me this morning.

Here is the result of the anamnese:

status of hardware: the critical 1 TB vmdk is stored on an aging SATA disk.

Last Veeam backup jobs faild because the temporary snapshot was located in a VMFS area that reported an I/O error.

At the moment there is no snapshot but trying Veeam again can easily fail again when the delta is allocated without luck again.

vmkfstools -i 1tb-source.vmdk other-datastore/new.vmdk -d thin was able to start - it even advanced up to 15 % or so - but waiting for the result was no option because of production needs.

vmkfstools -p 0 source-flat.vmdk > mapping.txt was possible but size of mapping.txt was about 7 MB and the rate of 1 MB fragments was very very large.

As earlier attempts to use GUI based cloned functions had failed before.

So the optionsremained were limited.

Extra problem: we had to expect I/O errors during any vmkfstools based clone operation.

That means we need an approach that can handle I/O errors along the way - so ESXi native UI functions, vmkfstools -i could be ruled out as well.

No use in starting a clone process - with an expected run-time of 2 days or so that can not be resumed and needs to be restarted on errors.

So only 2 options remained:

1. use vmkfstools -p 0 result to create a sh.script with about 49000  one line dd-commands - add good logging to that so that it can be relaunched where it failed ....

In other words - creating that script would be a pain - quite error prone and it would be a primitive poor mans ddrescue at best.

2. create Linux LiveCD VM using http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso

Attach 1 TB old vmdk as /dev/sda

Attach new 1TB on target datastore as /dev/sdb

Attach small 1gb vmdk for logging as /dev/sdc

Format small 1gb vmdk so that it can used for writing ddrescuelogs

Then run ddrescue /dev/sda /dev/sdc rescue.log over the weekend.

Thats my favorite approach to clone large vmdks which reported I/O errors on previous attempts.

Major advantafe of using real ddrescue versus the poor man version with ESXi - dd and a handwritten script:

the expected error size of real ddrescue is in most cases way smaller

If you esxis dd with a script you have to manually add additional 2 lines when dd of one fragment errors out.

TIf you start with a 512 MB fragment which dd does not want to read and reports IO error you start with spliiting that 512 mb fragment into 2 smaller ones in the hope to get a large piece before the error and a large piece behind. Lots of guessing, testing, mistakes .... really painful work if you want get best possible results.

If the VMDK itself is still launchable the ddrescue from inside a Linux VM  compares to the "dd scripts from esxi approach" as dragging a bronze age ox cart yourself through knee deep mud compares to driving the same distance as a passenger in the backseat of a Rolls Royce Limo.

And the results are typically way better.

So we decided to try the Linux variant after production ends on friday evening.

Unknown variable: will the hardware survive ....

I give us a quite good prognosis to end with a usable VMDK after the weekend.

Anyway - all other options I had in mind have a way too high failure chance to bet on it.

Ulli