Lost VM after datastore full for snapshot

[lpapaleo]

Hello,

I've an host ESXi 6.5 with a big problem.

I think something's gone wrong with Veeam Replica since 14 days and all the snanpshot haven't been removed.

So the datastore collapse and become full.

After my collegue support now I've found this situation:

- the VM has attached just one hard drive (and it's the original one - 300gb size)

- the VM folder has the following file:

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000001-ctk.vmdk

-rw-------    1 root     root     91456798720 Feb 14 12:26 SERVER_0-000001-sesparse.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000002-ctk.vmdk

-rw-------    1 root     root     1423683584 Feb 13 22:25 SERVER_0-000002-sesparse.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000003-ctk.vmdk

-rw-------    1 root     root     1508777984 Feb 13 22:25 SERVER_0-000003-sesparse.vmdk

-rw-------    1 root     root           409 Feb 12 21:40 SERVER_0-000003.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000004-ctk.vmdk

-rw-------    1 root     root     1748148224 Feb 13 22:25 SERVER_0-000004-sesparse.vmdk

-rw-------    1 root     root           409 Feb 12 21:47 SERVER_0-000004.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000005-ctk.vmdk

-rw-------    1 root     root     11366363136 Feb 13 22:25 SERVER_0-000005-sesparse.vmdk

-rw-------    1 root     root           409 Feb 12 22:01 SERVER_0-000005.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000006-ctk.vmdk

-rw-------    1 root     root     3519012864 Feb 13 22:25 SERVER_0-000006-sesparse.vmdk

-rw-------    1 root     root           409 Feb 14 01:17 SERVER_0-000006.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000007-ctk.vmdk

-rw-------    1 root     root     3489665024 Feb 13 22:25 SERVER_0-000007-sesparse.vmdk

-rw-------    1 root     root           409 Feb 12 22:31 SERVER_0-000007.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000008-ctk.vmdk

-rw-------    1 root     root     5074522112 Feb 13 22:25 SERVER_0-000008-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 01:41 SERVER_0-000008.vmdk

-rw-------    1 root     root       4915712 Feb 13 22:24 SERVER_0-000009-ctk.vmdk

-rw-------    1 root     root     1288708096 Feb 13 22:25 SERVER_0-000009-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 01:42 SERVER_0-000009.vmdk

-rw-------    1 root     root       4915712 Feb 14 01:17 SERVER_0-000010-ctk.vmdk

-rw-------    1 root     root     10377076736 Feb 13 22:25 SERVER_0-000010-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 05:45 SERVER_0-000010.vmdk

-rw-------    1 root     root       4915712 Feb 14 01:18 SERVER_0-000011-ctk.vmdk

-rw-------    1 root     root     1288765440 Feb 13 22:25 SERVER_0-000011-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 05:46 SERVER_0-000011.vmdk

-rw-------    1 root     root       4915712 Feb 14 01:18 SERVER_0-000012-ctk.vmdk

-rw-------    1 root     root     1781026816 Feb 13 22:25 SERVER_0-000012-sesparse.vmdk

-rw-------    1 root     root           409 Feb 14 01:18 SERVER_0-000012.vmdk

-rw-------    1 root     root     1288704000 Feb 13 22:25 SERVER_0-000013-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 09:50 SERVER_0-000013.vmdk

-rw-------    1 root     root       4915712 Feb 14 07:27 SERVER_0-000014-ctk.vmdk

-rw-------    1 root     root     13981650944 Feb 14 07:27 SERVER_0-000014-sesparse.vmdk

-rw-------    1 root     root           409 Feb 13 09:51 SERVER_0-000014.vmdk

-rw-------    1 root     root       4915712 Feb 15 14:27 SERVER_0-ctk.vmdk

-rw-------    1 root     root     322122547200 Feb 15 14:27 SERVER_0-flat.vmdk

-rw-------    1 root     root           612 Feb 15 10:26 SERVER_0.vmdk

If I go to snapshot manager I haven't any snapshot pending.

If I turn on the VM the VM start but has the data of 29th January (before the first snapshot).

What I can notice from the list is that the SERVER_0-000002-ctk.vmdk file is missing

Some help or idea?

Thanks in advance

Luca

[lpapaleo]

Following this link

VMware Knowledge Base

with some change I've been able to rebuild the chain

Now the command:

vmkfstools -e SERVER_0-000014.vmdk

I get this output:

Disk chain is consistent.

But I still can't see snapshot in snapshot manager

What can I do in next step?

Luca

[continuum]

Snapshot manager depends on a healthy vmsd-file.
Unfortunately the vmsd-file is quite fragile and it often gets messed up by automatic backup tools.
I would recommend to clone the current snapshot chain into a new vmdk using vmkfstools -i.
If your current vmx-file references the base vmdk you will see an outdated state of the VM if you power it on.
Do not run the VM like this - power it off immediatly and check wether you assigned the correct vmdk-file

[continuum · Accepted solution]

Problem solved after a short skype chat.

[lpapaleo]

Really great support!! Thanks a lot!!

Luca