VMware Cloud Community
Data-mail-2000
Contributor
Contributor
Jump to solution

File VMDK disappear into sesparse and ctk into ESXi

Hello,

We are a problem with a snapshot. The VMDk from this snapshot has changed into 2 files -ctk and sesparse. How sync this files to restart VM ?

Have a nice day

 

Labels (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

Ok, with the files we have, here's what we can do (no guarantees though, due to the missing sesparse file)

As the first step, please ensure that your Veeam backup server, or a Veeam proxy (in case Hot-Add backup is used) does not still have the VM's virtual disks attached due to an unsuccessful backup attempt. In case it has, remove them from the Veeam VMs (do NOT delete them from disk!). Also make sure that the backup does not kick in while preforming the following tasks!

  1. create a sub-directory in the VM's folder
  2. move the following files to the sub-folder: *ctk.vmdk, *.vmsd, *.vmsn, *.log
  3. extract the 7 files from the attached .zip archive, and upload then to the VM's folder
  4. register Dummy1.vmx, and Dummy2.vmx as VMs from the Datatore Browser
  5. right-click the Dummy1 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
    --> after the task completes, you should have ~27GB free disk space on the datastore
  6. right-click the Dummy2 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
    --> now all of the 00000x.vmdk files in the VM's folder should be gone (merged into their parent files)
  7. unregister the two Dummy VMs from the inventory (do NOT delete from disk!)
  8. reload the VM from the command line (see https://kb.vmware.com/s/article/1026043)
    vim-cmd vmsvc/getallvms
    vim-cmd vmsvc/reload <vmid>
  9. delete the files Dummy1.vmx, and Dummy2.vmx from the command line
  10. create a VM snapshot, so that the VM's current files won't get modified
  11. power on the VM

If things work as expected, you may then delete the previously created sub-directory, and also the current VM snapshot.
Also remember to delete the files that you've copied to other datastores once you don't need them anymore. Make sure that you don't accidentally delete other files/folder too!

In case you have questions, please feel free to ask.

André.

View solution in original post

Reply
0 Kudos
15 Replies
a_p_
Leadership
Leadership
Jump to solution

>>> ... into 2 files -ctk and sesparse.
Isn't there also a small <vmname>-00000x.vmdk descriptor file for the snapshot in the VM's folder?

Please run ls -lisa from a command line in the VM's folder to get a complete file listing, and provide the text output in your next reply. The GUI "hides" some files.

André

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Hello,

Here the ls -lisa make into the datastore.

Thanks

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Anyway, it seems that some important files are missing, which sometimes occur if a datastore runs out of disk space.

To see what can be done, please run the following commands in the VM's folder:

  • df -h
  • ls -lisa
  • grep vmdk *.vmx

Please provide (copy&paste) the the commands' output as plain text rather than a screen shot with a truncated output.

In addition to the commands' output, use e.g. WinSCP, download all of the VM's descriptor .vmdk files (the small ones without flat, sesparse, or ctk in their names), then compress/zip them, and attach the .zip archive to your next reply,.

André

 

 

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Hello,

 

Here the logs :

[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] grep vmdk *.vmx
scsi0:0.fileName = "SRV-EXCH01.vmdk"
scsi0:1.fileName = "SRV-EXCH01_1.vmdk"

 

[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 830.8G 658.2G 172.6G 79% /vmfs/volumes/datastore1_15K
VMFS-6 447.0G 364.6G 82.4G 82% /vmfs/volumes/datastore2_SSD
VMFS-6 558.8G 545.2G 13.6G 98% /vmfs/volumes/datastore3_15K
vfat 285.8M 209.1M 76.8M 73% /vmfs/volumes/5b96778d-46f5c96a-d744-20677cd6dd34
vfat 4.0G 22.2M 4.0G 1% /vmfs/volumes/5b967794-c66f7d18-32ee-20677cd6dd34
vfat 249.7M 183.2M 66.5M 73% /vmfs/volumes/c4424f7d-503e5cff-1dea-9f7ecbbca204
vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/7cfb2376-3f54328b-d3ed-40e98635732b

 

[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] ls -lisa
total 570197504
3780 128 drwxr-xr-x 1 root root 86016 Nov 9 15:43 .
4 1024 drwxr-xr-t 1 root root 73728 Nov 9 12:26 ..
96469956 4096 -rw------- 1 root root 3932672 Nov 3 22:15 SRV-EXCH01-000001-ctk.vmdk
46138308 10146816 -rw------- 1 root root 10389823488 Nov 9 12:29 SRV-EXCH01-000001-sesparse.vmdk
83887044 4096 -rw------- 1 root root 3932672 Nov 9 14:06 SRV-EXCH01-000002-ctk.vmdk
25166788 4683776 -rw------- 1 root root 4795998208 Nov 9 14:06 SRV-EXCH01-000002-sesparse.vmdk
88081348 0 -rw-r--r-- 1 root root 53 May 8 2020 SRV-EXCH01-18ae0f4e.hlog
92275652 1024 -rw------- 1 root root 294474 Nov 9 13:35 SRV-EXCH01-Snapshot1283.vmsn
79692740 4096 -rw------- 1 root root 3932672 Nov 9 13:06 SRV-EXCH01-ctk.vmdk
16778180 62429184 -rw------- 1 root root 64424509440 Nov 9 13:06 SRV-EXCH01-flat.vmdk
29361092 1024 -rw------- 1 root root 270840 Nov 9 13:02 SRV-EXCH01.nvram
964 0 -rw------- 1 root root 567 Nov 9 13:02 SRV-EXCH01.vmdk
8389572 0 -rw-r--r-- 1 root root 1052 Nov 9 13:09 SRV-EXCH01.vmsd
20972484 64 -rw-r--r-- 1 root root 3832 Nov 9 15:40 SRV-EXCH01.vmx
37749700 0 -rw------- 1 root root 3184 Nov 8 2018 SRV-EXCH01.vmxf
58721220 8192 -rw------- 1 root root 7700992 Nov 9 14:51 SRV-EXCH01_1-000001-ctk.vmdk
50332612 26355712 -rw------- 1 root root 26987765760 Nov 9 12:32 SRV-EXCH01_1-000001-sesparse.vmdk
100664260 8192 -rw------- 1 root root 7700992 Nov 9 14:06 SRV-EXCH01_1-000002-ctk.vmdk
41944004 8192 -rw------- 1 root root 7700992 Nov 9 15:40 SRV-EXCH01_1-ctk.vmdk
33555396 466540544 -rw------- 1 root root 504658657280 Nov 9 13:06 SRV-EXCH01_1-flat.vmdk
12583876 0 -rw------- 1 root root 518 Nov 9 15:40 SRV-EXCH01_1.vmdk
71304132 64 -rw-r--r-- 1 root root 61922 Nov 9 12:56 vmware-11.log
67109828 64 -rw-r--r-- 1 root root 61922 Nov 9 13:00 vmware-12.log
75498436 1024 -rw-r--r-- 1 root root 271897 Nov 9 13:06 vmware-13.log
104858564 64 -rw------- 1 root root 60897 Nov 9 13:09 vmware-14.log
54526916 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware-15.log
62915524 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware.log

 

Have nice day

Thanks

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Ok, according the output, it seems that the VM is stored on "datastore3_15K" which ran out of disk space due to active snapshots.

I may be able to recreate some missing files, but I need to know exactly what happened, and what has been done so far in trying to fix this issue! It looks like some files have been deleted, and the configuration (.vmx) file has been modified.
Please explain as detailed as possible to avoid (more?) data loss.

André

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Ok I try to explain what happened :

- VM stop with message : There is no more space for virtual ... You might be able to continue this session by freeing ... Choose Retry or Cancel

- We try to move some file in another datastore: srv-exch01_Snapshot1282.vmem ... and the some srv-exch1-000X ( not a good idee..)

after we answer Cancel a the question and the file .vswp has automatically deleted

The VM not restart always, I change the Hard Disk by the file without -00001.vmdk (in attachment before, now).

The VM restart but with old data... > so I shut down the VM.

Unfortunately, I don't have a backup from vmx before. But I share you the version the nearest of the correct config

I remain available if you are missing any information

Thanks you for your help

Guillaume

 

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

You say that you've tried to move some files to another datastore.
Do you still have these files on the other datastore? Which ones (ls -lisa will help).

André

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

We try to move some folder... sorry

But we normally put all the files back under datastore3 (except the snapshot.vsem)
Maybe an oversight is causing the problem?

 

In Datastore1

[root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
total 6170752
4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk

Datastore2 - racine

[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
total 37836480
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk

Datastore2 - racine/old
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/OLD] ls -lisa
total 1152
1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
4197508 0 -rw------- 1 root root 393 Nov 8 10:07 SRV-EXCH01-000001.vmdk
16780420 0 -rw------- 1 root root 1052 Nov 9 13:07 SRV-EXCH01.vmsd
3204 0 -rw------- 1 root root 392 Nov 8 10:05 SRV-EXCH01_1-000001.vmdk

Datastore2 - racine/old2
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
total 41187456
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk

Thanks to your support

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Sorry, I don't know if my last post sent ..

There are some different folders and 2 date (because same problem yesterday but restart OK)

Datastore1 ( copid 8 November not today)

[root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
total 6170752
4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk

Datastore2 (mixed 8 (old copied) or 9 November (last)

[root@localhost:/vmfs/volumes] cd datastore2_SSD/
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
total 37836480
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:48 ..
4194308 8192 -r-------- 1 root root 7847936 Oct 3 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Oct 3 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Oct 3 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Oct 3 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Oct 3 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Oct 3 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Oct 3 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Oct 3 2018 .vh.sf
3332 128 drwxr-xr-x 1 root root 73728 Nov 9 14:06 EXCH0__02
1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 OLD
516 128 drwxr-xr-x 1 root root 73728 Nov 9 15:14 SRV-EXCH01
1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk
580 128 drwxr-xr-x 1 root root 81920 Nov 8 22:51 SRV-TS2016
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 old_2
3908 15360 -rw------- 1 root root 15125822 Nov 3 18:40 vmware-4.log
452 7168 -rw------- 1 root root 6418994 Nov 3 18:41 vmware-5.log
3716 1024 -rw------- 1 root root 706626 Nov 3 18:41 vmware-6.log
2372 1024 -rw------- 1 root root 501686 Nov 3 18:41 vmware-7.log
2308 1024 -rw------- 1 root root 417883 Nov 3 18:41 vmware-8.log

Datastore2 > Old2 (other copy)

[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
total 41187456
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk

Thank you

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

This is like playing a puzzle, but we're getting closer.

Anyway, what I'm missing is the "SRV-EXCH01_1-000002-sesparse.vmdk" file.
Can you please check if it's somewhere in one of the sub-directories?

What I do need to put the remaining pieces together as good as possible are the following files:

  • Datastore1: SRV-EXCH01-000002.vmdk, SRV-EXCH01_1-000002.vmdk
  • Datastore2: SRV-EXCH01.vmx
  • Datastore3: SRV-EXCH01.vmsd

Please attach these files compressed as a .zip archive in to reply post.

André

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Yes a little puzzle..

The file SRV-EXCH01_1-000002-sesparse.vmdk is missing, I didn't find in any datastore...

In attachment the asked files.

Guillaume

 

 

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Ok, with the files we have, here's what we can do (no guarantees though, due to the missing sesparse file)

As the first step, please ensure that your Veeam backup server, or a Veeam proxy (in case Hot-Add backup is used) does not still have the VM's virtual disks attached due to an unsuccessful backup attempt. In case it has, remove them from the Veeam VMs (do NOT delete them from disk!). Also make sure that the backup does not kick in while preforming the following tasks!

  1. create a sub-directory in the VM's folder
  2. move the following files to the sub-folder: *ctk.vmdk, *.vmsd, *.vmsn, *.log
  3. extract the 7 files from the attached .zip archive, and upload then to the VM's folder
  4. register Dummy1.vmx, and Dummy2.vmx as VMs from the Datatore Browser
  5. right-click the Dummy1 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
    --> after the task completes, you should have ~27GB free disk space on the datastore
  6. right-click the Dummy2 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
    --> now all of the 00000x.vmdk files in the VM's folder should be gone (merged into their parent files)
  7. unregister the two Dummy VMs from the inventory (do NOT delete from disk!)
  8. reload the VM from the command line (see https://kb.vmware.com/s/article/1026043)
    vim-cmd vmsvc/getallvms
    vim-cmd vmsvc/reload <vmid>
  9. delete the files Dummy1.vmx, and Dummy2.vmx from the command line
  10. create a VM snapshot, so that the VM's current files won't get modified
  11. power on the VM

If things work as expected, you may then delete the previously created sub-directory, and also the current VM snapshot.
Also remember to delete the files that you've copied to other datastores once you don't need them anymore. Make sure that you don't accidentally delete other files/folder too!

In case you have questions, please feel free to ask.

André.

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

bluescreen.png

 Hi, the VM starts now but with bluescreen. Do you have any suggestion now ? 

Thank you

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Not sure what exactly causes this. Due to the broken snapshot, and the fact that the VM had been powered on with the base disk, there might be some data corruption in the file system.

  • Did you try to start Windows in safe mode? If safe mode works, try to see whether the Windows Event logs help finding the cause.
  • When was your latest successful backup? In case the current virtual disk cannot be repaired, it might be an option to restore the 60GB system disk from the backup.

André

Reply
0 Kudos
Data-mail-2000
Contributor
Contributor
Jump to solution

Hello,

Thank you for your support. All it's OK now.

Have a nice day.

Guillaume

Reply
0 Kudos