Hello,
We are a problem with a snapshot. The VMDk from this snapshot has changed into 2 files -ctk and sesparse. How sync this files to restart VM ?
Have a nice day
Ok, with the files we have, here's what we can do (no guarantees though, due to the missing sesparse file)
As the first step, please ensure that your Veeam backup server, or a Veeam proxy (in case Hot-Add backup is used) does not still have the VM's virtual disks attached due to an unsuccessful backup attempt. In case it has, remove them from the Veeam VMs (do NOT delete them from disk!). Also make sure that the backup does not kick in while preforming the following tasks!
If things work as expected, you may then delete the previously created sub-directory, and also the current VM snapshot.
Also remember to delete the files that you've copied to other datastores once you don't need them anymore. Make sure that you don't accidentally delete other files/folder too!
In case you have questions, please feel free to ask.
André.
>>> ... into 2 files -ctk and sesparse.
Isn't there also a small <vmname>-00000x.vmdk descriptor file for the snapshot in the VM's folder?
Please run ls -lisa from a command line in the VM's folder to get a complete file listing, and provide the text output in your next reply. The GUI "hides" some files.
André
Anyway, it seems that some important files are missing, which sometimes occur if a datastore runs out of disk space.
To see what can be done, please run the following commands in the VM's folder:
Please provide (copy&paste) the the commands' output as plain text rather than a screen shot with a truncated output.
In addition to the commands' output, use e.g. WinSCP, download all of the VM's descriptor .vmdk files (the small ones without flat, sesparse, or ctk in their names), then compress/zip them, and attach the .zip archive to your next reply,.
André
Hello,
Here the logs :
[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] grep vmdk *.vmx
scsi0:0.fileName = "SRV-EXCH01.vmdk"
scsi0:1.fileName = "SRV-EXCH01_1.vmdk"
[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 830.8G 658.2G 172.6G 79% /vmfs/volumes/datastore1_15K
VMFS-6 447.0G 364.6G 82.4G 82% /vmfs/volumes/datastore2_SSD
VMFS-6 558.8G 545.2G 13.6G 98% /vmfs/volumes/datastore3_15K
vfat 285.8M 209.1M 76.8M 73% /vmfs/volumes/5b96778d-46f5c96a-d744-20677cd6dd34
vfat 4.0G 22.2M 4.0G 1% /vmfs/volumes/5b967794-c66f7d18-32ee-20677cd6dd34
vfat 249.7M 183.2M 66.5M 73% /vmfs/volumes/c4424f7d-503e5cff-1dea-9f7ecbbca204
vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/7cfb2376-3f54328b-d3ed-40e98635732b
[root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] ls -lisa
total 570197504
3780 128 drwxr-xr-x 1 root root 86016 Nov 9 15:43 .
4 1024 drwxr-xr-t 1 root root 73728 Nov 9 12:26 ..
96469956 4096 -rw------- 1 root root 3932672 Nov 3 22:15 SRV-EXCH01-000001-ctk.vmdk
46138308 10146816 -rw------- 1 root root 10389823488 Nov 9 12:29 SRV-EXCH01-000001-sesparse.vmdk
83887044 4096 -rw------- 1 root root 3932672 Nov 9 14:06 SRV-EXCH01-000002-ctk.vmdk
25166788 4683776 -rw------- 1 root root 4795998208 Nov 9 14:06 SRV-EXCH01-000002-sesparse.vmdk
88081348 0 -rw-r--r-- 1 root root 53 May 8 2020 SRV-EXCH01-18ae0f4e.hlog
92275652 1024 -rw------- 1 root root 294474 Nov 9 13:35 SRV-EXCH01-Snapshot1283.vmsn
79692740 4096 -rw------- 1 root root 3932672 Nov 9 13:06 SRV-EXCH01-ctk.vmdk
16778180 62429184 -rw------- 1 root root 64424509440 Nov 9 13:06 SRV-EXCH01-flat.vmdk
29361092 1024 -rw------- 1 root root 270840 Nov 9 13:02 SRV-EXCH01.nvram
964 0 -rw------- 1 root root 567 Nov 9 13:02 SRV-EXCH01.vmdk
8389572 0 -rw-r--r-- 1 root root 1052 Nov 9 13:09 SRV-EXCH01.vmsd
20972484 64 -rw-r--r-- 1 root root 3832 Nov 9 15:40 SRV-EXCH01.vmx
37749700 0 -rw------- 1 root root 3184 Nov 8 2018 SRV-EXCH01.vmxf
58721220 8192 -rw------- 1 root root 7700992 Nov 9 14:51 SRV-EXCH01_1-000001-ctk.vmdk
50332612 26355712 -rw------- 1 root root 26987765760 Nov 9 12:32 SRV-EXCH01_1-000001-sesparse.vmdk
100664260 8192 -rw------- 1 root root 7700992 Nov 9 14:06 SRV-EXCH01_1-000002-ctk.vmdk
41944004 8192 -rw------- 1 root root 7700992 Nov 9 15:40 SRV-EXCH01_1-ctk.vmdk
33555396 466540544 -rw------- 1 root root 504658657280 Nov 9 13:06 SRV-EXCH01_1-flat.vmdk
12583876 0 -rw------- 1 root root 518 Nov 9 15:40 SRV-EXCH01_1.vmdk
71304132 64 -rw-r--r-- 1 root root 61922 Nov 9 12:56 vmware-11.log
67109828 64 -rw-r--r-- 1 root root 61922 Nov 9 13:00 vmware-12.log
75498436 1024 -rw-r--r-- 1 root root 271897 Nov 9 13:06 vmware-13.log
104858564 64 -rw------- 1 root root 60897 Nov 9 13:09 vmware-14.log
54526916 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware-15.log
62915524 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware.log
Have nice day
Thanks
Ok, according the output, it seems that the VM is stored on "datastore3_15K" which ran out of disk space due to active snapshots.
I may be able to recreate some missing files, but I need to know exactly what happened, and what has been done so far in trying to fix this issue! It looks like some files have been deleted, and the configuration (.vmx) file has been modified.
Please explain as detailed as possible to avoid (more?) data loss.
André
Ok I try to explain what happened :
- VM stop with message : There is no more space for virtual ... You might be able to continue this session by freeing ... Choose Retry or Cancel
- We try to move some file in another datastore: srv-exch01_Snapshot1282.vmem ... and the some srv-exch1-000X ( not a good idee..)
after we answer Cancel a the question and the file .vswp has automatically deleted
The VM not restart always, I change the Hard Disk by the file without -00001.vmdk (in attachment before, now).
The VM restart but with old data... > so I shut down the VM.
Unfortunately, I don't have a backup from vmx before. But I share you the version the nearest of the correct config
I remain available if you are missing any information
Thanks you for your help
Guillaume
You say that you've tried to move some files to another datastore.
Do you still have these files on the other datastore? Which ones (ls -lisa will help).
André
We try to move some folder... sorry
But we normally put all the files back under datastore3 (except the snapshot.vsem)
Maybe an oversight is causing the problem?
In Datastore1
[root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
total 6170752
4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk
Datastore2 - racine
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
total 37836480
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk
Datastore2 - racine/old
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/OLD] ls -lisa
total 1152
1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
4197508 0 -rw------- 1 root root 393 Nov 8 10:07 SRV-EXCH01-000001.vmdk
16780420 0 -rw------- 1 root root 1052 Nov 9 13:07 SRV-EXCH01.vmsd
3204 0 -rw------- 1 root root 392 Nov 8 10:05 SRV-EXCH01_1-000001.vmdk
Datastore2 - racine/old2
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
total 41187456
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk
Thanks to your support
Sorry, I don't know if my last post sent ..
There are some different folders and 2 date (because same problem yesterday but restart OK)
Datastore1 ( copid 8 November not today)
[root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
total 6170752
4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk
Datastore2 (mixed 8 (old copied) or 9 November (last)
[root@localhost:/vmfs/volumes] cd datastore2_SSD/
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
total 37836480
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:48 ..
4194308 8192 -r-------- 1 root root 7847936 Oct 3 2018 .fbb.sf
8388612 132096 -r-------- 1 root root 134807552 Oct 3 2018 .fdc.sf
33554436 263168 -r-------- 1 root root 268632064 Oct 3 2018 .jbc.sf
25165828 17408 -r-------- 1 root root 16908288 Oct 3 2018 .pb2.sf
12582916 1024 -r-------- 1 root root 65536 Oct 3 2018 .pbc.sf
16777220 1049600 -r-------- 1 root root 1074331648 Oct 3 2018 .sbc.sf
29360132 1024 drwx------ 1 root root 69632 Oct 3 2018 .sdd.sf
20971524 7168 -r-------- 1 root root 7340032 Oct 3 2018 .vh.sf
3332 128 drwxr-xr-x 1 root root 73728 Nov 9 14:06 EXCH0__02
1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 OLD
516 128 drwxr-xr-x 1 root root 73728 Nov 9 15:14 SRV-EXCH01
1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk
580 128 drwxr-xr-x 1 root root 81920 Nov 8 22:51 SRV-TS2016
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 old_2
3908 15360 -rw------- 1 root root 15125822 Nov 3 18:40 vmware-4.log
452 7168 -rw------- 1 root root 6418994 Nov 3 18:41 vmware-5.log
3716 1024 -rw------- 1 root root 706626 Nov 3 18:41 vmware-6.log
2372 1024 -rw------- 1 root root 501686 Nov 3 18:41 vmware-7.log
2308 1024 -rw------- 1 root root 417883 Nov 3 18:41 vmware-8.log
Datastore2 > Old2 (other copy)
[root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
total 41187456
1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk
Thank you
This is like playing a puzzle, but we're getting closer.
Anyway, what I'm missing is the "SRV-EXCH01_1-000002-sesparse.vmdk" file.
Can you please check if it's somewhere in one of the sub-directories?
What I do need to put the remaining pieces together as good as possible are the following files:
Please attach these files compressed as a .zip archive in to reply post.
André
Ok, with the files we have, here's what we can do (no guarantees though, due to the missing sesparse file)
As the first step, please ensure that your Veeam backup server, or a Veeam proxy (in case Hot-Add backup is used) does not still have the VM's virtual disks attached due to an unsuccessful backup attempt. In case it has, remove them from the Veeam VMs (do NOT delete them from disk!). Also make sure that the backup does not kick in while preforming the following tasks!
If things work as expected, you may then delete the previously created sub-directory, and also the current VM snapshot.
Also remember to delete the files that you've copied to other datastores once you don't need them anymore. Make sure that you don't accidentally delete other files/folder too!
In case you have questions, please feel free to ask.
André.
Hi, the VM starts now but with bluescreen. Do you have any suggestion now ?
Thank you
Not sure what exactly causes this. Due to the broken snapshot, and the fact that the VM had been powered on with the base disk, there might be some data corruption in the file system.
André
Hello,
Thank you for your support. All it's OK now.
Have a nice day.
Guillaume