Hello forum,
i had bad luck trying to move a VM on an old ESXi 5.5 to a NFS Storage to consolidate the VMs to another server.
While i was working on the cables i accidently powered off the VM host. The "move" process was disrupted. I thought no problem - the files remain on the datasore1 as long as the file was not copied by 100%. Seems it got removed from datastore even when it was not moved by 100% to the NFS storage.
Of course i would not write here if i had a backup... shame on me.
Well i did some repair attempt on my own but it did not work out well.
What i have is a 21 GB flat.vmdk file which i copied back from NFS storage to the VMware host. The VM had originally a 250 GB provisioned harddisk. Now vSphere tells me it is thick provisioned eager zeroed.
Before i tried to move the vm folder i made a snapshot but it seems on try and error i removed the snapshot in the vSphere client - tho i still have 3 delta files, one which is quiet old and big with 54 GB in size. So maybe there is a chance to get a flat file restored out of all 3 delta files? If so, what are the steps?
Thanks in advance.
Regards.
//mga
This will help you to resolve the issue.
To get an overview, please post a complete list of files in the VM's folder, i.e. the output of ls -lisa from the command line.
If there are still some vmware.log files from the time when the VM was still healthy, please attach them to a reply post.
André
Hello a.p. ,
here is the requested output:
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # ls -lisa
436244036 8 drwxr-xr-x 1 root root 3080 Mar 10 12:59 .
4 1024 drwxr-xr-t 1 root root 2380 Mar 10 12:59 ..
574656068 54838272 -rw------- 1 root root 56153858048 Mar 7 18:50 Nuvola Cloud VPS-000001-delta.vmdk
578850372 0 -rw------- 1 root root 336 Jul 13 2015 Nuvola Cloud VPS-000001.vmdk
427855364 17408 -rw------- 1 root root 17293312 Mar 7 19:07 Nuvola Cloud VPS-000002-delta.vmdk
432049668 0 -rw------- 1 root root 343 Mar 7 18:50 Nuvola Cloud VPS-000002.vmdk
738233860 1024 -rw------- 1 root root 516096 Mar 8 10:44 Nuvola Cloud VPS-000003-delta.vmdk
281054724 0 -rw------- 1 root root 343 Mar 8 14:28 Nuvola Cloud VPS-000003.vmdk
192974340 0 -rw-r--r-- 1 root root 13 Mar 8 11:08 Nuvola Cloud VPS-aux.xml
683707908 21499904 -rw------- 1 root root 22015639552 Mar 8 10:23 Nuvola Cloud VPS-flat.vmdk
708873732 1024 -rw------- 1 root root 8684 Mar 9 14:44 Nuvola Cloud VPS.nvram
570461700 0 -rw------- 1 root root 504 Mar 8 15:45 Nuvola Cloud VPS.vmdk
453021252 0 -rw-r--r-- 1 root root 43 Mar 8 14:34 Nuvola Cloud VPS.vmsd
717262340 8 -rwx------ 1 root root 2944 Mar 9 14:44 Nuvola Cloud VPS.vmx
713068036 0 -rw------- 1 root root 271 Mar 9 14:19 Nuvola Cloud VPS.vmxf
687902212 1024 -rw-r--r-- 1 root root 140069 Mar 8 15:04 vmware-41.log
704679428 1024 -rw-r--r-- 1 root root 143411 Mar 8 15:13 vmware-42.log
796954116 1024 -rw-r--r-- 1 root root 35739 Mar 8 15:22 vmware-43.log
176197124 1024 -rw-r--r-- 1 root root 95256 Mar 8 16:29 vmware-44.log
201362948 1024 -rw-r--r-- 1 root root 142999 Mar 9 09:31 vmware-45.log
251694596 1024 -rw-r--r-- 1 root root 142282 Mar 9 14:19 vmware-46.log
268471812 1024 -rw-r--r-- 1 root root 142677 Mar 9 14:44 vmware.log
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS #
Unfortunately the logs have been overwritten with all of my own attempts to repair / restore the disk descriptor and flat file while firing up the VM again. My bad - should have made a fulll backup before playing around. *sigh*
Regards,
//mga
Thank you Buddhlka01 and Himadri.
I tried that but the VM was not booting. Tho as it was controller type pvscsi originally, i will try now to do it as lsilogic controller type.
Well, was not booting is not correct at all - the VM did boot but the Linux bootloader Grub only reported "grub-rescue: error: out of disk." and it seems the MBR and partition table and so on seems missing - which is no wonder as the flat file i copied back from NFS storage is broken.
My hope is that with the delta files some data can be recovered even if the whole disk can't be restored.
Regards,
//mga
Please download, and compress/zip the following files, then attach the .zip archive to a reply post.
578850372 0 -rw------- 1 root root 336 Jul 13 2015 Nuvola Cloud VPS-000001.vmdk
432049668 0 -rw------- 1 root root 343 Mar 7 18:50 Nuvola Cloud VPS-000002.vmdk
281054724 0 -rw------- 1 root root 343 Mar 8 14:28 Nuvola Cloud VPS-000003.vmdk
570461700 0 -rw------- 1 root root 504 Mar 8 15:45 Nuvola Cloud VPS.vmdk
717262340 8 -rwx------ 1 root root 2944 Mar 9 14:44 Nuvola Cloud VPS.vmx
Hint: You may use e.g. WinSCP to download these files from the datasatore.
Am I right in assuming that you've only created/modified the "Nuvola Cloud VPS.vmdk" so far?
André
Hi a.p. ,
you are right, i recreated Nuvola Cloud VPS.vmdk as that file was missing along with the .flat file.
But guess what. I only used the vSphere datastore browser to retain the .flat file from the NFS Storage. Now i found via WinSCP there ist the original "Nuvola Cloud VPS.vmdk" left on the NFS folder. Thats the content:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=7b450b10
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 524288000 VMFS "Nuvola Cloud VPS-flat.vmdk"
# The Disk Data Base
#DDB
ddb.virtualHWVersion = "8"
ddb.longContentID = "80a8fb400cbdcb5686a385397b450b10"
ddb.uuid = "60 00 C2 92 60 a3 ff bb-c9 c0 78 31 87 e0 6b 77"
ddb.geometry.cylinders = "32635"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"
ddb.toolsVersion = "9221"
Maybe that original disk descriptor helps us.
I attached the requested files in a zip file.
Thank you sofar for the support.
Regards,
//mga
What should work is to copy the original .vmdk file over to the VM's folder (i.e. overwrite the manually created one), and also edit the configuration (.vmx) file!!!
It looks like you've tried to attach the virtual disk from the GUI, which added the base disk instead of the current snapshot.
Please change the virtual disk's file name in the .vmx file to:
scsi0:0.fileName = "Nuvola Cloud VPS-000003.vmdk"
then reload the VM, to make the host aware of the change. Follow steps 2 and 3 from https://kb.vmware.com/kb/1026043
Once done, I'd recommend that you create a new, temporary snapshot before you power on the VM. This way the current .vmdk files won't get modified, and you'll have a way back in case it's necessary.
André
Okay, i copied the original saved .vmdk into the vm folder and did overwrite the one i created.
I edited the vmx and change the file nam,e to the 00003.vmdk and reloaded the vm using vim-cmd.
THen i created an offline snapshot and fired up the vm after that.
Now i get the error message:
"Cannot open the disk '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004.vmdk' or one of the snapshot disks it depends on.
Failed to lock the file"
I checked wether the disk chain is consitent and get:
"/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # vmkfstools -e Nuvola\ Cloud\ VPS.vmdk
Failed to open disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS.vmdk :The file specified is not a virtual disk (15)Disk chain is not consistent : The file specified is not a virtual disk (15)
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # "
I did a test on the snapshot 4 file too and get:
"/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # vmkfstools -e Nuvola\ Cloud\ VPS-000004.vmdk
Disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004.vmdk successfully opened.
Disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000003.vmdk successfully opened.
Disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000002.vmdk successfully opened.
Failed to open disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk :Failed to lock the file (16392)Disk chain is not consistent : The parent of this virtual disk could not be opened (23)
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # "
So, i guess the whole chain backwards is fine - just the reference from 000001.vmdk to the broken/truncated flat file is the issue - right?
Regards,
//mga
I'm afraid you're right regarding the flat file corruption.
Rather than trying other things which may make things worse, try to contact continuum. He's a specialist when it comes to data recovery. Maybe he can extract missing data from the VMFS volume.
André
Thank you André.
I will contact continuum.
Maybe another hint:
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # vmkfstools -e -v10 Nuvola\ Cloud\ VPS-000001.vmdk
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 0)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 1)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 2)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 3)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 4)
OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' : Failed to lock the file (262146).
DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk': Failed to lock the file (40002)
DISKLIB-LINK : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk" : failed to open (Failed to lock the file).
DISKLIB-CHAIN : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk" : failed to open (Failed to lock the file).
DISKLIB-LIB : Failed to open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' with flags 0xe Failed to lock the file (16392).
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 0)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 1)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 2)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 3)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' locked (try 4)
OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' : Failed to lock the file (262146).
DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk': Failed to lock the file (40002)
DISKLIB-LINK : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk" : failed to open (Failed to lock the file).
DISKLIB-CHAIN : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk" : failed to open (Failed to lock the file).
DISKLIB-LIB : Failed to open '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' with flags 0xc Failed to lock the file (16392).
Failed to open disk link /vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk :Failed to lock the file (16392)Disk chain is not consistent : Failed to lock the file (16392)
AIOMGR-S : stat o=12 r=0 w=0 i=0 br=0 bw=0
OBJLIB-LIB : ObjLib cleanup done.
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # vmkfstools -D Nuvola\ Cloud\ VPS-000001.vmdk
Lock [type 10c00001 offset 249729024 v 256, hb offset 3321856
gen 73, mode 1, owner 5c81754f-08af6cee-e741-0015c5f16c2c mtime 19422
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 569, 138>, gen 215, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 336, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS #
From what i understand is that a process is locking the 1st delta file so the chain is inconsitent. How can i fix that to try a bit more?
Regards,
//mga
Ha!
After a reboot of the server the chain is now clean/consisten - tho the VM boots but the bootloader is not able to start the Linux system as it is missing files.
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS # vmkfstools -e -v10 Nuvola\ Cloud\ VPS-000004.vmdk
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004-delta.vmdk" : open successful (14) size = 516096, hd = 27104. Type 8
DISKLIB-DSCPTR: Opened [0]: "Nuvola Cloud VPS-000004-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004.vmdk' (0xe): vmfsSparse, 524288000 sectors / 250 GB.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000003-delta.vmdk" : open successful (14) size = 516096, hd = 27105. Type 8
DISKLIB-DSCPTR: Opened [0]: "Nuvola Cloud VPS-000003-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000003.vmdk' (0xe): vmfsSparse, 524288000 sectors / 250 GB.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000002-delta.vmdk" : open successful (14) size = 17293312, hd = 27106. Type 8
DISKLIB-DSCPTR: Opened [0]: "Nuvola Cloud VPS-000002-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000002.vmdk' (0xe): vmfsSparse, 524288000 sectors / 250 GB.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001-delta.vmdk" : open successful (14) size = 56153858048, hd = 27107. Type 8
DISKLIB-DSCPTR: Opened [0]: "Nuvola Cloud VPS-000001-delta.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001.vmdk' (0xe): vmfsSparse, 524288000 sectors / 250 GB.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-flat.vmdk" : open successful (14) size = 268435456000, hd = 27108. Type 3
DISKLIB-DSCPTR: Opened [0]: "Nuvola Cloud VPS-flat.vmdk" (0xe)
DISKLIB-LINK : Opened '/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS.vmdk' (0xe): vmfs, 524288000 sectors / 250 GB.
DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 5, numSubChains = 1
DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 27108, extentType = 2
DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 27107, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChain:(2) fid = 27106, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChain:(3) fid = 27105, extentType = 0
DISKLIB-CHAINESX : ChainESXOpenSubChain:(4) fid = 27104, extentType = 0
DISKLIB-LIB : Opened "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004.vmdk" (flags 0xe, type vmfs).
Disk chain is consistent.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000004-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000003-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000002-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS/Nuvola Cloud VPS-flat.vmdk" : closed.
AIOMGR-S : stat o=5 r=15 w=0 i=0 br=245760 bw=0
AIOMGR-U : stat o=6 r=4 w=0 i=11 br=8192 bw=0
OBJLIB-LIB : ObjLib cleanup done.
/vmfs/volumes/53daa82a-ce8997b8-ddd9-0015c5f16c2c/Nuvola_Cloud_VPS #
But it may be a 1st step towards getting some data restored somehow.
Any further ideas?
Regards,
//mga
Since the base/flat disk seems to be incomplete (250GB provisioned, ~21GB remaining), I'd think the best you can do is to boot the VM from a Live Linux ISO image, and try to extract/backup important data. Attempts to reinstall, or fix the guest OS may overwrite data. However, since you've created a new snapshot, you can basically try different things, and revert to that snapshot if needed.
André
Currently i have created a new flat file with the exact 250 GB and placed it into the delta chain. I was able to boot with a live linux iso and currently a sh*t ton of unattached inodes are processed by fsk and i assume tons of files are going to be dumped into the lost and found folder. Ah, yes - prior testdisk tool from GPartes Live iso repaired the partition table and other stuff.
Since i run here with an empty / blank flat file and the deltas, i expect to miss a lot of files.
I wonder if i should run the same procedurte with that "rescued" 21 GB truncated flat file. Not sure if i can do something like linux dd to copy the content of my 21 GB file into/over that newly created 250 Gb flat file. Would that work?
Regards,
//mga