QAQKent
Contributor
Contributor

My vm wont boot after changing the folder of its snapshop vmdk file

HI there

I was runing my vm on  esxi6.7 platform and i had create a snapshop yestoday(so since them the vm is runing on the latest snapshot vmdk files)

and then i decide to move the my snapshot vmdk while the vm is running. after I've done this, I found the orginal snapshot vmdk file has changed from vm1-000006.vmdk to vm1-000006-sesparse.vmdk(the size does not changed),with  a very small file (I belive is 400KB)located in the destination folder,same name as the vm1-000006.vmdk file。i dont know what went wrong so I shutdown the vm and delete that 400KB file,then I try to boot my vm but it won't boot now.

any idear 😨?

 

0 Kudos
14 Replies
a_p_
Leadership
Leadership

Virtual disk consist of two .vmdk files, a descriptor file, and a data file (as you just found out). 
Moving snapshot files will break the VM. In your case what happened is that due to the file lock on the data (sesparse) file, only the descriptor file got moved.
Unless moving the descriptor file back to the VM's folder fixes the issue, it may be necessary to manually do some manual modifications to recover from the issue.

André

0 Kudos
a_p_
Leadership
Leadership

.. sorry, I missed the part that you deleted the descriptor file.

In this case, enable SSH on the ESXi host, and use e.g. WinSCP to connect to it. Then go the the VM's folder (e.g. "vmfs/volumes/<dataastore-name>/<vm-name>"). From there download all the small .vmdk descriptor files that still exist, compress/zip them, and attach the .zip archive to your next reply.

André

0 Kudos
continuum
Immortal
Immortal

> i dont know what went wrong so I shutdown the vm and delete that 400KB file,then I try to boot my vm but it won't boot now.

Strange strategy - if something fails you should rather NOT delete any file that you dont know the purpose of.
Next time first read the vmware.log - it would have explained what is wrong ...

Ulli

 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
QAQKent
Contributor
Contributor

QAQKent_0-1652062419932.png

Thanks for the help!

I followed your instructions and as we can see from the ssh tool the descriptor file from 000001 to 000007 is deleted,I've download all the other remain descriptpor files and upload them down here,hope this would help us solve the problem.

0 Kudos
QAQKent
Contributor
Contributor

That's very correct 

 Hopefully  I could I fix the problem this time.

0 Kudos
QAQKent
Contributor
Contributor

Thanks for the help

I've attached the descriptor files here.

And I've checked the Instruction here https://kb.vmware.com/s/article/1002511?lang=en_us

From the webGUI here is the storage information of this vm

QAQKent_0-1652064457006.png         QAQKent_1-1652064598168.png

 

I found the vm was running on two vmdk files :LIVECLOUD_1_1-000006.vmdk (which  is missing!)and LIVECLOUD_1-000024.vmdk (which is existed )  

so I guess we just need to fix the file  for LIVECLOUD_1_1-000006.vmdk but I do not see its sesparse or decriptor file!

Then I noticed that the Size of these two vmdks  from the web GUI does not show correctly

Then I  typed in the following command

# ls -ltr to check the size
and I get these result

-rw------- 1 root root 2989527040 May 7 15:46 LIVECLOUD_1-000024-sesparse.vmdk

I think I did not delete any vmdk files after I found I get into troble,but the LIVECLOUD_1_1-000006.vmdk is just missing. Not sure what I can do next...😥

 

0 Kudos
QAQKent
Contributor
Contributor

I‘m pretty sure  the 000006_1_1.vmdk file (both sesparse and descritor file) was missing after the crash .I can find out in the vmkernel log(which i will attach down there.

So I'd like to show a whole picture of whats going on now.

The VM is hosted for a NAS use, a centos7.6 system with a Nextcloud NAS apps on it. In case something goes wrong, I create a new virtual DIsk to store my NAS files(I believe its 500GB), and another virtual disk runs the actual OS (40GB I believe) , incase I need expand my NAS space,I create another VG for the NAS virtual Disk.

On 2022/5/6 nghit(UTC+8), before  I'm off my work, I toke a snapshot for the VM (around 22:09 UTC8 and  the VM was still running) ,then I left the company

On 2022/5/7(UTC+8) moning around 10:00 am, I login to the ESXI web surface,and I get a message says "This virtual machine needs to have its disks consolidated. You should consolidate disks to improve performance and optimize disk utilization."

So I started a disk utlization but the I get a failure message said "no  enough space for disk utilization".

Then I decided to free some space for the datastore. I have two datastore, datastore1 and datastore2. the vm is running on datastore1.

What I did is I try to move the vm-000000X.vmdk files to datastore2 (while the vm is still up and running )since I do not know if they are just some useless old snapshop files or not. But due to the file lock, what I moved is just the descriptor files( which I did not aware about and deleted)

Then I  click the reboot button of the vm and found the vm wont boot anymore.

During the time of finding solutions, I tried

1.click scroll back  to the snapshot I created last time --not able to scrollback

2.unregister the vm and re-register  ---not able to boot or change the setting of the vm

3. create a temp.vmdk with same size of the vm-1_000006-sesparse.vmdk and rename the descriptor file to match that vm-1_000006-sesparse.vmdk---still wont boot, and seems like the vm is using vm-1_1_000006.vmdk not vm-1_000006.vmdk.

4. check the remaining files of the vm: ssh to its dir and use ls -l command

and I got following result
total 1076104448
-rw-r--r-- 1 root root 53 Aug 31 2021 LIVECLOUD-a4ffb0ed.hlog
-rw------- 1 root root 8684 Dec 10 09:15 LIVECLOUD.nvram
-rw------- 1 root root 91211956224 May 7 10:12 LIVECLOUD_1-000001-sesparse.vmdk
-rw------- 1 root root 16310702080 May 7 09:02 LIVECLOUD_1-000002-sesparse.vmdk
-rw------- 1 root root 6462459904 May 7 01:57 LIVECLOUD_1-000003-sesparse.vmdk
-rw------- 1 root root 5489471488 May 7 01:57 LIVECLOUD_1-000004-sesparse.vmdk
-rw------- 1 root root 10613239808 May 7 01:57 LIVECLOUD_1-000005-sesparse.vmdk
-rw------- 1 root root 3937927168 May 7 01:57 LIVECLOUD_1-000006-sesparse.vmdk
-rw------- 1 root root 460 May 9 03:37 LIVECLOUD_1-000006.vmdk
-rw------- 1 root root 13089460224 May 7 01:57 LIVECLOUD_1-000007-sesparse.vmdk
-rw------- 1 root root 2536579072 May 7 01:57 LIVECLOUD_1-000008-sesparse.vmdk
-rw------- 1 root root 327 Jul 27 2021 LIVECLOUD_1-000008.vmdk
-rw------- 1 root root 60904456192 May 7 01:57 LIVECLOUD_1-000009-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 08:35 LIVECLOUD_1-000009.vmdk
-rw------- 1 root root 2848980992 May 7 01:57 LIVECLOUD_1-000010-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 04:49 LIVECLOUD_1-000010.vmdk
-rw------- 1 root root 3301187584 May 7 01:57 LIVECLOUD_1-000011-sesparse.vmdk
-rw------- 1 root root 327 Dec 9 09:23 LIVECLOUD_1-000011.vmdk
-rw------- 1 root root 3176865792 May 7 01:57 LIVECLOUD_1-000012-sesparse.vmdk
-rw------- 1 root root 327 Dec 9 09:23 LIVECLOUD_1-000012.vmdk
-rw------- 1 root root 5472587776 May 7 01:57 LIVECLOUD_1-000013-sesparse.vmdk
-rw------- 1 root root 327 Jul 28 2021 LIVECLOUD_1-000013.vmdk
-rw------- 1 root root 24581804032 May 7 01:57 LIVECLOUD_1-000014-sesparse.vmdk
-rw------- 1 root root 327 Sep 6 2021 LIVECLOUD_1-000014.vmdk
-rw------- 1 root root 206547492864 May 7 01:57 LIVECLOUD_1-000015-sesparse.vmdk
-rw------- 1 root root 327 Nov 24 02:58 LIVECLOUD_1-000015.vmdk
-rw------- 1 root root 3814195200 May 7 01:57 LIVECLOUD_1-000016-sesparse.vmdk
-rw------- 1 root root 327 Nov 25 03:47 LIVECLOUD_1-000016.vmdk
-rw------- 1 root root 294809133056 May 7 05:37 LIVECLOUD_1-000017-sesparse.vmdk
-rw------- 1 root root 327 Dec 3 05:27 LIVECLOUD_1-000017.vmdk
-rw------- 1 root root 222418702336 May 7 01:57 LIVECLOUD_1-000018-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 05:43 LIVECLOUD_1-000018.vmdk
-rw------- 1 root root 28023193600 May 7 01:57 LIVECLOUD_1-000019-sesparse.vmdk
-rw------- 1 root root 327 Nov 25 06:29 LIVECLOUD_1-000019.vmdk
-rw------- 1 root root 7279529984 May 7 01:57 LIVECLOUD_1-000020-sesparse.vmdk
-rw------- 1 root root 327 Dec 6 05:38 LIVECLOUD_1-000020.vmdk
-rw------- 1 root root 2508328960 May 7 01:57 LIVECLOUD_1-000021-sesparse.vmdk
-rw------- 1 root root 327 Dec 8 06:30 LIVECLOUD_1-000021.vmdk
-rw------- 1 root root 4466417664 May 7 01:57 LIVECLOUD_1-000022-sesparse.vmdk
-rw------- 1 root root 327 Dec 7 06:29 LIVECLOUD_1-000022.vmdk
-rw------- 1 root root 2436382720 May 7 01:57 LIVECLOUD_1-000023-sesparse.vmdk
-rw------- 1 root root 327 Dec 21 02:54 LIVECLOUD_1-000023.vmdk
-rw------- 1 root root 2989527040 May 7 15:46 LIVECLOUD_1-000024-sesparse.vmdk
-rw------- 1 root root 327 May 7 01:57 LIVECLOUD_1-000024.vmdk
-rw------- 1 root root 30356422656 May 7 01:57 LIVECLOUD_1-000025-sesparse.vmdk
-rw------- 1 root root 327 May 6 14:09 LIVECLOUD_1-000025.vmdk
-rw------- 1 root root 8589934592 Dec 9 2020 LIVECLOUD_1-Snapshot1.vmem
-rw------- 1 root root 6012243 Dec 9 2020 LIVECLOUD_1-Snapshot1.vmsn
-rw------- 1 root root 8589934592 Jun 7 2021 LIVECLOUD_1-Snapshot2.vmem
-rw------- 1 root root 5805307 Jun 7 2021 LIVECLOUD_1-Snapshot2.vmsn
-rw------- 1 root root 21203 May 6 14:10 LIVECLOUD_1-Snapshot32.vmsn
-rw------- 1 root root 536870912000 Dec 9 2020 LIVECLOUD_1-flat.vmdk
-rw------- 1 root root 534 Nov 10 2020 LIVECLOUD_1.vmdk
-rw-r--r-- 1 root root 1237 May 7 11:11 LIVECLOUD_1.vmsd
-rwx------ 1 root root 4172 May 7 14:37 LIVECLOUD_1.vmx
-rw------- 1 root root 42949672960 May 6 16:10 LIVECLOUD_1_1-flat.vmdk
-rw------- 1 root root 453 May 6 16:10 LIVECLOUD_1_1.vmdk
-rw------- 1 root root 43008 Oct 30 2020 _deviceImage-0.iso
-rw------- 1 root root 3937927168 May 9 03:27 temp-flat.vmdk
-rw------- 1 root root 568286 May 6 13:54 vmware-48.log
-rw------- 1 root root 744707 May 7 02:08 vmware-49.log
-rw------- 1 root root 316428 May 7 02:11 vmware-50.log
-rw------- 1 root root 318896 May 7 02:43 vmware-51.log
-rw------- 1 root root 104398 May 9 03:37 vmware-52.log
-rw------- 1 root root 104312 May 9 03:41 vmware-53.log
-rw------- 1 root root 105776 May 9 04:49 vmware.log
-rw------- 1 root root 98566144 Mar 29 2021 vmx-LIVECLOUD_1-3569746826-1.vswp

The vmdks In color shows the same modified time as I create the last snapshot

when I try try to boot the vm I get these error message

State

Failed - File LIVECLOUD_1_1-000006.vmdk was not found

Errors

 

  • VMware ESX cannot find the virtual disk "LIVECLOUD_1_1-000006.vmdk". Verify the path is valid and try again.
  • File system specific implementation of MakeOID[file] failed
  • File system specific implementation of MakeOID[file] failed
  • The system cannot find the file specified
  • Cannot open the disk 'LIVECLOUD_1_1-000006.vmdk' or one of the snapshot disks it depends on.
  • Module 'Disk' power on failed.
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • File system specific implementation of Ioctl[file] failed
  • The capacity of the parent virtual disk and the capacity of the child disk are different
  • Cannot open the disk '/vmfs/volumes/5f94679f-d5c0cb50-0f19-0894efaf4258/LIVECLOUD_1/LIVECLOUD_1-000024.vmdk' or one of the snapshot disks it depends on.
  • Failed to start the virtual machine.

Im assuming that I deleted the 1_1_000006.vmdk by accident😥, is it possible to make the vm runs from  the very last snapshot? 

not sure if these will help.

best reguards.

0 Kudos
continuum
Immortal
Immortal

Hi
the timestamps of your files are very close to each other - that either means that you copied them manually - or that they were created by an automatic backup-tool which used a very short time for "retry after failure".
Second option would suggest that skipping some of them would do no harm.

Another problem is the "File system specific implementation of Ioctl[file] failed"
Anyway - I think that I can bring your VM back to live but it would be way more reliable if I could do that with direct access via Teamviewer.
If that is an option for you - contact me via skype.
I expect that you have WinSCP and putty installed on your admin host.

Ulli

 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
a_p_
Leadership
Leadership

If we are lucky, the contents of the vmware*.log files in the VM's folder might help with re-chaining at least one of the 2 virtual disks.
Please download these files from the VM's folder and attach them (compressed as a .zip archive) to your next reply.

André

 

0 Kudos
continuum
Immortal
Immortal

Andre - this VM was tortured by renaming innocent files ...
The first vmdk may already work again - but it took ages to boot.
The second vmdk is missing several snapshots - we may need to fall back to the basedisk.
Poor VM ...

Ulli

 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
QAQKent
Contributor
Contributor

Okay maybe these logs can do some help

0 Kudos
continuum
Immortal
Immortal

This logs are old - show the vmware.log from the start I launched yesterday.

Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
QAQKent
Contributor
Contributor

there's no further logs for the VM
here's the vmkernel log may show what went wrong

0 Kudos
QAQKent
Contributor
Contributor

I managed to figure out how to get my data back here's the way I checked the vmkernel log and find out what the right Pid and Cid is , since i have all the seSpare files of the datastore disk remained I and ready to go. But I got no luck with the OS partition, it's broken. So I tried to migrade another mv's OS vmdks to this VM but it seems conflit with the old LV and VG settings. What I do next is load a Ubuntu iso disk and mount the old VG and LV then I got my files and folders back! I‘m now copy these data to somewhere safe and hopefully get my nas running again very shortly. CHEERS!
0 Kudos