HI there
I was runing my vm on esxi6.7 platform and i had create a snapshop yestoday(so since them the vm is runing on the latest snapshot vmdk files)
and then i decide to move the my snapshot vmdk while the vm is running. after I've done this, I found the orginal snapshot vmdk file has changed from vm1-000006.vmdk to vm1-000006-sesparse.vmdk(the size does not changed),with a very small file (I belive is 400KB)located in the destination folder,same name as the vm1-000006.vmdk file。i dont know what went wrong so I shutdown the vm and delete that 400KB file,then I try to boot my vm but it won't boot now.
any idear 😨?
Virtual disk consist of two .vmdk files, a descriptor file, and a data file (as you just found out).
Moving snapshot files will break the VM. In your case what happened is that due to the file lock on the data (sesparse) file, only the descriptor file got moved.
Unless moving the descriptor file back to the VM's folder fixes the issue, it may be necessary to manually do some manual modifications to recover from the issue.
André
.. sorry, I missed the part that you deleted the descriptor file.
In this case, enable SSH on the ESXi host, and use e.g. WinSCP to connect to it. Then go the the VM's folder (e.g. "vmfs/volumes/<dataastore-name>/<vm-name>"). From there download all the small .vmdk descriptor files that still exist, compress/zip them, and attach the .zip archive to your next reply.
André
> i dont know what went wrong so I shutdown the vm and delete that 400KB file,then I try to boot my vm but it won't boot now.
Strange strategy - if something fails you should rather NOT delete any file that you dont know the purpose of.
Next time first read the vmware.log - it would have explained what is wrong ...
Ulli
That's very correct
Hopefully I could I fix the problem this time.
Thanks for the help
I've attached the descriptor files here.
And I've checked the Instruction here https://kb.vmware.com/s/article/1002511?lang=en_us
From the webGUI here is the storage information of this vm
I found the vm was running on two vmdk files :LIVECLOUD_1_1-000006.vmdk (which is missing!)and LIVECLOUD_1-000024.vmdk (which is existed )
so I guess we just need to fix the file for LIVECLOUD_1_1-000006.vmdk but I do not see its sesparse or decriptor file!
Then I noticed that the Size of these two vmdks from the web GUI does not show correctly
Then I typed in the following command
# ls -ltr to check the size
and I get these result
-rw------- 1 root root 2989527040 May 7 15:46 LIVECLOUD_1-000024-sesparse.vmdk
I think I did not delete any vmdk files after I found I get into troble,but the LIVECLOUD_1_1-000006.vmdk is just missing. Not sure what I can do next...😥
I‘m pretty sure the 000006_1_1.vmdk file (both sesparse and descritor file) was missing after the crash .I can find out in the vmkernel log(which i will attach down there.
So I'd like to show a whole picture of whats going on now.
The VM is hosted for a NAS use, a centos7.6 system with a Nextcloud NAS apps on it. In case something goes wrong, I create a new virtual DIsk to store my NAS files(I believe its 500GB), and another virtual disk runs the actual OS (40GB I believe) , incase I need expand my NAS space,I create another VG for the NAS virtual Disk.
On 2022/5/6 nghit(UTC+8), before I'm off my work, I toke a snapshot for the VM (around 22:09 UTC8 and the VM was still running) ,then I left the company
On 2022/5/7(UTC+8) moning around 10:00 am, I login to the ESXI web surface,and I get a message says "This virtual machine needs to have its disks consolidated. You should consolidate disks to improve performance and optimize disk utilization."
So I started a disk utlization but the I get a failure message said "no enough space for disk utilization".
Then I decided to free some space for the datastore. I have two datastore, datastore1 and datastore2. the vm is running on datastore1.
What I did is I try to move the vm-000000X.vmdk files to datastore2 (while the vm is still up and running )since I do not know if they are just some useless old snapshop files or not. But due to the file lock, what I moved is just the descriptor files( which I did not aware about and deleted)
Then I click the reboot button of the vm and found the vm wont boot anymore.
During the time of finding solutions, I tried
1.click scroll back to the snapshot I created last time --not able to scrollback
2.unregister the vm and re-register ---not able to boot or change the setting of the vm
3. create a temp.vmdk with same size of the vm-1_000006-sesparse.vmdk and rename the descriptor file to match that vm-1_000006-sesparse.vmdk---still wont boot, and seems like the vm is using vm-1_1_000006.vmdk not vm-1_000006.vmdk.
4. check the remaining files of the vm: ssh to its dir and use ls -l command
and I got following result
total 1076104448
-rw-r--r-- 1 root root 53 Aug 31 2021 LIVECLOUD-a4ffb0ed.hlog
-rw------- 1 root root 8684 Dec 10 09:15 LIVECLOUD.nvram
-rw------- 1 root root 91211956224 May 7 10:12 LIVECLOUD_1-000001-sesparse.vmdk
-rw------- 1 root root 16310702080 May 7 09:02 LIVECLOUD_1-000002-sesparse.vmdk
-rw------- 1 root root 6462459904 May 7 01:57 LIVECLOUD_1-000003-sesparse.vmdk
-rw------- 1 root root 5489471488 May 7 01:57 LIVECLOUD_1-000004-sesparse.vmdk
-rw------- 1 root root 10613239808 May 7 01:57 LIVECLOUD_1-000005-sesparse.vmdk
-rw------- 1 root root 3937927168 May 7 01:57 LIVECLOUD_1-000006-sesparse.vmdk
-rw------- 1 root root 460 May 9 03:37 LIVECLOUD_1-000006.vmdk
-rw------- 1 root root 13089460224 May 7 01:57 LIVECLOUD_1-000007-sesparse.vmdk
-rw------- 1 root root 2536579072 May 7 01:57 LIVECLOUD_1-000008-sesparse.vmdk
-rw------- 1 root root 327 Jul 27 2021 LIVECLOUD_1-000008.vmdk
-rw------- 1 root root 60904456192 May 7 01:57 LIVECLOUD_1-000009-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 08:35 LIVECLOUD_1-000009.vmdk
-rw------- 1 root root 2848980992 May 7 01:57 LIVECLOUD_1-000010-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 04:49 LIVECLOUD_1-000010.vmdk
-rw------- 1 root root 3301187584 May 7 01:57 LIVECLOUD_1-000011-sesparse.vmdk
-rw------- 1 root root 327 Dec 9 09:23 LIVECLOUD_1-000011.vmdk
-rw------- 1 root root 3176865792 May 7 01:57 LIVECLOUD_1-000012-sesparse.vmdk
-rw------- 1 root root 327 Dec 9 09:23 LIVECLOUD_1-000012.vmdk
-rw------- 1 root root 5472587776 May 7 01:57 LIVECLOUD_1-000013-sesparse.vmdk
-rw------- 1 root root 327 Jul 28 2021 LIVECLOUD_1-000013.vmdk
-rw------- 1 root root 24581804032 May 7 01:57 LIVECLOUD_1-000014-sesparse.vmdk
-rw------- 1 root root 327 Sep 6 2021 LIVECLOUD_1-000014.vmdk
-rw------- 1 root root 206547492864 May 7 01:57 LIVECLOUD_1-000015-sesparse.vmdk
-rw------- 1 root root 327 Nov 24 02:58 LIVECLOUD_1-000015.vmdk
-rw------- 1 root root 3814195200 May 7 01:57 LIVECLOUD_1-000016-sesparse.vmdk
-rw------- 1 root root 327 Nov 25 03:47 LIVECLOUD_1-000016.vmdk
-rw------- 1 root root 294809133056 May 7 05:37 LIVECLOUD_1-000017-sesparse.vmdk
-rw------- 1 root root 327 Dec 3 05:27 LIVECLOUD_1-000017.vmdk
-rw------- 1 root root 222418702336 May 7 01:57 LIVECLOUD_1-000018-sesparse.vmdk
-rw------- 1 root root 327 Dec 10 05:43 LIVECLOUD_1-000018.vmdk
-rw------- 1 root root 28023193600 May 7 01:57 LIVECLOUD_1-000019-sesparse.vmdk
-rw------- 1 root root 327 Nov 25 06:29 LIVECLOUD_1-000019.vmdk
-rw------- 1 root root 7279529984 May 7 01:57 LIVECLOUD_1-000020-sesparse.vmdk
-rw------- 1 root root 327 Dec 6 05:38 LIVECLOUD_1-000020.vmdk
-rw------- 1 root root 2508328960 May 7 01:57 LIVECLOUD_1-000021-sesparse.vmdk
-rw------- 1 root root 327 Dec 8 06:30 LIVECLOUD_1-000021.vmdk
-rw------- 1 root root 4466417664 May 7 01:57 LIVECLOUD_1-000022-sesparse.vmdk
-rw------- 1 root root 327 Dec 7 06:29 LIVECLOUD_1-000022.vmdk
-rw------- 1 root root 2436382720 May 7 01:57 LIVECLOUD_1-000023-sesparse.vmdk
-rw------- 1 root root 327 Dec 21 02:54 LIVECLOUD_1-000023.vmdk
-rw------- 1 root root 2989527040 May 7 15:46 LIVECLOUD_1-000024-sesparse.vmdk
-rw------- 1 root root 327 May 7 01:57 LIVECLOUD_1-000024.vmdk
-rw------- 1 root root 30356422656 May 7 01:57 LIVECLOUD_1-000025-sesparse.vmdk
-rw------- 1 root root 327 May 6 14:09 LIVECLOUD_1-000025.vmdk
-rw------- 1 root root 8589934592 Dec 9 2020 LIVECLOUD_1-Snapshot1.vmem
-rw------- 1 root root 6012243 Dec 9 2020 LIVECLOUD_1-Snapshot1.vmsn
-rw------- 1 root root 8589934592 Jun 7 2021 LIVECLOUD_1-Snapshot2.vmem
-rw------- 1 root root 5805307 Jun 7 2021 LIVECLOUD_1-Snapshot2.vmsn
-rw------- 1 root root 21203 May 6 14:10 LIVECLOUD_1-Snapshot32.vmsn
-rw------- 1 root root 536870912000 Dec 9 2020 LIVECLOUD_1-flat.vmdk
-rw------- 1 root root 534 Nov 10 2020 LIVECLOUD_1.vmdk
-rw-r--r-- 1 root root 1237 May 7 11:11 LIVECLOUD_1.vmsd
-rwx------ 1 root root 4172 May 7 14:37 LIVECLOUD_1.vmx
-rw------- 1 root root 42949672960 May 6 16:10 LIVECLOUD_1_1-flat.vmdk
-rw------- 1 root root 453 May 6 16:10 LIVECLOUD_1_1.vmdk
-rw------- 1 root root 43008 Oct 30 2020 _deviceImage-0.iso
-rw------- 1 root root 3937927168 May 9 03:27 temp-flat.vmdk
-rw------- 1 root root 568286 May 6 13:54 vmware-48.log
-rw------- 1 root root 744707 May 7 02:08 vmware-49.log
-rw------- 1 root root 316428 May 7 02:11 vmware-50.log
-rw------- 1 root root 318896 May 7 02:43 vmware-51.log
-rw------- 1 root root 104398 May 9 03:37 vmware-52.log
-rw------- 1 root root 104312 May 9 03:41 vmware-53.log
-rw------- 1 root root 105776 May 9 04:49 vmware.log
-rw------- 1 root root 98566144 Mar 29 2021 vmx-LIVECLOUD_1-3569746826-1.vswp
The vmdks In color shows the same modified time as I create the last snapshot
when I try try to boot the vm I get these error message
Failed - File LIVECLOUD_1_1-000006.vmdk was not found
Im assuming that I deleted the 1_1_000006.vmdk by accident😥, is it possible to make the vm runs from the very last snapshot?
not sure if these will help.
best reguards.
Hi
the timestamps of your files are very close to each other - that either means that you copied them manually - or that they were created by an automatic backup-tool which used a very short time for "retry after failure".
Second option would suggest that skipping some of them would do no harm.
Another problem is the "File system specific implementation of Ioctl[file] failed"
Anyway - I think that I can bring your VM back to live but it would be way more reliable if I could do that with direct access via Teamviewer.
If that is an option for you - contact me via skype.
I expect that you have WinSCP and putty installed on your admin host.
Ulli
If we are lucky, the contents of the vmware*.log files in the VM's folder might help with re-chaining at least one of the 2 virtual disks.
Please download these files from the VM's folder and attach them (compressed as a .zip archive) to your next reply.
André
Andre - this VM was tortured by renaming innocent files ...
The first vmdk may already work again - but it took ages to boot.
The second vmdk is missing several snapshots - we may need to fall back to the basedisk.
Poor VM ...
Ulli
This logs are old - show the vmware.log from the start I launched yesterday.
Ulli