VM disappeared after patching to ESXi 8.0b and rebooting

[evanc4]

Hello, I have a very strange issue. One of my VMs completely vanished after patching to ESXi 8.0b and rebooting the server.  It's vmdk and vmx files along with any logs are all gone.  It is only affected one VM.  All other VMs on the same datastore are working fine still.

If I try to browse to its folder in the datastore browser, it say an error occurred please try again. It won't let me delete the folder. In the DCUI shell, ls -la shows the VM folder is empty but it won't let me delete it there either, it says invalid argument.

I have a backup but I cannot restore it either since it says insufficient space. But I can't clear space since it won't let me delete the folder.

I am left to believe the VM folder contents are still there just somehow invisible or inaccessible since apparently it is still taking up space.

I have 3 other VMs on the same datastore and they are all intact. Just this one VM disappeared with no explanation. The monitor for the datastore shows nothing. I am not sure where else to look or if it is even possible to get the VM back.

The problem VM is Windows Server 2019.  The other working VMs are Server 2019 and 2022 so I don't think it is the known issue with Server 2022 and secure boot.  I did try rolling back the update to 8.0a but that had no effect, the files for the problem VM are still gone.  One thing I did notice was that while the problem VM was still powered on, I could not create snapshots.  I don't remember what it said as to why.

Does anyone have any suggestions? Searching online found nothing. Every result was irrelevant.

[maksym007]

Strange issue. No one else reported such issues. 

What type of Datastore do you have? NFS or VMFS? 

[Shen88]

@evanc4 

This is really strange, I understand you're trying to find the cause and see if it could be recovered here. Well, if you have a logging tool in place like vRealize Log Insight/Aira Operations for Logs or any other 3rd party tool like Dynatrace and enabled logging mechanism from the VMs and VMware environment, you may be able to find what triggered this action while you performed the ESXi upgrade.

[evanc4]

@maksym007

This one is a VMFS store running off a standalone NVMe drive.

[evanc4]

@Shen88 

Unfortunately, I don't have any 3rd party logging tools.  This server is standalone ESXi with a single VMFS datastore running off of one NVMe drive.  I don't have vCenter or vRealize or any other service.  I did run VOMA and it said it could not reserve the drive even though the server was in maintenance mode and no VMs were running.  I also checked vmkernel.log file and it reported the drive was corrupt.

One thing I did notice was that before the patching, earlier in the day I was attempting to create a snapshot on that lost VM for an unrelated purpose and it failed without explanation.  I deferred investigation on that issue but that may have been my downfall.  But at the same time, even if the snapshot failed the files would have still be there in the VM folder.  To add insult to injury, on Saturday I added some files to the VM (it is a file server) and my last backup is from Friday so I lost the most recent data....

The only other clue I had was after rebooting from patching, I saw the lost VM listed by its GUID in the web GUI rather than its name.  My first thought was to unregister it and try to re-register, and that is when I found the files to be gone.

I am attempting to replace the drive and restore from backup.  Some of the files are replaceable but others are gone for good.

[maksym007]

IS VM listed on Datastore? I mean do you have the VM folder and files there? vmdk, vmx and all the rest?

If yes - try to register VM once again

[maksym007]

Or even better - how did you upgrade your ESXi? Via console or via vCenter? 

Maybe you have overwritten all settings and VMs which were there.

[evanc4]

@maksym007 

The VM folder itself was listed in the web gui datastore browser but all the vm files within such as the vmdk and vmx were all gone.  When I tried to open the vm folder in the web gui datastore browser, that is when it said an error occurred try again.  Then in the shell, I could traverse to the vm folder but ls -la shows zero files.

So /vmfs/volumes/nvme/vmname was empty.

As far as updating, I first set the server in maintenance mode and ran the update from the shell.  It indicated that the update was successful and then I rebooted:

esxcli software vib install -d "/vmfs/volumes/library/patch/VMware-ESXi-8.0b-21203435-depot.zip"

The library datastore was on a mounted NFS share where I kept the update patches.

[maksym007]

really strange. I dont know what to recommend you. 

In case it's homelab - its disaster if its prod - maybe backup will help

[ldclancy2]

What version of VMFS?

[evanc4]

@maksym007 

Agreed really strange.  Thanks anyway, I did replace the drive and restored from backup.  I'll probably run a data recovery tool on the broken drive to see if I can retrieve the vmdk file.

[evanc4]

@ldclancy2 

It is VMFS 6.