Hello All,
I have run into a critical issue here.
We had a ransomware attack in our datacenter last week. We started the restore process from backups. Two of our critical servers were live mounted from the backup as NFS datastore for instant recovery with the write cache redirected to another production datastore. The VMs ran fine for two days. But, suddenly a DB server VM which was live mounted went into a hung state.
The VM had "disks consolidation needed" alert on it at the time due to failed attempts to migrate it to the production datastore but there were no snapshots displayed in the Snapshot Manager.
The hung VM was rebooted expecting it to come back online. But, after the reboot the VM fails to come online. I see the below error when I power on the VM.
An error was received from the ESX host while powering on VM VM-LinksDB01-LM_migrated.
Failed to start the virtual machine.
Cannot open the disk '/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-000004.vmdk' or one of the snapshot disks it depends on.
The system cannot find the file specified
VMware ESX cannot find the virtual disk "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk". Verify the path is valid and try again.
Module DiskEarly power on failed.
Cannot open the disk '/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-000004.vmdk' or one of the snapshot disks it depends on.
The system cannot find the file specified
VMware ESX cannot find the virtual disk "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk". Verify the path is valid and try again.
When I look into the VMware.log files, I see that the flat file for the base vmdk is missing.
2019-07-31T18:53:19.036Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk' persistent R[]
2019-07-31T18:53:19.054Z| Worker#0| I120: AIOGNRC: Failed to open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk' : Could not find the file (60003) (0x2013).
2019-07-31T18:53:19.054Z| Worker#0| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk' : Could not find the file (393218).
2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-VMFS : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk" : failed to open (The system cannot find the file specified): ObjLib_Open failed. Type 3
2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-LINK : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk" : failed to open (The system cannot find the file specified).
2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk" : failed to open (The system cannot find the file specified).
2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk' with flags 0xa The system cannot find the file specified (25).
Now, I have a VM with disks spread across 2 Datastores(Base VM in NFS Datastore, All the delta and ckt files in the write-cached datastore) and the VM does not power on.
Below is a list of files in both the datastores.
/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e # ls -l
total 82972541
-rwxrwxrwx 1 1 1 8192968 Jul 31 11:11 VM-LinksDB01-ctk.vmdk
-rwxrwxrwx 1 1 1 84955234304 Jul 31 12:03 VM-LinksDB01.cache
-rwxrwxrwx 1 1 1 8684 Jul 31 13:02 VM-LinksDB01.nvram
-rwxrwxrwx 1 1 1 616 Jul 31 15:21 VM-LinksDB01.vmdk
-rwxrwxrwx 1 1 1 77 Jul 31 15:18 VM-LinksDB01.vmsd
-rwxrwxrwx 1 1 1 3040 Jul 31 2019 VM-LinksDB01.vmx
-rwxrwxrwx 1 1 1 3267 Jul 29 09:50 VM-LinksDB01.vmxf
-rwxrwxrwx 1 1 1 0 Jul 29 09:51 cache
-rwxrwxrwx 1 1 1 8 Jul 31 12:03 cache.records
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-12.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-13.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-14.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-15.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-16.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware-17.log
-rwxrwxrwx 1 1 1 62353 Jul 31 2019 vmware.log
/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e # ls -l
total 43186176
-rw-rw-rw- 1 root root 26056044544 Jul 29 23:45 VM-LinksDB01-000001-delta.vmdk
-rw-rw-rw- 1 root root 437 Jul 30 04:09 VM-LinksDB01-000001.vmdk
-rw-rw-rw- 1 root root 1276096512 Jul 30 04:38 VM-LinksDB01-000002-delta.vmdk
-rw-rw-rw- 1 root root 414 Jul 30 05:10 VM-LinksDB01-000002.vmdk
-rw-rw-rw- 1 root root 8192968 Jul 31 14:31 VM-LinksDB01-000003-ctk.vmdk
-rw-rw-rw- 1 root root 16644026368 Jul 31 10:35 VM-LinksDB01-000003-delta.vmdk
-rw-rw-rw- 1 root root 508 Jul 31 14:46 VM-LinksDB01-000003.vmdk
-rw-rw-rw- 1 root root 219131904 Jul 31 12:27 VM-LinksDB01-000004-delta.vmdk
-rw-rw-rw- 1 root root 336 Jul 31 12:37 VM-LinksDB01-000004.vmdk
-rw-rw-rw- 1 root root 8192968 Jul 31 14:38 VM-LinksDB01-000005-ctk.vmdk
-rw-rw-rw- 1 root root 1028096 Jul 31 14:38 VM-LinksDB01-000005-delta.vmdk
-rw-rw-rw- 1 root root 430 Jul 31 14:46 VM-LinksDB01-000005.vmdk
-rw-rw-rw- 1 root root 8192968 Jul 31 15:35 VM-LinksDB01-000006-ctk.vmdk
-rw-rw-rw- 1 root root 1028096 Jul 31 14:41 VM-LinksDB01-000006-delta.vmdk
-rw-rw-rw- 1 root root 407 Jul 31 14:41 VM-LinksDB01-000006.vmdk
How do I fix this issue and consolidate all the disks to a single datastore? Should I restore the -flat.vmdk file from the backup to the base VM folder and try powering on? What about the huge size cache files that I see in the folder?
For consolidation, once the VM is up, can I just create a new snapshot and then "Delete All"?
.vmx file attached for reference.
Below are the version details:
ESXi 5.5.0 2403361
Veeam Backup and Replication 9.5 U 4a
Any help is greatly appreciated.