VMware Cloud Community
pramodsd_wannag
Contributor
Contributor
‎07-31-2019 08:34 PM

Unable to power on VM. VMDK file not found error.

Hello All,

I have run into a critical issue here.

We had a ransomware attack in our datacenter last week. We started the restore process from backups. Two of our critical servers were live mounted from the backup as NFS datastore for instant recovery with the write cache redirected to another production datastore. The VMs ran fine for two days. But, suddenly a DB server VM which was live mounted went into a hung state.

The VM had "disks consolidation needed" alert on it at the time due to failed attempts to migrate it to the production datastore but there were no snapshots displayed in the Snapshot Manager.

The hung VM was rebooted expecting it to come back online. But, after the reboot the VM fails to come online. I see the below error when I power on the VM.

An error was received from the ESX host while powering on VM VM-LinksDB01-LM_migrated.

Failed to start the virtual machine.

Cannot open the disk '/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-000004.vmdk' or one of the snapshot disks it depends on.

The system cannot find the file specified

VMware ESX cannot find the virtual disk "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk". Verify the path is valid and try again.

Module DiskEarly power on failed.

Cannot open the disk '/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-000004.vmdk' or one of the snapshot disks it depends on.

The system cannot find the file specified

VMware ESX cannot find the virtual disk "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk". Verify the path is valid and try again. 

When I look into the VMware.log files, I see that the flat file for the base vmdk is missing.

2019-07-31T18:53:19.036Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk' persistent R[]

2019-07-31T18:53:19.054Z| Worker#0| I120: AIOGNRC: Failed to open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk' : Could not find the file (60003) (0x2013).

2019-07-31T18:53:19.054Z| Worker#0| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk' : Could not find the file (393218).

2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-VMFS  : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01-flat.vmdk" : failed to open (The system cannot find the file specified): ObjLib_Open failed. Type 3

2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk" : failed to open (The system cannot find the file specified). 

2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk" : failed to open (The system cannot find the file specified).

2019-07-31T18:53:19.054Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e/VM-LinksDB01.vmdk' with flags 0xa The system cannot find the file specified (25).

Now, I have a VM with disks spread across 2 Datastores(Base VM in NFS Datastore, All the delta and ckt files in the write-cached datastore) and the VM does not power on.

Below is a list of files in both the datastores.

/vmfs/volumes/79472a69-54e95c8b/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e # ls -l

total 82972541

-rwxrwxrwx    1 1        1          8192968 Jul 31 11:11 VM-LinksDB01-ctk.vmdk

-rwxrwxrwx    1 1        1        84955234304 Jul 31 12:03 VM-LinksDB01.cache

-rwxrwxrwx    1 1        1             8684 Jul 31 13:02 VM-LinksDB01.nvram

-rwxrwxrwx    1 1        1              616 Jul 31 15:21 VM-LinksDB01.vmdk

-rwxrwxrwx    1 1        1               77 Jul 31 15:18 VM-LinksDB01.vmsd

-rwxrwxrwx    1 1        1             3040 Jul 31  2019 VM-LinksDB01.vmx

-rwxrwxrwx    1 1        1             3267 Jul 29 09:50 VM-LinksDB01.vmxf

-rwxrwxrwx    1 1        1                0 Jul 29 09:51 cache

-rwxrwxrwx    1 1        1                8 Jul 31 12:03 cache.records

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-12.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-13.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-14.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-15.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-16.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware-17.log

-rwxrwxrwx    1 1        1            62353 Jul 31  2019 vmware.log

/vmfs/volumes/554462ef-bcfa354e-3d66-984be10466c2/VeeamIR/VM-LinksDB01-LM_f2e9cd4f-8c81-49ad-ad54-1a1ad4c47c1e # ls -l

total 43186176

-rw-rw-rw-    1 root     root     26056044544 Jul 29 23:45 VM-LinksDB01-000001-delta.vmdk

-rw-rw-rw-    1 root     root           437 Jul 30 04:09 VM-LinksDB01-000001.vmdk

-rw-rw-rw-    1 root     root     1276096512 Jul 30 04:38 VM-LinksDB01-000002-delta.vmdk

-rw-rw-rw-    1 root     root           414 Jul 30 05:10 VM-LinksDB01-000002.vmdk

-rw-rw-rw-    1 root     root       8192968 Jul 31 14:31 VM-LinksDB01-000003-ctk.vmdk

-rw-rw-rw-    1 root     root     16644026368 Jul 31 10:35 VM-LinksDB01-000003-delta.vmdk

-rw-rw-rw-    1 root     root           508 Jul 31 14:46 VM-LinksDB01-000003.vmdk

-rw-rw-rw-    1 root     root     219131904 Jul 31 12:27 VM-LinksDB01-000004-delta.vmdk

-rw-rw-rw-    1 root     root           336 Jul 31 12:37 VM-LinksDB01-000004.vmdk

-rw-rw-rw-    1 root     root       8192968 Jul 31 14:38 VM-LinksDB01-000005-ctk.vmdk

-rw-rw-rw-    1 root     root       1028096 Jul 31 14:38 VM-LinksDB01-000005-delta.vmdk

-rw-rw-rw-    1 root     root           430 Jul 31 14:46 VM-LinksDB01-000005.vmdk

-rw-rw-rw-    1 root     root       8192968 Jul 31 15:35 VM-LinksDB01-000006-ctk.vmdk

-rw-rw-rw-    1 root     root       1028096 Jul 31 14:41 VM-LinksDB01-000006-delta.vmdk

-rw-rw-rw-    1 root     root           407 Jul 31 14:41 VM-LinksDB01-000006.vmdk

How do I fix this issue and consolidate all the disks to a single datastore? Should I restore the -flat.vmdk file from the backup to the base VM folder and try powering on? What about the huge size cache files that I see in the folder?

For consolidation, once the VM is up, can I just create a new snapshot and then "Delete All"?

.vmx file attached for reference.

Below are the version details:

ESXi 5.5.0 2403361

Veeam Backup and Replication 9.5 U 4a

Any help is greatly appreciated.

Tags (1)
VM-LinksDB01.vmx.zip
Reply
0 Kudos
0 Replies