Repairing vmdk with snapshots

[binome333]

Short story: I shut down gracefully ESXi 5.0 to change some disks (move 4 and and 3).

When I power on again, no boot. (possibly a battery failure on my PERC 5/i as it appears to be weak now or a PSU problem - I changed both).

Partition table has been lost, as well as most ESXi partitions. VMFS data store on the same disk seems to be recoverable.

I saved (dd with linux live CD) all the disk to another one and copied again VMFS datastore to a new partition ; I reinstalled ESXi from scratch.

Achieved to mount again VMFS and copy back first virtual machine to main datastore.

Now the VM doesn't start, complaining about vmdk problem :

/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan # ls -l
-rw-------    1 root     root         1006698496 Apr 22 21:53 Kaitan-000001-delta.vmdk
-rw-------    1 root     root                315 Apr 22 21:53 Kaitan-000001.vmdk
-rw-------    1 root     root         6794838016 Apr 22 21:57 Kaitan-000002-delta.vmdk
-rw-------    1 root     root                322 Apr 22 21:57 Kaitan-000002.vmdk
-rw-------    1 root     root        19511967744 Apr 22 22:08 Kaitan-000003-delta.vmdk
-rw-------    1 root     root                348 Apr 22 22:08 Kaitan-000003.vmdk
-rw-------    1 root     root              28833 Apr 22 21:52 Kaitan-Snapshot1.vmsn
-rw-------    1 root     root              28847 Apr 22 21:53 Kaitan-Snapshot2.vmsn
-rw-------    1 root     root         2148825717 Apr 22 21:58 Kaitan-Snapshot3.vmsn
-rw-------    1 root     root        32473405440 Apr 22 21:51 Kaitan-flat.vmdk
-rw-------    1 root     root               8684 Apr 22 21:52 Kaitan.nvram
-rw-------    1 root     root                615 Apr 22 21:51 Kaitan.vmdk
-rw-r--r--    1 root     root               1427 Apr 22 21:34 Kaitan.vmsd
-rwxr-xr-x    1 root     root               3386 Apr 22 21:34 Kaitan.vmx
-rw-r--r--    1 root     root               3466 Apr 22 21:34 Kaitan.vmxf
-rw-------    1 root     root              20480 Apr 22 21:53 Kaitan_1-000001-delta.vmdk
-rw-------    1 root     root                319 Apr 22 21:53 Kaitan_1-000001.vmdk
-rw-------    1 root     root           16797696 Apr 22 21:57 Kaitan_1-000002-delta.vmdk
-rw-------    1 root     root                326 Apr 22 21:57 Kaitan_1-000002.vmdk
-rw-------    1 root     root         2466271232 Apr 22 22:10 Kaitan_1-000003-delta.vmdk
-rw-------    1 root     root                352 Apr 22 22:10 Kaitan_1-000003.vmdk
-rw-------    1 root     root         9113610240 Apr 22 21:52 Kaitan_1-flat.vmdk
-rw-------    1 root     root                591 Apr 22 21:52 Kaitan_1.vmdk
-rw-r--r--    1 root     root             160243 Apr 22 21:52 vmware-10.log
-rw-r--r--    1 root     root             408784 Apr 22 21:52 vmware-5.log
-rw-r--r--    1 root     root             165638 Apr 22 21:52 vmware-6.log
-rw-r--r--    1 root     root             159301 Apr 22 22:10 vmware-7.log
-rw-r--r--    1 root     root             555471 Apr 22 21:52 vmware-8.log
-rw-r--r--    1 root     root             159025 Apr 22 21:52 vmware-9.log
-rw-r--r--    1 root     root             160854 Apr 22 21:52 vmware.log

/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan # vmkfstools -e Kaitan-000001.vmdk -v 10
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001-delta.vmdk" : open successful (14) size = 1006698496, hd = 1068279. Type 8
DISKLIB-DSCPTR: Opened [0]: "Kaitan-000001-delta.vmdk" (0xe)
DISKLIB-LINK  : Opened '/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk' (0xe): vmfsSparse, 63424620 sectors / 30.2 GB.
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-flat.vmdk" : open successful (14) size = 32473405440, hd = 871672. Type 3
DISKLIB-DSCPTR: Opened [0]: "Kaitan-flat.vmdk" (0xe)
DISKLIB-LINK  : Opened '/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan.vmdk' (0xe): vmfs, 63424620 sectors / 30.2 GB.
DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
DISKLIB-CHAINESX : ChainESXOpenSubChainNode: can't create deltadisk node 7fdf066c-Kaitan-000001-delta.vmdk failed with error Invalid argument (0xbad0007, Bad parameter)
DISKLIB-CHAIN : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk" : failed to open (Invalid argument).
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001-delta.vmdk" : closed.
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-flat.vmdk" : closed.
DISKLIB-LIB   : Failed to open '/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk' with flags 0xe Invalid argument (1441801).
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001-delta.vmdk" : open successful (12) size = 1006698496, hd = 888056. Type 8
DISKLIB-DSCPTR: Opened [0]: "Kaitan-000001-delta.vmdk" (0xc)
DISKLIB-LINK  : Opened '/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk' (0xc): vmfsSparse, 63424620 sectors / 30.2 GB.
DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
DISKLIB-CHAINESX : ChainESXOpenSubChainNode: can't create deltadisk node 5d75e9f1-Kaitan-000001-delta.vmdk failed with error Invalid argument (0xbad0007, Bad parameter)
DISKLIB-CHAIN : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk" : failed to open (Invalid argument).
DISKLIB-VMFS  : "/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001-delta.vmdk" : closed.
DISKLIB-LIB   : Failed to open '/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk' with flags 0xc Invalid argument (1441801).
Failed to open disk link /vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan/Kaitan-000001.vmdk :Invalid argument (1441801)Disk chain is not consistent : Invalid argument (1441801)
AIOMGR-U : stat o=3 r=2 w=0 i=3 br=4096 bw=0
AIOMGR-S : stat o=3 r=9 w=0 i=0 br=147456 bw=0

/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan # vmkfstools -x check Kaitan-000001.vmdk
Disk is error free

/vmfs/volumes/516da918-df9a2c66-694d-0015171dfbb4/Kaitan # vmkfstools -x check Kaitan.vmdk
Disk is error free

I'm running out of ideas, if someone can help...

[continuum]

Can you please post the small vmdk descriptor files ?

[binome333]

Here they are with vmsd file (for 1st drive only as I'm limited to 5 files and 2nd drive is not really important).

[continuum]

the vmdk-files look ok - the chain uses
Kaitan-000003.vmdk -> Kaitan-000002.vmdk -> Kaitan-000001.vmdk > Kaitan.vmdk
but the vmsd seems to be bad.
Which vmdk do you set in the vmx-file ?
Does it help to rename or remove the vmsd file ?

[lvaibhavt]

Continum, can you please help me understand what did you check in the VMSD file that looks incorrect.

I am just trying to learn here

Thanks

[continuum]

if you look at the vmsd you see no reference to *-000003.vmdks - so it looks like it is not uptodate.

If you want - lets have a look via teamviewer and skype ...

[binome333]

Renaming vmsd file does not change anything.

(it is recreated empty when I start the VM, but the same errors are raised).

I have also tried to start the VM with the Kaitan.vmdk instead of Kaitan-000003.vmdk (quite older, but worth the try).

Again, the partition table was corrupted, but once repaired, the VM boots again (at least I have something !)

I still cannot use the remaining snapshots with this repaired vmdk (which is not surprising).

By the way, doesn't it look strange to have so many partition tables crashed (real and virtual) ?

I thought of my perc 5/i battery failure, but it's looking strange to me...