VMware Cloud Community
MBIII
Contributor
Contributor

Corrupt delta file - Sync IO issued on cow disk xxx failed with short IO 0 bytes actual 16384

After a weird system hang I restarted the ESXI host and ended up not able to boot up of the VM's. The VM had a snapshot and it seems the -delta file of the snapshot got corrupted, or final block was not written to it.

All CID's are OK from what I can tell.

I get below error in vmkernel.log when trying to boot up the VM:

2017-07-01T15:44:47.025Z cpu2:161448 opID=6db8d44c)World: 15446: VC opID 4560189D-000000A5-f3d2 maps to vmkernel opID 6db8d44c

2017-07-01T15:44:47.025Z cpu2:161448 opID=6db8d44c)Config: 679: "SIOControlFlag2" = 1, Old Value: 0, (Status: 0x0)

2017-07-01T15:44:48.223Z cpu6:221918)World: vm 221919: 1646: Starting world vmm0:afrparv-web01_1 of type 8

2017-07-01T15:44:48.223Z cpu6:221918)Sched: vm 221919: 6485: Adding world 'vmm0:afrparv-web01_1', group 'host/user', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1

2017-07-01T15:44:48.223Z cpu6:221918)Sched: vm 221919: 6500: renamed group 1394888 to vm.221918

2017-07-01T15:44:48.224Z cpu6:221918)Sched: vm 221919: 6517: group 1394888 is located under group 4

2017-07-01T15:44:48.246Z cpu6:221918)MemSched: vm 221918: 8113: extended swap to 43458 pgs

2017-07-01T15:44:48.358Z cpu2:221918)WARNING: MemSched: 6236: Psharing is disabled but ballooning is not.

2017-07-01T15:44:48.462Z cpu0:221918)World: vm 221921: 1646: Starting world vmm1:afrparv-web01_1 of type 8

2017-07-01T15:44:48.643Z cpu6:221918)WARNING: VmfsSparse: 6385: Bad leaf entry: leafEntry->sectorOffset[2312] + sparseInfo->granularity (bytes = 370060288) >= length (154148864)

2017-07-01T15:44:48.643Z cpu6:221918)WARNING: VmfsSparse: 6385: Bad leaf entry: leafEntry->sectorOffset[2313] + sparseInfo->granularity (bytes = 370060800) >= length (154148864)

2017-07-01T15:44:48.643Z cpu6:221918)WARNING: VmfsSparse: 6385: Bad leaf entry: leafEntry->sectorOffset[2314] + sparseInfo->granularity (bytes = 370061312) >= length (154148864)

2017-07-01T15:44:48.643Z cpu6:221918)WARNING: VmfsSparse: 6385: Bad leaf entry: leafEntry->sectorOffset[2315] + sparseInfo->granularity (bytes = 370061824) >= length (154148864)

2017-07-01T15:44:48.645Z cpu6:221918)WARNING: VmfsSparse: 6353: Bad root entry: (page 0, slot 6) (bytes=185909760) >= length (154148864)

2017-07-01T15:44:48.645Z cpu6:221918)Cow: 2033: Sync IO issued on cow disk 'afrparv-web01-000001-delta.vmdk' failed with short IO 0 bytes actual 16384

2017-07-01T15:44:48.645Z cpu6:221918)VmfsSparse: 2076: Sparse COW open (fd 804319, isSparse: 0, sparseInfo 0x439dc14cca40) finished with status: I/O error

2017-07-01T15:44:48.645Z cpu6:221918)Cow: 698: Failed to open file afrparv-web01-000001-delta.vmdk, level 1 of 3 (extent 'vmfssparse' fhID804319) with I/O error

2017-07-01T15:44:48.645Z cpu6:221918)WARNING: DD: 679: Opening the cow hierarchy for 3b17500e-afrparv-web01-000005-delta.vmdk failed with I/O error

2017-07-01T15:44:49.566Z cpu5:161434)Config: 679: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

Trying to clone the vmdk file:

[root@AFRPARS-ESXI01:/vmfs/volumes/553e3c22-c98645b0-66b8-0cc47a0e98f8/backups01] vmkfstools -i afrparv-web01-000001.vmdk  fixed.vmdk

Failed to open 'afrparv-web01-000001.vmdk': Input/output error (327689).

I've attached all logs and vmdk files.

continuum​ or any other superduper expert, please help!

0 Kudos
3 Replies
continuum
Immortal
Immortal

Hello
I can work around I/O errors but with delta vmdks the result can still be unusable - anyway to try it, download

http://sanbarrow.com/livecds/moa64-nogui/MOA64-nogui-incl-src-111014-efi.iso
and call me via skype.
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos
parmarr
VMware Employee
VMware Employee

This is an IO error caused by system outage. We need to validate if any storage error encountered during  outage. No vmkernel logs. the only option is to restore the file from backup. For reference: https://kb.vmware.com/kb/2052893 

Sincerely, Rahul Parmar VMware Support Moderator
0 Kudos
continuum
Immortal
Immortal

> the only option is to restore the file from backup.
Dont you think that it is a bit early to make such a statement ?
I had success in similar cases often enough to not give up so early.
Ulli


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

0 Kudos