Failed to start the virtual machine. 5 (Input/output error)

[GregCatalfo]

We tried rebooting a new VM we just created from a physical server the other day and it fails when powering on. The event states:

An error was received from the ESX host while powering on VM Peterbuilt.

Failed to start the virtual machine.

Module DiskEarly power on failed.

Cannot open the disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' or one of the snapshot disks it depends on.

5 (Input/output error)

The vmware.log from the first time it failed power on contains this:

2014-05-03T10:18:40.564Z| Worker#1| I120: DISK: OPEN scsi0:1 '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' persistent R[]

2014-05-03T10:18:40.564Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' persistent R[]

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" : open successful (10) size = 32462929920, hd = 29894442. Type 8

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" (0xa)

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' (0xa): vmfsSparse, 62912512 sectors / 30 GB.

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : open successful (10) size = 576889700352, hd = 32384813. Type 8

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" (0xa)

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' (0xa): vmfsSparse, 1118001152 sectors / 533.1 GB.

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb'.

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb" : failed to open (Input/output error): . Type 3

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error). 

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk": Failed to open parent "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk": Input/output error.

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-05-03T10:18:40.684Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : closed.

2014-05-03T10:18:40.684Z| Worker#1| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' with flags 0xa The parent of this virtual disk could not be opened (23).

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" : open successful (14) size = 32211206144, hd = 32581421. Type 3

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" (0xe)

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt.vmdk' (0xe): vmfs, 62912512 sectors / 30 GB.

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 32581421, extentType = 2

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 29894442, extentType = 0

2014-05-03T10:18:40.705Z| Worker#0| I120: DISKLIB-LIB   : Resuming change tracking.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 29894442.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Successfuly created cbt node 1ca272e-cbt.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/1ca272e-cbt

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk" (flags 0xa, type vmfs).

2014-05-03T10:18:40.711Z| Worker#0| I120: DISK: Disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' has UUID '60 00 c2 9c 4d 6e 95 99-d1 c1 5f c9 c0 e0 df e3'

2014-05-03T10:18:40.711Z| Worker#0| I120: DISK: OPEN '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' Geo (3916/255/63) BIOS Geo (3916/255/63)

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : open successful (8) size = 576889700352, hd = 30091058. Type 8

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" (0x8)

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' (0x8): vmfsSparse, 1118001152 sectors / 533.1 GB.

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 30091058, extentType = 0

2014-05-03T10:18:40.718Z| Worker#1| I120: DISKLIB-LIB   : Resuming change tracking.

2014-05-03T10:18:40.722Z| Worker#1| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 30091058.

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-CBT   : Successfuly created cbt node 233273a-cbt.

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/233273a-cbt

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk" (flags 0x8, type vmfsSparse).

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb'.

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb" : failed to open (Input/output error): . Type 3

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error). 

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk' with flags 0xe Input/output error (327689).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISK: Cannot open disk "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk": Input/output error (327689).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk' : Input/output error.

2014-05-03T10:18:40.725Z| Worker#1| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 30091058.

2014-05-03T10:18:40.725Z| Worker#1| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2014-05-03T10:18:40.737Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : closed.

2014-05-03T10:18:40.742Z| vmx| I120: DISK: Opening disks took 179 ms.

2014-05-03T10:18:40.742Z| vmx| I120: Module DiskEarly power on failed.

2014-05-03T10:18:40.742Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

2014-05-03T10:18:40.742Z| vmx| I120: scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)

2014-05-03T10:18:40.742Z| vmx| I120: Closing disk scsi0:0

2014-05-03T10:18:40.743Z| vmx| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 29894442.

2014-05-03T10:18:40.743Z| vmx| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2014-05-03T10:18:40.748Z| vmx| I120: DISKLIB-VMFS  : "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" : closed.

2014-05-03T10:18:40.749Z| vmx| I120: DISKLIB-VMFS  : "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" : closed.

[GregCatalfo]

The trouble started when the VM was shut down. As soon as it powered off, vCenter said it was "inaccessible". We removed it from inventory, then reregistered it from the datastore. The first boot attempt after that generated the errors above.

[depping]

I have seen similar issues in the past, but only when using an unsupported hardware configuration. Which disk controller / disk / SSD do you have? Also, if you have support I would suggest contacting GSS and file a ticket.

[GregCatalfo]

We do have support and have opened a ticket. The better part of the day was spent on the phone with them trying to resolve the issue. As for hardware, it all came from the HCL... at least it was on there when we purchased it. Seems some things have apparently been "dropped" from the supported list.

We have three Dell R710's, Micron p420m 700GB PCIe SSD (MTFDGAL700MAX-1AG1ZABYY on the HCL), Dell 2.5" 10k 600GB hot plug HDDs, Dell PERC H200 SAS controllers (...??), Intel dual 10GbE NICs from the HCL, and Netgear XS708E 10GbE switches (one dedicated to VSAN with IGMP Snooping disabled).

[ZUNYI_LIN]

Hi Creg, did you get this solved? I had an exact same problem. This is a very serious bug.

[ZUNYI_LIN]

2014-08-25T00:23:07.582Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:07.582Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:07.582Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:07.582Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:07.582Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:07.582Z| Worker#0| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000011.vmdk": Failed to open parent "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error.

2014-08-25T00:23:07.582Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-08-25T00:23:07.584Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://02dbb753-23a5-8232-55a8-60a44cea6f64" : closed.

2014-08-25T00:23:07.586Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" : closed.

2014-08-25T00:23:07.589Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : closed.

2014-08-25T00:23:07.591Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : closed.

2014-08-25T00:23:07.593Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:07.593Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000011.vmdk' with flags 0xe The parent of this virtual disk could not be opened (23).

2014-08-25T00:23:07.740Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://02dbb753-23a5-8232-55a8-60a44cea6f64" : open successful (12) size = 541066244096, hd = 143962381. Type 8

2014-08-25T00:23:07.740Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://02dbb753-23a5-8232-55a8-60a44cea6f64" (0xc)

2014-08-25T00:23:07.740Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000011.vmdk' (0xc): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:07.740Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-08-25T00:23:07.740Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 143962381, extentType = 0

2014-08-25T00:23:07.741Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000011.vmdk" (flags 0xc, type vmfsSparse).

2014-08-25T00:23:07.779Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" : open successful (14) size = 541066244096, hd = 145600784. Type 8

2014-08-25T00:23:07.779Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" (0xe)

2014-08-25T00:23:07.779Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:07.809Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : open successful (14) size = 541066244096, hd = 145535250. Type 8

2014-08-25T00:23:07.809Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" (0xe)

2014-08-25T00:23:07.809Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:07.969Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : open successful (14) size = 541066244096, hd = 138785044. Type 8

2014-08-25T00:23:07.969Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" (0xe)

2014-08-25T00:23:07.969Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.003Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : open successful (14) size = 541066244096, hd = 142586134. Type 8

2014-08-25T00:23:08.003Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" (0xe)

2014-08-25T00:23:08.003Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.005Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:08.005Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:08.005Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:08.005Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:08.005Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.005Z| Worker#0| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk": Failed to open parent "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error.

2014-08-25T00:23:08.005Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-08-25T00:23:08.008Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" : closed.

2014-08-25T00:23:08.010Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : closed.

2014-08-25T00:23:08.135Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : closed.

2014-08-25T00:23:08.259Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:08.259Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk' with flags 0xe The parent of this virtual disk could not be opened (23).

2014-08-25T00:23:08.300Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" : open successful (12) size = 541066244096, hd = 142651670. Type 8

2014-08-25T00:23:08.300Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" (0xc)

2014-08-25T00:23:08.300Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk' (0xc): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.300Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-08-25T00:23:08.300Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 142651670, extentType = 0

2014-08-25T00:23:08.301Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk" (flags 0xc, type vmfsSparse).

2014-08-25T00:23:08.354Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : open successful (14) size = 541066244096, hd = 133673241. Type 8

2014-08-25T00:23:08.354Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" (0xe)

2014-08-25T00:23:08.354Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.384Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : open successful (14) size = 541066244096, hd = 141275419. Type 8

2014-08-25T00:23:08.384Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" (0xe)

2014-08-25T00:23:08.384Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.460Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : open successful (14) size = 541066244096, hd = 145666333. Type 8

2014-08-25T00:23:08.460Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" (0xe)

2014-08-25T00:23:08.460Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.473Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:08.473Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:08.473Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:08.473Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:08.473Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.473Z| Worker#0| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk": Failed to open parent "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error.

2014-08-25T00:23:08.473Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-08-25T00:23:08.475Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : closed.

2014-08-25T00:23:08.478Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : closed.

2014-08-25T00:23:08.480Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:08.480Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk' with flags 0xe The parent of this virtual disk could not be opened (23).

2014-08-25T00:23:08.509Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : open successful (12) size = 541066244096, hd = 145731869. Type 8

2014-08-25T00:23:08.509Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" (0xc)

2014-08-25T00:23:08.509Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk' (0xc): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.509Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-08-25T00:23:08.509Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 145731869, extentType = 0

2014-08-25T00:23:08.510Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk" (flags 0xc, type vmfsSparse).

2014-08-25T00:23:08.546Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : open successful (14) size = 541066244096, hd = 142913824. Type 8

2014-08-25T00:23:08.546Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" (0xe)

2014-08-25T00:23:08.546Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.586Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : open successful (14) size = 541066244096, hd = 159428898. Type 8

2014-08-25T00:23:08.586Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" (0xe)

2014-08-25T00:23:08.586Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.588Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:08.588Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:08.588Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:08.588Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:08.588Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.588Z| Worker#0| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk": Failed to open parent "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error.

2014-08-25T00:23:08.588Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-08-25T00:23:08.591Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : closed.

2014-08-25T00:23:08.594Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:08.594Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' with flags 0xe The parent of this virtual disk could not be opened (23).

2014-08-25T00:23:08.613Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : open successful (12) size = 541066244096, hd = 159494434. Type 8

2014-08-25T00:23:08.613Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" (0xc)

2014-08-25T00:23:08.613Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' (0xc): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.613Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-08-25T00:23:08.614Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 159494434, extentType = 0

2014-08-25T00:23:08.615Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk" (flags 0xc, type vmfsSparse).

2014-08-25T00:23:08.650Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : open successful (14) size = 541066244096, hd = 130789669. Type 8

2014-08-25T00:23:08.650Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" (0xe)

2014-08-25T00:23:08.650Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' (0xe): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.652Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:08.652Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:08.652Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:08.652Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:08.652Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.652Z| Worker#0| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk": Failed to open parent "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error.

2014-08-25T00:23:08.652Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-08-25T00:23:08.654Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:08.654Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' with flags 0xe The parent of this virtual disk could not be opened (23).

2014-08-25T00:23:08.676Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : open successful (12) size = 541066244096, hd = 130855205. Type 8

2014-08-25T00:23:08.676Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" (0xc)

2014-08-25T00:23:08.676Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' (0xc): vmfsSparse, 1048576000 sectors / 500 GB.

2014-08-25T00:23:08.676Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-08-25T00:23:08.676Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 130855205, extentType = 0

2014-08-25T00:23:08.677Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk" (flags 0xc, type vmfsSparse).

2014-08-25T00:23:08.679Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-08-25T00:23:08.679Z| Worker#0| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64'.

2014-08-25T00:23:08.679Z| Worker#0| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-08-25T00:23:08.679Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://e3d0b753-6c93-cfb1-33bd-60a44cea6f64" : failed to open (Input/output error): . Type 3

2014-08-25T00:23:08.679Z| Worker#0| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.679Z| Worker#0| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk" : failed to open (Input/output error).

2014-08-25T00:23:08.679Z| Worker#0| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk' with flags 0xe Input/output error (327689).

2014-08-25T00:23:08.679Z| Worker#0| I120: DISK: Cannot open disk "/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk": Input/output error (327689).

2014-08-25T00:23:08.679Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmdk' : Input/output error.

2014-08-25T00:23:08.682Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://edd2b753-d820-51e1-02d7-60a44cea6f64" : closed.

2014-08-25T00:23:08.682Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000002.vmdk' : Input/output error.

2014-08-25T00:23:08.685Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://73d6b753-9c39-80f7-36db-60a44cea6f64" : closed.

2014-08-25T00:23:08.685Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000006.vmdk' : Input/output error.

2014-08-25T00:23:08.687Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://79d9b753-a4ff-4d76-7b46-60a44cea6f64" : closed.

2014-08-25T00:23:08.687Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000007.vmdk' : Input/output error.

2014-08-25T00:23:08.689Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://eadab753-08c7-0f2a-f43c-60a44cea6f64" : closed.

2014-08-25T00:23:08.689Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000008.vmdk' : Input/output error.

2014-08-25T00:23:08.813Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://02dbb753-23a5-8232-55a8-60a44cea6f64" : closed.

2014-08-25T00:23:08.813Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000011.vmdk' : Input/output error.

2014-08-25T00:23:08.938Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://8ddfb753-7cf0-7dfc-281e-60a44cea6f64" : closed.

2014-08-25T00:23:08.938Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000004.vmdk' : Input/output error.

2014-08-25T00:23:09.062Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://98e3b753-8db0-219e-48bd-60a44cea6f64" : closed.

2014-08-25T00:23:09.062Z| Worker#0| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1-000009.vmdk' : Input/output error.

2014-08-25T00:23:09.187Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://ce05b853-a8ac-40ec-5624-60a44cea6f64" : closed.

2014-08-25T00:23:09.187Z| vmx| I120: DISK: Opening disks took 8808 ms.

2014-08-25T00:23:09.187Z| vmx| I120: Module DiskEarly power on failed.

2014-08-25T00:23:09.187Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

2014-08-25T00:23:09.187Z| vmx| I120: SVMotion_PowerOff: Not running Storage vMotion. Nothing to do

2014-08-25T00:23:09.206Z| vmx| I120: WORKER: asyncOps=2 maxActiveOps=1 maxPending=0 maxCompleted=1

2014-08-25T00:23:09.206Z| vmx| I120: Vix: [12602616 mainDispatch.c:1201]: VMAutomationPowerOff: Powering off.

2014-08-25T00:23:09.206Z| vmx| W110: /vmfs/volumes/vsan:520a56ce62c347de-4b0d702ef9709107/dcd0b753-52a0-425f-28d7-60a44cea6f64/U8SERVER1.vmx: Cannot remove symlink /var/run/vmware/root_0/1408926179914626_12602616/configFile: No such file or directory

2014-08-25T00:23:09.230Z| vmx| I120: Vix: [12602616 mainDispatch.c:3964]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0

2014-08-25T00:23:09.231Z| vmx| I120: Locale_Errno2LocalString: "C" locale found. LC_ALL=(null), LANG=C

2014-08-25T00:23:09.232Z| vmx| I120: Locale_Errno2LocalString: locale(LC_ALL)=C locale(LC_MESSAGES)=C

2014-08-25T00:23:09.233Z| vmx| I120: Vix: [12602616 mainDispatch.c:3964]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1 additionalError=0

2014-08-25T00:23:09.233Z| vmx| I120: Transitioned vmx/execState/val to poweredOff

2014-08-25T00:23:09.233Z| vmx| I120: Vix: [12602616 mainDispatch.c:3964]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=0 additionalError=0

2014-08-25T00:23:09.233Z| vmx| I120: Vix: [12602616 mainDispatch.c:4003]: Error VIX_E_FAIL in VMAutomation_ReportPowerOpFinished(): Unknown error

2014-08-25T00:23:09.233Z| vmx| I120: Vix: [12602616 mainDispatch.c:3964]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1 additionalError=0

2014-08-25T00:23:09.233Z| vmx| I120: Transitioned vmx/execState/val to poweredOff

[GregCatalfo]

It was, but vmware support took care of it and wound up writing us a custom hot fix which is supposed to be incorporated into VSAN 2.0.

[ZUNYI_LIN]

Thanks very much for your quick reply.

So did the custom hot fix get your data back? Or it just prevent the issue from happening again?

I have a dead VM right now with valuable data in it. I am so disappointed at VSAN.

If you could share your hot fix, that will save my life.

[GregCatalfo]

The hot fix was only preventative. The data recovery actually took those guys the better part of a day to figure out.

[ZUNYI_LIN]

After 22 hour non-sleep trying every method I can think of, luckily I got it solved.

I did two things I am not sure which one actually fixed the problem. But it got be one in these two.

1. I rebooted the host that the VM reside in.

2. I changed the name of the VM.

Then surprising I can boot the VM.

So I hope this can help other people.

I got go to sleep......

[HamR]

Thanks for posting your solution here. I've been able to resolve this faster than a Sev1 with GSS.

I had the same error with VSAN 2.0 and it took two reboots to get the VM to move and restart on another host, with vSphere 6 U1. In my case, the VMs were one PSC and the vCenter Server.

HamR

[LandenT]

Same problem here. This all started for me when trying to put a host into maintenance mode. It took about 10 hours, then a VM stopped working and now will not boot with a I/O error.  All hardware is Dell and on the HCL. Running 6.0 U2 all patches applied.  Clearly vSAN is not ready for a production environment and is a beta product. Maybe in 5 years this thing is something I would trust for our prod stuff,even our dev stuff. 

[LandenT]

Same thing happened to us this week, using all Dell R720 hardware on the HCL with 6.0 U2 (latest patches).  Put a host into maintenance mode and a VM went out of HA and stopped running and now wont boot with I/O error.  vSAN still buggy and not ready for production.

[depping]

have you contacted support @LandenT?

[ChrisKuhns]

I am sad to say I have to agree. I was the biggest champion of it here. I had a four node custom built vSAN that ran flawlessly until 6.0 U2. I used it to talk my team in to investing in an 8 Node All-Flash vSAN Ready Node to replace our school district's data center we are collapsing. Before the hardware arrived I decided to upgrade my hybrid cluster to 6.0 U2 and I have regretted it and the purchase ever since.

Starting in July of 2016, the original vSAN began crashing daily. Every crash resulted in the datastore growing bigger. I was able to move everything off except for the largest email archive server I had and it took weeks of no sleep and extreme stress. I used the new vSAN to offload it. As of November the old one was unrecoverable. It took my vendor and Dell from August to this month to stabilize the myriad of issues with the Ready Node. The amount of work and extra capital to bring it online with even a hint of stability was the most complex thing I have ever experienced and I have built entire data centers. We ended up having to get dedicated switch modules on the cores and PCIe cards on the FX2s chassis to handle the vSAN traffic. This was after months of rebuilding the cluster on some level. Finally after it was working and my test VMs were putting it through its paces, VMware told our vendor to do the firmware upgrades on the chassis and servers. Upon reboot entire disk groups across the cluster were no longer seen. Patches were run, VIBs installed, reboots, unmounts, mounts, all managed to bring up 15 of 16 of the groups.After a week of phone tag VMware has me run another cluster wide patch. This did bring up the last disk group. During the reboot of all of the servers another disk group decided it wasn't going to function.

While I am sitting here typing this I am waiting on the other disk group to fix itself. By the looks of it, I don't think it will come back online either. It will be just like the other group that doesn't work.

I am due to move eighteen hosts worth of VMs down to 8 and I frankly have ZERO faith left in this product. I had it for two years, loved it, bragged about it, and seven months of lost VMs and unrecoverable data I am so pissed at myself for signing five years worth of it. I look like a fool among my peers for getting us to move over to this platform. An EMC SAN that has had no issues for seven years is getting replaced by something me, our vendor, Dell, and even VMware can't seem to get to work right.

[zdickinson]

Good afternoon, I sympathize with your situation.  I'm not sure I would go as far as a blanket statement of, not ready for production across the board.  I would say it is not ready for all production in every case.  I would not recommend it for your one-all-be-all for production.  We used it for DR and it was a great solution for that.  But if I think about the problems I have had with it in DR and apply that to production, it is not pleasant.  I like vSAN for DR, test, dev, or production if it is a large team with many skills and resources to maintain it.

What it did show me is that I love HCI, CI but do not want to maintain it myself.  VxRail is very much in my sights.  A lesson I learned, if you have vSAN for production and 1 vSAN only, stay a rev or two behind.  Thank you, Zach.

[joshphp]

I am having the same issue with Horizon View 7 and linked clones running ESXi 6.0 U2..

[x86optimizer]

I completely agree with this.  I had a power interruption on a 6 node Dell 620 setup with 6.0 U2 vSAN hybrid. All strictly HCL and drivers/firmware as well.  2 of our critical VM's went into error 5(Input/output error).  

vSAN is way to buggy and unstable to be used for anything but development and used for testing. We have been burned too many times.

I really wish this worked. I am a big advocate of this technology and what it SHOULD be able to do, but its just not where it needs to be for production use.

Now back to trying to figure out how to get the VM's back on-line.

[x86optimizer]

Happened on my setup   Everything on HCL Drivers and firmware.

(6) R620

Server Specs

H710 Controller
(2) SSD Dell Each

(6) 1.2TB HD Dell each
  256GB  Each

2x1G Ethernet Dell

2x10G Ethernet Dell

1Gbit & 10Gbit Networks.

6.0 U2  vSAN Hybrid.

[x86optimizer]

Didn't fix the bug apparently :smileyconfused: