Failed to start the virtual machine. 5 (Input/output error)

[GregCatalfo]

We tried rebooting a new VM we just created from a physical server the other day and it fails when powering on. The event states:

An error was received from the ESX host while powering on VM Peterbuilt.

Failed to start the virtual machine.

Module DiskEarly power on failed.

Cannot open the disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' or one of the snapshot disks it depends on.

5 (Input/output error)

The vmware.log from the first time it failed power on contains this:

2014-05-03T10:18:40.564Z| Worker#1| I120: DISK: OPEN scsi0:1 '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' persistent R[]

2014-05-03T10:18:40.564Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' persistent R[]

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" : open successful (10) size = 32462929920, hd = 29894442. Type 8

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" (0xa)

2014-05-03T10:18:40.659Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' (0xa): vmfsSparse, 62912512 sectors / 30 GB.

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : open successful (10) size = 576889700352, hd = 32384813. Type 8

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" (0xa)

2014-05-03T10:18:40.671Z| Worker#1| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' (0xa): vmfsSparse, 1118001152 sectors / 533.1 GB.

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb'.

2014-05-03T10:18:40.683Z| Worker#1| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb" : failed to open (Input/output error): . Type 3

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error). 

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-CHAIN :"/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk": Failed to open parent "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk": Input/output error.

2014-05-03T10:18:40.683Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (The parent of this virtual disk could not be opened).

2014-05-03T10:18:40.684Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : closed.

2014-05-03T10:18:40.684Z| Worker#1| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' with flags 0xa The parent of this virtual disk could not be opened (23).

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-VMFS  : "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" : open successful (14) size = 32211206144, hd = 32581421. Type 3

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" (0xe)

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt.vmdk' (0xe): vmfs, 62912512 sectors / 30 GB.

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 32581421, extentType = 2

2014-05-03T10:18:40.704Z| Worker#0| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 29894442, extentType = 0

2014-05-03T10:18:40.705Z| Worker#0| I120: DISKLIB-LIB   : Resuming change tracking.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 29894442.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Successfuly created cbt node 1ca272e-cbt.

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/1ca272e-cbt

2014-05-03T10:18:40.711Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk" (flags 0xa, type vmfs).

2014-05-03T10:18:40.711Z| Worker#0| I120: DISK: Disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' has UUID '60 00 c2 9c 4d 6e 95 99-d1 c1 5f c9 c0 e0 df e3'

2014-05-03T10:18:40.711Z| Worker#0| I120: DISK: OPEN '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt-000002.vmdk' Geo (3916/255/63) BIOS Geo (3916/255/63)

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : open successful (8) size = 576889700352, hd = 30091058. Type 8

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" (0x8)

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk' (0x8): vmfsSparse, 1118001152 sectors / 533.1 GB.

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1

2014-05-03T10:18:40.717Z| Worker#1| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 30091058, extentType = 0

2014-05-03T10:18:40.718Z| Worker#1| I120: DISKLIB-LIB   : Resuming change tracking.

2014-05-03T10:18:40.722Z| Worker#1| I120: DISKLIB-CBT   : Initializing ESX kernel change tracking for fid 30091058.

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-CBT   : Successfuly created cbt node 233273a-cbt.

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-CBT   : Opening cbt node /vmfs/devices/cbt/233273a-cbt

2014-05-03T10:18:40.723Z| Worker#1| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1-000002.vmdk" (flags 0x8, type vmfsSparse).

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjReadPolicy: failed to readPolicy for object: Input/output error (327682).

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-VSANOBJ: VsanObjGetExtParams: Could not read policy for 'vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb'.

2014-05-03T10:18:40.724Z| Worker#1| I120: OBJLIB-LIB: ObjLib_GetExtParams : Failed to get ext params : Input/output error (327684)

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1eee6253-1e80-39fa-229e-842b2b78ebfb" : failed to open (Input/output error): . Type 3

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-LINK  : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error). 

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk" : failed to open (Input/output error).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk' with flags 0xe Input/output error (327689).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISK: Cannot open disk "/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk": Input/output error (327689).

2014-05-03T10:18:40.724Z| Worker#1| I120: DISK: Cannot open disk '/vmfs/volumes/vsan:528a8f794664ebd8-6300c6c1887b898b/17ee6253-9171-b202-26f4-842b2b78ebfb/Peterbuilt_1.vmdk' : Input/output error.

2014-05-03T10:18:40.725Z| Worker#1| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 30091058.

2014-05-03T10:18:40.725Z| Worker#1| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2014-05-03T10:18:40.737Z| Worker#1| I120: DISKLIB-VMFS  : "vsan://1f846353-d29f-90dd-fe1b-d4ae52836f43" : closed.

2014-05-03T10:18:40.742Z| vmx| I120: DISK: Opening disks took 179 ms.

2014-05-03T10:18:40.742Z| vmx| I120: Module DiskEarly power on failed.

2014-05-03T10:18:40.742Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

2014-05-03T10:18:40.742Z| vmx| I120: scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)

2014-05-03T10:18:40.742Z| vmx| I120: Closing disk scsi0:0

2014-05-03T10:18:40.743Z| vmx| I120: DISKLIB-CBT   : Shutting down change tracking for untracked fid 29894442.

2014-05-03T10:18:40.743Z| vmx| I120: DISKLIB-CBT   : Successfully disconnected CBT node.

2014-05-03T10:18:40.748Z| vmx| I120: DISKLIB-VMFS  : "vsan://1d846353-60b1-b2e4-7696-d4ae52836f43" : closed.

2014-05-03T10:18:40.749Z| vmx| I120: DISKLIB-VMFS  : "vsan://1cee6253-07b9-e212-6f6a-842b2b78ebfb" : closed.

[jameseydoyle]

What kind of a power interruption did you have? Did it affect multiple/all servers? Did they go down at different times? These are all major considerations that you didn't mention. What other issues have you faced in the past that caused you get burned?

We hate to hear of negative experiences like this. Did you engage support with this or previous issues to investigate your setup to see if there were ways to mitigate against these failures?

[x86optimizer]

All 6 servers lost power at the same time or within a few minutes of each other.  There are two circuits, one lost power and all the load shifted to the other

and it overloaded and shut down. I think it all took place in about 5 mins or less.

Have had power interruptions before, and had have lost VM's.  We have had backup's thankfully, but this one, the back up is very out dated.

[x86optimizer]

Thanks will try this

[shigmas]

I didn't think that'd work, but it did!!