I've started to see this issue on random VM's on our companys ESX cluster (3 hosts). I've had to stop working on this cluster until we can resolve the issue
I've run into a similar issue before that related to snapshots, however this problem is occuring on machines that have never been snapshotted, and in the past, had been working flawlessly.
Here is a snippet from the VMware log file of the VM:
2013-07-09T00:41:32.918Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' persistent R[]
2013-07-09T00:41:32.918Z| Worker#1| I120: DISK: OPEN scsi0:1 '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' persistent R[]
2013-07-09T00:41:32.919Z| Worker#1| I120: FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
2013-07-09T00:41:32.919Z| Worker#1| I120: AIOGNRC: Failed to open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk' : Cannot allocate memory (c0002) (0x2013).
2013-07-09T00:41:32.919Z| Worker#1| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk' : Cannot allocate memory (786435).
2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk" : failed to open (Cannot allocate memory): ObjLib_Open failed. Type 3
2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-LINK : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk" : failed to open (Cannot allocate memory).
2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk" : failed to open (Cannot allocate memory).
2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' with flags 0xa Cannot allocate memory (786441).
2013-07-09T00:41:32.920Z| Worker#1| I120: DISK: Cannot open disk "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk": Cannot allocate memory (786441).
2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-VMFS : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager-flat.vmdk" : open successful (10) size = 53687091200, hd = 3565896. Type 3
2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "JD-5-manager-flat.vmdk" (0xa)
2013-07-09T00:41:32.920Z| Worker#1| I120: Msg_Post: Error
2013-07-09T00:41:32.920Z| Worker#1| I120: [msg.literal] Cannot allocate memory
2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-LINK : Opened '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' (0xa): vmfs, 104857600 sectors / 50 GB.
2013-07-09T00:41:32.920Z| Worker#1| I120: [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' or one of the snapshot disks it depends on.
2013-07-09T00:41:32.920Z| Worker#1| I120: ----------------------------------------
2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-LIB : Opened "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk" (flags 0xa, type vmfs).
2013-07-09T00:41:32.920Z| Worker#0| I120: DISK: Disk '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' has UUID '60 00 c2 9c cf e7 7a d0-c9 a4 29 f2 3f 42 e7 e9'
2013-07-09T00:41:32.920Z| Worker#0| I120: DISK: OPEN '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' Geo (6527/255/63) BIOS Geo (0/0/0)
2013-07-09T00:41:32.933Z| vmx| I120: DISK: Opening disks took 16 ms.
2013-07-09T00:41:32.933Z| vmx| I120: Module DiskEarly power on failed.
2013-07-09T00:41:32.933Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0
2013-07-09T00:41:32.933Z| vmx| I120: scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)
2013-07-09T00:41:32.933Z| vmx| I120: Closing disk scsi0:0
2013-07-09T00:41:32.933Z| vmx| I120: DISKLIB-VMFS : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager-flat.vmdk" : closed.
2013-07-09T00:41:32.937Z| vmx| I120: WORKER: asyncOps=3 maxActiveOps=2 maxPending=0 maxCompleted=2
And the vmdk file of the affected disk:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=4ed52c7e
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 629145600 VMFS "JD-5-manager_1-flat.vmdk"
# The Disk Data Base
#DDB
ddb.deletable = "true"
ddb.virtualHWVersion = "8"
ddb.longContentID = "8264138edeb9f6fb2afaf3be4ed52c7e"
ddb.uuid = "60 00 C2 98 c9 03 10 f9-19 1a c3 34 a1 ba e3 1c"
ddb.geometry.cylinders = "39162"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.thinProvisioned = "1"
ddb.adapterType = "lsilogic"
As you can see the parentCID is the correct setting for a vmdk with no snapshots.
Any tips for troubleshooting this problem?
Thanks kindly.
After that prompting I checked the vmkwarning log and found the following:
/var/log # tail vmkwarning.log
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
Which lead me to the following VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100442...
Using PowerCLI, I checked the total size of all VMDK files on one member of the cluster with the following command:
PowerCLI > Get-HardDisk -Datastore <datastore> | select -ExpandProperty CapacityGB | measure -sum -ave -max -min
Count : 78
Average : 197.730180789263
Sum : 15422.9541015625
Maximum : 544.49609375
Minimum : 50
Property :
According to that article, the maximum supported size of all opened VMDK files was only 8TB, so I was well above that, therefore running into issues.
I've adjusted the VMFS3.MaxHeapSizeMB to the maxmum 256MB to compensate, and that has resolved this issue.
Recreate the issue and upload the latest vmkernel log and full vmware.log
After that prompting I checked the vmkwarning log and found the following:
/var/log # tail vmkwarning.log
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand
2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan
2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa
Which lead me to the following VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100442...
Using PowerCLI, I checked the total size of all VMDK files on one member of the cluster with the following command:
PowerCLI > Get-HardDisk -Datastore <datastore> | select -ExpandProperty CapacityGB | measure -sum -ave -max -min
Count : 78
Average : 197.730180789263
Sum : 15422.9541015625
Maximum : 544.49609375
Minimum : 50
Property :
According to that article, the maximum supported size of all opened VMDK files was only 8TB, so I was well above that, therefore running into issues.
I've adjusted the VMFS3.MaxHeapSizeMB to the maxmum 256MB to compensate, and that has resolved this issue.