VMware Cloud Community
cgillpwe
Contributor
Contributor
Jump to solution

Unable to power on VM - Cannot allocate memory

I've started to see this issue on random VM's on our companys ESX cluster (3 hosts). I've had to stop working on this cluster until we can resolve the issue Smiley Sad

I've run into a similar issue before that related to snapshots, however this problem is occuring on machines that have never been snapshotted, and in the past, had been working flawlessly.

Here is a snippet from the VMware log file of the VM:

2013-07-09T00:41:32.918Z| Worker#0| I120: DISK: OPEN scsi0:0 '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' persistent R[]

2013-07-09T00:41:32.918Z| Worker#1| I120: DISK: OPEN scsi0:1 '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' persistent R[]

2013-07-09T00:41:32.919Z| Worker#1| I120: FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory

2013-07-09T00:41:32.919Z| Worker#1| I120: AIOGNRC: Failed to open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk' : Cannot allocate memory (c0002) (0x2013).

2013-07-09T00:41:32.919Z| Worker#1| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk' : Cannot allocate memory (786435).

2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-VMFS  : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1-flat.vmdk" : failed to open (Cannot allocate memory): ObjLib_Open failed. Type 3

2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-LINK  : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk" : failed to open (Cannot allocate memory).

2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-CHAIN : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk" : failed to open (Cannot allocate memory).

2013-07-09T00:41:32.920Z| Worker#1| I120: DISKLIB-LIB   : Failed to open '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' with flags 0xa Cannot allocate memory (786441).

2013-07-09T00:41:32.920Z| Worker#1| I120: DISK: Cannot open disk "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk": Cannot allocate memory (786441).

2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-VMFS  : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager-flat.vmdk" : open successful (10) size = 53687091200, hd = 3565896. Type 3

2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-DSCPTR: Opened [0]: "JD-5-manager-flat.vmdk" (0xa)

2013-07-09T00:41:32.920Z| Worker#1| I120: Msg_Post: Error

2013-07-09T00:41:32.920Z| Worker#1| I120: [msg.literal] Cannot allocate memory

2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' (0xa): vmfs, 104857600 sectors / 50 GB.

2013-07-09T00:41:32.920Z| Worker#1| I120: [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager_1.vmdk' or one of the snapshot disks it depends on.

2013-07-09T00:41:32.920Z| Worker#1| I120: ----------------------------------------

2013-07-09T00:41:32.920Z| Worker#0| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk" (flags 0xa, type vmfs).

2013-07-09T00:41:32.920Z| Worker#0| I120: DISK: Disk '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' has UUID '60 00 c2 9c cf e7 7a d0-c9 a4 29 f2 3f 42 e7 e9'

2013-07-09T00:41:32.920Z| Worker#0| I120: DISK: OPEN '/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager.vmdk' Geo (6527/255/63) BIOS Geo (0/0/0)

2013-07-09T00:41:32.933Z| vmx| I120: DISK: Opening disks took 16 ms.

2013-07-09T00:41:32.933Z| vmx| I120: Module DiskEarly power on failed.

2013-07-09T00:41:32.933Z| vmx| I120: VMX_PowerOn: ModuleTable_PowerOn = 0

2013-07-09T00:41:32.933Z| vmx| I120: scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)

2013-07-09T00:41:32.933Z| vmx| I120: Closing disk scsi0:0

2013-07-09T00:41:32.933Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/51afe5a5-0bf2a06a-2bd9-842b2bfe9c89/JD-5-manager/JD-5-manager-flat.vmdk" : closed.

2013-07-09T00:41:32.937Z| vmx| I120: WORKER: asyncOps=3 maxActiveOps=2 maxPending=0 maxCompleted=2

And the vmdk file of the affected disk:

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=4ed52c7e

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 629145600 VMFS "JD-5-manager_1-flat.vmdk"

# The Disk Data Base

#DDB

ddb.deletable = "true"

ddb.virtualHWVersion = "8"

ddb.longContentID = "8264138edeb9f6fb2afaf3be4ed52c7e"

ddb.uuid = "60 00 C2 98 c9 03 10 f9-19 1a c3 34 a1 ba e3 1c"

ddb.geometry.cylinders = "39162"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.thinProvisioned = "1"

ddb.adapterType = "lsilogic"

As you can see the parentCID is the correct setting for a vmdk with no snapshots.

Any tips for troubleshooting this problem?

Thanks kindly.

0 Kudos
1 Solution

Accepted Solutions
cgillpwe
Contributor
Contributor
Jump to solution

After that prompting I checked the vmkwarning log and found the following:

/var/log # tail vmkwarning.log

2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand

2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai

2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand

2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai

2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

Which lead me to the following VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100442...

Using PowerCLI, I checked the total size of all VMDK files on one member of the cluster with the following command:

PowerCLI > Get-HardDisk -Datastore <datastore> | select -ExpandProperty CapacityGB | measure -sum -ave -max -min

Count    : 78

Average  : 197.730180789263

Sum      : 15422.9541015625

Maximum  : 544.49609375

Minimum  : 50

Property :

According to that article, the maximum supported size of all opened VMDK files was only 8TB, so I was well above that, therefore running into issues.

I've adjusted the VMFS3.MaxHeapSizeMB to the maxmum 256MB to compensate, and that has resolved this issue.

View solution in original post

0 Kudos
2 Replies
admin
Immortal
Immortal
Jump to solution

Recreate the issue and upload the latest vmkernel log and full vmware.log

cgillpwe
Contributor
Contributor
Jump to solution

After that prompting I checked the vmkwarning log and found the following:

/var/log # tail vmkwarning.log

2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:10:13.901Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:10:13.920Z cpu14:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand

2013-07-09T01:10:14.010Z cpu2:325385)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai

2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expand

2013-07-09T01:11:03.238Z cpu0:326782)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fai

2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 2638: Heap vmfs3 already at its maximum size. Cannot expan

2013-07-09T01:19:39.291Z cpu13:327237)WARNING: Heap: 3019: Heap_Align(vmfs3, 2099216/2099216 bytes, 8 align) fa

Which lead me to the following VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100442...

Using PowerCLI, I checked the total size of all VMDK files on one member of the cluster with the following command:

PowerCLI > Get-HardDisk -Datastore <datastore> | select -ExpandProperty CapacityGB | measure -sum -ave -max -min

Count    : 78

Average  : 197.730180789263

Sum      : 15422.9541015625

Maximum  : 544.49609375

Minimum  : 50

Property :

According to that article, the maximum supported size of all opened VMDK files was only 8TB, so I was well above that, therefore running into issues.

I've adjusted the VMFS3.MaxHeapSizeMB to the maxmum 256MB to compensate, and that has resolved this issue.

0 Kudos