Heap_Align?

[krc1517]

I have 2 dl585s that experienced some issues over the weekend. After looking through some of the logs, i see the following repeatedly on both servers.

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.119 cpu2:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.119 cpu2:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

I've uninstalled and reinstalled the HP 7.7.0 agents for VI3 and chose not to install the storage agents this time. Still, same errors.

I can't find any info on this. Anyone else have any thoughts?

server - dl585

san - eva8k

qlocic hbas

hp 7.7 agents

VMware ESX Server 3.0.1 build-32039 on both

=--SERVER VMHOST43 --=

\[root@vmhost43 770]# tail -25 /var/log/vmkernel

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.119 cpu2:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.119 cpu2:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.126 cpu1:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.126 cpu1:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.129 cpu6:1037)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.129 cpu6:1037)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.133 cpu6:1037)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.133 cpu6:1037)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.325 cpu0:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.325 cpu0:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.329 cpu2:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.329 cpu2:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.588 cpu2:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.588 cpu2:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.592 cpu7:1037)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:56 vmhost43 vmkernel: 60:19:59:42.592 cpu7:1037)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:57 vmhost43 vmkernel: 60:19:59:42.885 cpu3:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:57 vmhost43 vmkernel: 60:19:59:42.885 cpu3:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:54:57 vmhost43 vmkernel: 60:19:59:42.889 cpu3:1038)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (266240)

Jun 25 10:54:57 vmhost43 vmkernel: 60:19:59:42.889 cpu3:1038)WARNING: Heap: 1370: Heap_Align(vmfs3, 262144/262144 bytes, 4 align) failed. caller: 0x8ffc9e

Jun 25 10:55:57 vmhost43 vmkernel: 60:20:00:43.710 cpu0:1044)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jun 25 10:55:57 vmhost43 vmkernel: 60:20:00:43.710 cpu3:1044)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jun 25 10:55:57 vmhost43 vmkernel: 60:20:00:43.710 cpu0:1044)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

Jun 25 10:55:57 vmhost43 last message repeated 3 times

Jun 25 10:55:57 vmhost43 vmkernel: 60:20:00:43.711 cpu0:1044)WARNING: SCSI: 1731: Unexpected status returned: bad000a I/O error

=SERVER VMHOST47=

\[root@vmhost47 770]# tail -25 /var/log/vmkernel

Jun 25 11:05:41 vmhost47 vmkernel: 60:18:30:02.527 cpu3:1284)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:02.857 cpu0:1207)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:02.857 cpu0:1207)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.026 cpu2:1237)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.026 cpu2:1237)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.170 cpu6:1325)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.170 cpu6:1325)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.258 cpu2:1237)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.258 cpu2:1237)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.258 cpu1:1207)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.258 cpu1:1207)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.445 cpu2:1227)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.445 cpu2:1227)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.527 cpu3:1284)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:42 vmhost47 vmkernel: 60:18:30:03.527 cpu3:1284)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:03.857 cpu1:1207)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:03.857 cpu1:1207)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.026 cpu2:1237)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.026 cpu2:1237)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.172 cpu6:1325)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.173 cpu6:1325)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.258 cpu3:1237)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.258 cpu3:1237)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.258 cpu1:1207)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (36864)

Jun 25 11:05:43 vmhost47 vmkernel: 60:18:30:04.258 cpu1:1207)WARNING: Heap: 1370: Heap_Align(vmfs3, 33008/33008 bytes, 4 align) failed. caller: 0x8fc3a4

Thanks,

KC

[krc1517]

anyone?

After doing some research and changing the keywords used to search, it looks like it could just be a simple error that didn't cause my issues.

the only problem is, it happens so much, it is hard to tell what actually caused the problems from this weekend.

[krc1517]

bump

[krc1517]

ttt

[grasshopper]

Hey K... sry nobody caught your msg sooner. This is a pretty exotic error, so I probably cannot help. However, we can get some information on the table to help identify the issue.

I have 2 dl585s that experienced some issues over the

weekend.

1. Can you please elaborate what specifically happened? Power outage? San issue?

2. What actions did you take besides reinstalling the agents?

3. Any noticeable symptoms in the VMs/Guests since the issue occurred?

4. Any VMs on local disk or all on SAN?

5. Size of LUNs?

6. Pathing policy (MRU or Fixed)?

7. Do you use any Spanning/Extents

8. Was this an upgrade from 2.x? If so, any hiccups along the way?

9. Do you have a support subscription with VMware?

[krc1517]

woohoo.... a reply.

1. Can you please elaborate what specifically happened? Power outage? San issue?

\- No power outages. It appeared that my /var/log filled up and that's what prompted me to start looking around.

2. What actions did you take besides reinstalling the agents?

\- Reinstalled the agents minus the storage agents. still no difference.

3. Any noticeable symptoms in the VMs/Guests since the issue occurred?

\- VMs run fine. Since we have frequent problems with these VMs though (unrelated to this issue) we run into many more issues on these servers than the other 40 or so ESX hosts

that we have.

4. Any VMs on local disk or all on SAN?

\- Nope. All SAN attached.

5. Size of LUNs?

-15GB system drives on a 500GB LUN, and 2TB Data drives. VMs are used for desktop backup servers. There are about 40VMs on these 2 systems. All configured the same. 15GB C:\ and 2TB (4x500GB) data drives in windows.

6. Pathing policy (MRU or Fixed)?

-Single path since it's for desktop backups. MPIO not needed.

7. Do you use any Spanning/Extents

\- Nope

8. Was this an upgrade from 2.x? If so, any hiccups along the way?

\- Clean install to 3.0.1

9. Do you have a support subscription with VMware?

\- Of course I do but what fun would that be?

This stuff writes so much it fills up my /var/log partition every few weeks. 4GBs of logs.

[grasshopper]

Here ya go bud. It's a known bug:

https://www.vmware.com/community/thread.jspa?threadID=65775&start=0&tstart=0

[krc1517]

Thanks Grasshopper. I must have searched using the wrong terms but searching for heap usually ends up being my thread.

I'm not sure it's the same exact error but is probably very similar. I guess I better get VMware to have a look since my backup will need to run the servers for 3 weeks while i'm out.

I'll post the results/outcome after speaking with Vmware.

[grasshopper]

you should consider adding the appropriate patches for 3.0.1 or upgrade to 3.0.2 which was released last night.

[krc1517]

Finally got around to calling Vmware on this. I was told that there was some memory leak within the LVM that caused this error to repeatedly come up.

There isn't a bug number for it that the tech could find but he thought it would be fixed by 3.0.2 possibly.

Restarting vpxa didn't help

restarting mgmt-vmware didn't help.

After a reboot of the VI3 host, everything seems to be working much better.

Althought the error has started to resurface again after a few weeks.

An upgrade to 3.0.2 is in the cards for the near future.

[grasshopper]

excellent... thanks for the info!

[jade_]

In my experience with this error, it was related to exceeding the set value for LVM heap due to some unique configurations of allocated storage to a number of VM's (19.5T split between 15 VM's across 4 physicals) .

VMFS3 has a max heap size right now of 16MB. One of the uses of the heap in the VMFS3 module is to cache vmdk file locations (pointer blocks) on disk. Each pointer block is 4 Kilobytes in size. So the max amount of storage any number of powered on VM's on a single ESX server could address with a 16MB VMFS3 heap size is ~4TB.

If you are still experiencing the issue, try to power down some of the VM's, or migrate to an alternate host and see if they continue to pop up.

There is supposed to be a GA release that addresses this up and coming, but I have not received word on the release date yet.