VMware

This Question is Answered

1 2 3 Previous Next 34 Replies Last post: Jul 27, 2009 11:46 AM by HMC-Frank   Go to original post

Re: The shit has hit the fan

15. Jan 9, 2009 9:39 AM in response to: charlesleaver…
Click to view Dave.Mishchenko's profile Guru
Dave.Mishchen…
8,943 posts since
Nov 15, 2005
Have you tried running esxtop at the console to see if any process is using too much memory?

Re: The shit has hit the fan

16. Jan 9, 2009 10:05 AM in response to: Dave.Mishchen…
Click to view Bolgard's profile Hot Shot
Bolgard
113 posts since
Aug 11, 2007
No, I'll run that and memtest tomorrow when I can access the server physically again. Get back to you then. And charlesleaverdd can't because he can't access the ESXi host at all. Are you guys on any IRC-channel I can join?

Re: The shit has hit the fan

17. Jan 9, 2009 10:43 AM in response to: Bolgard
Click to view Dave.Mishchenko's profile Guru
Dave.Mishchen…
8,943 posts since
Nov 15, 2005
I'm not on IRC but you can send me a PM as I would be interested in your results.

Re: The shit has hit the fan

18. Jan 10, 2009 8:12 AM in response to: Dave.Mishchen…
Click to view Bolgard's profile Hot Shot
Bolgard
113 posts since
Aug 11, 2007
Ok, so I can now physically access the server. I pressed Alt+F1, and first thing that met was a screen with the same message repeated:

"/etc/init.d/sfcbd-watchdog: /etc/init.d/sfcbd-watchdog: 309: Cannot fork"

"sh: 0: unknown operand"

I then typed "unsupported" and pressed Enter. Continuously the above messages keep popping up. At the login prompt a new message is shown:

"failure forking: Cannot allocate memory"

I entered my password, but I can't seem to access bash (and thus cannot enter any commands).

I'll have to do a reboot...

EDIT: Rebooted. Everything is looking fine now. Too bad I couln't view the esxtop when I had the problem. At the unsupported console there keep popping up messages saying "child still alive with a status of 0" and then on the next row "Killed". I don't know if this is normal. I have also enabled SSH-access to be able to check settings remotely next time. I'll take it offline for memtest now, we'll see what that will tell us.

EDIT2: Memtest have completed a pass now. "Pass complete, no errors, press Esc to exit". So problem is not with the RAM. I'm starting to believe this is a bug...

Re: The shit has hit the fan

19. Jan 10, 2009 7:59 PM in response to: Bolgard
Click to view Dave.Mishchenko's profile Guru
Dave.Mishchen…
8,943 posts since
Nov 15, 2005
Do you have the Linux version of the RCLI (either the install or appliance)? It will have resxtop and I would suggest starting that up early this week and leaving it running to see what happens with memory. What build of ESXi do you have?

Re: The shit has hit the fan

20. Jan 12, 2009 11:05 AM in response to: Dave.Mishchen…
Click to view Bolgard's profile Hot Shot
Bolgard
113 posts since
Aug 11, 2007

I had a VM converted from VMware Server into VMware ESXi. When it was running on VMware Server I installed VMware Tools on it. I then never uninstalled and re-installed VMware Tools when I moved it to ESXi. Could the problem have been caused by this VM and it VMware Tools that belonged to VMware Server?

I have now uninstalled and re-installed VMware Tools for ESXi. Problem haven't reoccured yet (though it usually takes a few weeks). I don't have a RCLI up and running now, but if it happens again I will set it up. It works fine to SSH into ESXi and run esxtop from there, and everything looks normal.

I'm running VMware ESX Server 3i, 3.5.0, 123629 according to VMware Infrastructure Client.


Re: The shit has hit the fan

21. Jan 12, 2009 11:05 PM in response to: Bolgard
Click to view Dave.Mishchenko's profile Guru
Dave.Mishchen…
8,943 posts since
Nov 15, 2005
It may have been something like that. If you run esxtop on a daily basis you may be able to see if some process is slowly using more and more memory.

Re: The shit has hit the fan

22. Jan 18, 2009 11:06 PM in response to: Bolgard
Click to view charlesleaverdd's profile Novice
charlesleaver…
6 posts since
Aug 7, 2008


EDIT2: Memtest have completed a pass now. "Pass complete, no errors, press Esc to exit". So problem is not with the RAM. I'm starting to believe this is a bug...

Wow dude that's pretty serious and incredibly annoying. I'm having to go to a lot of trouble to try and get these virtual machines off my ESXi box so that I can bounce it. I almost hoped that it was due to faulty RAM, just so the issue would be solved. Are you sure you memtested for long enough? I was planning on running it for like a week or more even. Cos the RAM doesn't necessarily illustrate its flaw immediately, does it?

By the way I also get this: "failure forking: Cannot allocate memory".

I'm amazed that all the virtual machines are 100% fine. Including the one that's hammering the box.

Re: The s***has hit the fan

23. Jul 4, 2009 1:11 AM in response to: charlesleaver…
Click to view charlesleaverdd's profile Novice
charlesleaver…
6 posts since
Aug 7, 2008
Oh dear. How did we overlook this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007507 ?


... and I quote: "A memory corruption condition might occur in the virtual machine hardware. A malicious request sent from the guest operating system to the virtual hardware might cause the virtual hardware to write to uncontrolled physical memory." which is the first issue that the advisory mentions will be solved by the update. I had not done the update yet because I'm not able to bounce that box very easily due to change control etc.

Today the box finally gave in. Boom. Four notifications in my mailbox with the dreaded "Host DOWN alert for" in their subject. On arrival at the box I found this on the screen. I then hard powered the box off, powered back on, booted normally, and then everything that was meant to auto-start did exactly that and everything returned to normal.

As far as I'm concerned my issue is definitely caused by the fact that I have not updated. I'm not going to chase this anymore. I'll change my mind if after the upgrade I experience the same issue.

Re: The s*** has hit the fan

24. Jul 4, 2009 1:09 AM in response to: charlesleaver…
Click to view RS_1's profile Enthusiast
RS_1
58 posts since
Nov 3, 2006
I got the same problem on an IBM 3850 M2 ESX 3i 3.5.0 130755 (up to date) :

vmkernel: 27:18:33:59.650 cpu2:1376)WARNING: Heap: 1397: Heap globalCartel already at its maximumSize. Cannot expand.
vmkernel: 27:18:33:59.650 cpu2:1376)WARNING: Heap: 1522: Heap_Align(globalCartel, 48/48 bytes, 4 align) failed. caller: 0x73a8ce
vmkernel: 27:18:33:59.650 cpu2:1376)WARNING: World: vm 11666870: 910: init fn user failed with: Out of memory!
vmkernel: 27:18:33:59.650 cpu2:1376)WARNING: World: vm 11666870: 1775: WorldInit failed: trying to cleanup.
inetd1370: fork: Cannot allocate memory
vmkernel: 27:18:33:54.452 cpu6:1370)WARNING: Heap: 1397: Heap globalCartel already at its maximumSize. Cannot expand.
vmkernel: 27:18:33:54.452 cpu6:1370)WARNING: Heap: 1522: Heap_Align(globalCartel, 48/48 bytes, 4 align) failed. caller: 0x73a8ce
vmkernel: 27:18:33:54.452 cpu6:1370)WARNING: World: vm 11675061: 910: init fn user failed with: Out of memory!
vmkernel: 27:18:33:54.452 cpu6:1370)WARNING: World: vm 11675061: 1775: WorldInit failed: trying to cleanup.

Re: The s*** has hit the fan

25. Jul 4, 2009 1:10 AM in response to: charlesleaver…
Click to view Bolgard's profile Hot Shot
Bolgard
113 posts since
Aug 11, 2007
charlesleaverdd: That patch is for the whole ESXi, right? Did you install the patch or just update the VMware Tools on the guests? My problem seems to be gone after updating VMware Tools on the guest.

Re: The s*** has hit the fan

26. Jul 4, 2009 1:10 AM in response to: Bolgard
Click to view 3sh's profile Enthusiast
3sh
35 posts since
Feb 28, 2008
have you run any diagnostics on your drives?


I assume the adaptec has some utilities you can use the check the health of the drives and raid array.

Re: The s*** has hit the fan

27. Jul 4, 2009 12:03 AM in response to: 3sh
Click to view Bolgard's profile Hot Shot
Bolgard
113 posts since
Aug 11, 2007
Thanks for your interest, but the problem is solved when updating VMware Tools.

I have currently NO way of checking the RAID array, because VMware's idea of I/O-compability does not cover monitoring and management ;-) I recommend people to check that the RAID card they're buying is actually monitorable in VMware ESXi!

Re: The s*** has hit the fan

28. Jul 4, 2009 12:02 AM in response to: Bolgard
Click to view StuartLittle's profile Novice
StuartLittle
24 posts since
Oct 29, 2008
I've been reading your thread guys and have the exact same issue/log messages on our Dell PowerEdge 1950III server. Your post explained the exact scenario our server is in at the moment (follow thread http://communities.vmware.com/thread/203957?start=15&tstart=0)

Bolgard: can you confirm how you updated the VMware Tools please and what version? Did you simply use the VI Client to update/reinstall VMware Tools on all your guest VM's when your host was fully operational again or did you need to download the latest VMware Tools from VMware's website and manually run on the guest VM's?!


Reason i'm asking is I would have thought updating VMware Tools from the VI Client would simply just reinstall the same version of the VMware tools on the guest VM's and not technically fix the issue in the long term.


Thanks for your help and for making this post (and to the other posters like charelesleaverdd) as it's been such a relief to see someone else has experienced the pain and sheer terror of their host causing problems! (not saying I enjoy reading about other people's pain - just that it's good to know i'm not alone on this issue).

Re: The s*** has hit the fan

29. Jul 4, 2009 12:32 AM in response to: StuartLittle
Click to view J1mbo's profile Expert
J1mbo
565 posts since
May 20, 2009

Given the portability of VMs between hardware, I would approach this by moving all VMs elsewhere and then using the manufacturers own diagnostics - destructive if necessary. Also Microsoft have a particularly good memory diagnostic utility available on the Windows 7 CD (boot to recovery mode) or downloadable from http://oca.microsoft.com/en/windiag.asp.

As an aside I would advise against 3rd party components in a production server. The cost of downtime will far outweigh capital savings.

1 2 3 Previous Next Go to original post

Actions

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities