VMware Cloud Community
here4real
Contributor
Contributor

8 of my VMs powered off last night

Very bizarre...  8 VMs spread across 2 ESXs powered off last night...  Other VMs on those 2 ESX had nothing happen to them...  A 3rd ESX was not effected either...  In guest OS Event Log, it says Unexpected Shutdown...    In vSphere Infrastructure Client, it shows the tasks when I started them back up but I don't see any power offs...  How else could they have powered off and are there any logs that might show it???  I was thinking maybe from the ESX console???

In 2 1/2 years nothing like this has ever happened...

Thank you...

Reply
0 Kudos
7 Replies
MauroBonder
VMware Employee
VMware Employee

Hello,

You just checked a event log of VMware if had any problem with Hardware ? Fail of hardware ?

Space on datastore ?

Do you´r using HA ?

Please, don't forget the awarding points for "helpful" and/or "correct" answers.


Mauro Bonder - Moderator

LinkedIn Facebook
*Please, don't forget the awarding points for "helpful" and/or "correct" answers. *Por favor, não esqueça de atribuir os pontos se a resposta foi útil ou resolveu o problema.* Thank you/Obrigado
Reply
0 Kudos
a_p_
Leadership
Leadership

That's really interesting. Take a look at the vmware*.log files in the VM's folder (or zip/attach them to your next post). Maybe they contain hints on what happened!?

André

Reply
0 Kudos
firestartah
Virtuoso
Virtuoso

The 8 machines that powered off, are they the same OS or different ones? Maybe a windows update rebooted them or some kind of scheduled maintenance task?

Gregg

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful". Gregg http://thesaffageek.co.uk
Reply
0 Kudos
here4real
Contributor
Contributor

We thought of that...

1) It wouldn't have left it in a powered off state...  It was have restarted the VM...

2) 2 of the VMs are running Novell...

Reply
0 Kudos
here4real
Contributor
Contributor

Very good idea...

This is the pertinent part:

Nov 30 17:21:09.882: vmx| GuestRpcSendTimedOut: message to toolbox timed out.
Nov 30 19:07:25.350: vmx| GuestRpcSendTimedOut: message to toolbox timed out.
Nov 30 19:07:40.350: vmx| GuestRpcSendTimedOut: message to toolbox timed out.
Nov 30 19:07:40.350: vmx| GuestRpc: app toolbox's second ping timeout; assuming app is down

Nov 30 19:20:54.676: vcpu-0| Writing monitor corefile "/vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMBM2/vmware-core.gz"
Nov 30 19:20:55.238: vcpu-0| CoreDump: dumping core with superuser privileges
Nov 30 19:20:57.142: vcpu-0| VMK Stack for vcpu 0 is at 0xc0000000
Nov 30 19:20:57.142: vcpu-0| Saving busmem frames
Nov 30 19:20:57.147: vcpu-0| Saving anonymous memory
Nov 30 19:20:57.151: vcpu-0| Beginning monitor coredump
Nov 30 19:20:57.469: vcpu-0| End monitor coredump
Nov 30 19:20:57.470: vcpu-0| Beginning extended monitor coredump
Nov 30 19:20:57.471: vcpu-0| Writing anonymous pages at pos: 408000
Nov 30 19:21:20.275: vcpu-0| Msg_Post: Error
Nov 30 19:21:20.276: vcpu-0| [msg.log.monpanic] *** VMware ESX internal monitor error ***
Nov 30 19:21:20.276: vcpu-0| vmk: vcpu-0:Unable to read swapped out pgNum(0x2244d) from swap slot(0x100587a7) for VM(65128)
Nov 30 19:21:20.277: vcpu-0| [msg.log.monpanic.report] Please report this problem by selecting menu item Help > VMware on the Web > Request Support, or by going to the Web page "http://www.vmware.com/info?id=8&logFile=vmware%2elog&coreLocation=%2fvmfs%2fvolumes%2f4e32f2f1%2d423...". Please provide us with the log file (vmware.log) and the core file(s) (/vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMBM2/vmware-core.gz, /vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMBM2/vmware-vmx-zdump.000).
Nov 30 19:21:20.277: vcpu-0| [msg.log.monpanic.serverdebug] If the problem is repeatable, please set 'Use Debug Monitor' to 'Yes' in the 'Misc' section of the Configure Virtual Machine Web page. Then reproduce the incident and file it according to the instructions.
Nov 30 19:21:20.278: vcpu-0| [msg.log.monpanic.vmSupport.vmx86] To collect data to submit to VMware support, run "vm-support".
Nov 30 19:21:20.278: vcpu-0| [msg.log.monpanic.entitlement] We will respond on the basis of your support entitlement.
Nov 30 19:21:20.279: vcpu-0| [msg.log.monpanic.finish] We appreciate your feedback,
Nov 30 19:21:20.279: vcpu-0|   -- the VMware ESX team.
Nov 30 19:21:20.279: vcpu-0| ----------------------------------------
Nov 30 19:21:20.284: vcpu-0| Exiting vcpu-0
Nov 30 19:21:21.281: vmx| VTHREAD watched thread 4 "vcpu-0" died
Nov 30 19:21:21.382: mks| VTHREAD watched thread 0 "vmx" died

Reply
0 Kudos
here4real
Contributor
Contributor

That last post was from the log on a Novell VM...

This one is from Windows VM...  Same thing...

Nov 30 19:18:38.093: vcpu-0| Writing monitor corefile "/vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMWIFI/vmware-core.gz"
Nov 30 19:18:38.324: vcpu-0| CoreDump: dumping core with superuser privileges
Nov 30 19:18:43.555: vcpu-0| VMK Stack for vcpu 0 is at 0xc0000000
Nov 30 19:18:43.587: vcpu-0| Saving busmem frames
Nov 30 19:18:43.590: vcpu-0| Saving anonymous memory
Nov 30 19:18:43.597: vcpu-0| Beginning monitor coredump
Nov 30 19:18:43.913: vcpu-0| End monitor coredump
Nov 30 19:18:43.914: vcpu-0| Beginning extended monitor coredump
Nov 30 19:18:43.915: vcpu-0| Writing anonymous pages at pos: 408000
Nov 30 19:19:11.794: vcpu-0| Msg_Post: Error
Nov 30 19:19:11.800: vcpu-0| [msg.log.monpanic] *** VMware ESX internal monitor error ***
Nov 30 19:19:11.800: vcpu-0| vmk: vcpu-0:Unable to read swapped out pgNum(0x81fa7) from swap slot(0x10080abc) for VM(72286)
Nov 30 19:19:11.800: vcpu-0| [msg.log.monpanic.report] Please report this problem by selecting menu item Help > VMware on the Web > Request Support, or by going to the Web page "http://www.vmware.com/info?id=8&logFile=vmware%2elog&coreLocation=%2fvmfs%2fvolumes%2f4e32f2f1%2d423...". Please provide us with the log file (vmware.log) and the core file(s) (/vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMWIFI/vmware-core.gz, /vmfs/volumes/4e32f2f1-4234cf04-080e-e61f135570fb/JMWIFI/vmware-vmx-zdump.000).
Nov 30 19:19:11.800: vcpu-0| [msg.log.monpanic.serverdebug] If the problem is repeatable, please set 'Use Debug Monitor' to 'Yes' in the 'Misc' section of the Configure Virtual Machine Web page. Then reproduce the incident and file it according to the instructions.
Nov 30 19:19:11.800: vcpu-0| [msg.log.monpanic.vmSupport.vmx86] To collect data to submit to VMware support, run "vm-support".
Nov 30 19:19:11.801: vcpu-0| [msg.log.monpanic.entitlement] We will respond on the basis of your support entitlement.
Nov 30 19:19:11.801: vcpu-0| [msg.log.monpanic.finish] We appreciate your feedback,
Nov 30 19:19:11.801: vcpu-0|   -- the VMware ESX team.
Nov 30 19:19:11.801: vcpu-0| ----------------------------------------
Nov 30 19:19:11.805: vcpu-0| Exiting vcpu-0
Nov 30 19:19:12.803: vmx| VTHREAD watched thread 4 "vcpu-0" died
Nov 30 19:19:12.904: mks| VTHREAD watched thread 0 "vmx" died

Reply
0 Kudos
a_p_
Leadership
Leadership

Looks like the shutdowns happened at almost the same time!? Maybe there was an an issue with the storage access!? Does this show any logs?

Are there any other "unusual" entries in the vmware.log before the dump was created?

Anyway, if you do have active SnS, I'd recoomend you open a call with VMware.

André

Reply
0 Kudos