2008 R2 VM's unstable after upgrade to 32gb RAM

Dalsia · ‎05-20-2012

Hi there,

I have recently upgraded my ESXi 5 test lab from 16gb of RAM to 32gb of RAM, I have gone from 4x4gb sticks to 4x8gb sticks. After the upgrade I have now started having performance issues with my virtual machines to point of them crashing constantly. All VMs are server 2008 R2 SP1 servers.

I had initially suspected bad RAM so as part of my troubleshooting I removed 2 of the 8gb sticks and put the 2 of 4gb sticks back in place (total of 24gb of RAM) and everything went back to running smoothly. After this i have then swapped the 2 8gb sticks around to test the 2 previously removed and again everything was running smoothly. This leads me to believe the RAM is fine and that there is something else at play here.

I havent got the error message on me at the moment as i am not near the ESXi 5 server, however the error message didnt seem overly useful anyway. I can get the error message in a couple of hours once i am near the ESXi 5 server again if required.

I am not overly familiar with ESXi 5 logs so I am not sure which logs to be going through to check for messages, however happy to include some log info as well once i have access to the ESXi 5 server, just let me know which ones

Any assistance would be greatly appreciated as currently 2 of my 8gb RAM sticks are useless since I cant put them in the ESXi 5 server.

Dave_Mishchenko · ‎05-20-2012

Welcome to the VMware Communities forums. I'd start with a look at the log file for the VMs that have had an issue. If you look at the folder for the VM, you'll see a number of vmware.log and vmware-X.log files. I'd also suggest running a memory check tool like http://www.memtest86.com/.

Dalsia · ‎05-20-2012

Thanks Dave, will take a look at those logs as soon as i can and post the info.

I have a small issue with the memtest though, this server no longer has a graphics card in it so i havent been able to plug in a display to be able to boot into a memtest. I have been trying to source one to do a proper memtest but havent had much luck yet, will probably have to go buy a cheap one and install it soon so i can do a proper test.

I am not sure if this is a legitimate test or not but i did do a a memtest inside one of the VM's. I had allocated it the whole 32gb of RAM and booted into the memtest86+ iso and let that run. This did not produce any RAM faults at all, however the VM suffered the same fate in that it would crash after 5-10 minutes of running. I can try this again with just the individual 8gb sticks in one at a time, which I have been meaning to do, but am still unsure if this is a legitimate way to test?

a_p_ · ‎05-20-2012

Assuming the RAM itself is ok, please double check whether the hardware requires a special placement of the memory modules. It's also important to check the compatibility of the modules against your system (2- rank, 4-rank, speed, ...)

André

Dalsia · ‎05-21-2012

Ok some more information at this stage:

Running ESXi 5.0.0 702118

I've attached a screenshot of the error message i am getting within the VSphere Client

The last couple of minutes from the vmware.log file before the crash are as follows:

2012-05-21T08:51:41.642Z| mks| MKS enabling SVGA
2012-05-21T08:52:34.953Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 0
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 30
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 31
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 32
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 33
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 35
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 36
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 38
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 42
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 43
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 45
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 46
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 47
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 64
2012-05-21T08:52:34.954Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 65
2012-05-21T08:52:34.955Z| vcpu-0| CDROM: Emulate GET CONFIGURATION RT 2 starting feature 259
2012-05-21T08:53:01.044Z| vmx| GuestRpcSendTimedOut: message to toolbox timed out.
2012-05-21T08:53:01.044Z| vmx| Vix: [3520 guestCommands.c:2194]: Error VIX_E_TOOLS_NOT_RUNNING in VMAutomationTranslateGuestRpcError(): VMware Tools are not running in the guest
2012-05-21T08:53:03.595Z| vcpu-0| Balloon: Start: vmmemctl reset balloon
2012-05-21T08:53:03.595Z| vcpu-0| Balloon: Reset (n 1 pages 0)
2012-05-21T08:53:03.595Z| vcpu-0| Balloon: Reset: nUnlocked 0 size 0
2012-05-21T08:53:17.941Z| vcpu-0| GuestRpc: Channel 0, guest application toolbox.
2012-05-21T08:53:18.011Z| vcpu-0| TOOLS ToolsCapabilityGuestTempDirectory received 1 C:\Windows\TEMP\vmware-SYSTEM
2012-05-21T08:53:18.012Z| vcpu-0| TOOLS soft reset detected.
2012-05-21T08:53:18.012Z| vcpu-0| TOOLS autoupgrade protocol version 2
2012-05-21T08:53:18.013Z| vcpu-0| TOOLS ToolsCapabilityGuestConfDirectory received C:\ProgramData\VMware\VMware Tools
2012-05-21T08:53:18.013Z| vcpu-0| TOOLS Received tools.set.version rpc call, version = 8389.
2012-05-21T08:53:18.013Z| vcpu-0| ToolsSetVersionWork did nothing; new tools version (8389) matches old Tools version
2012-05-21T08:53:18.013Z| vcpu-0| TOOLS Setting autoupgrade-checked FALSE.
2012-05-21T08:53:18.013Z| vcpu-0| Starting copy guest manifest.
2012-05-21T08:53:18.249Z| vcpu-0| Compared tools manifest from host and from the guest. Status = 3.
2012-05-21T08:53:18.249Z| vcpu-0| VMXVmdb_SetToolsVersionStatus: status value set to 'ok', 'current', install possible
2012-05-21T08:53:18.249Z| vcpu-0| TOOLS installed legacy version 8389, available legacy version 8389
2012-05-21T08:53:18.249Z| vcpu-0| TOOLS manifest update status is 3
2012-05-21T08:53:18.249Z| vcpu-0| TOOLS can be autoupgraded.
2012-05-21T08:53:18.249Z| vcpu-0| TOOLS Setting autoupgrade-checked TRUE.
2012-05-21T08:53:18.249Z| vcpu-0| TOOLS unified loop capability requested by 'toolbox'; now sending options via TCLO
2012-05-21T08:53:18.250Z| vcpu-0| Guest: toolbox: Version: build-652272
2012-05-21T08:54:10.434Z| vcpu-1| MONITOR PANIC: vcpu-0:VMM fault 13: src=MONITOR rip=0xfffffffffc283c2a regs=0xfffffffffc011ba0
2012-05-21T08:54:10.434Z| vcpu-1| Core dump with build build-702118
2012-05-21T08:54:10.434Z| vcpu-0| Exiting vcpu-0
2012-05-21T08:54:10.434Z| vcpu-1| Writing monitor corefile "/vmfs/volumes/4de67ffa-513d052a-4f8a-001b21ad8a94/CDC01/vmmcores.gz"
2012-05-21T08:54:10.742Z| vcpu-1| Saving anonymous memory
2012-05-21T08:54:10.755Z| vcpu-1| Dumping core for vcpu-0
2012-05-21T08:54:10.755Z| vcpu-1| CoreDump: dumping core with superuser privileges
2012-05-21T08:54:10.755Z| vcpu-1| VMK Stack for vcpu 0 is at 0x412217041000
2012-05-21T08:54:10.755Z| vcpu-1| Beginning monitor coredump
2012-05-21T08:54:12.295Z| vcpu-1| End monitor coredump
2012-05-21T08:54:12.296Z| vcpu-1| Dumping core for vcpu-1
2012-05-21T08:54:12.296Z| vcpu-1| CoreDump: dumping core with superuser privileges
2012-05-21T08:54:12.296Z| vcpu-1| VMK Stack for vcpu 1 is at 0x4122170c1000
2012-05-21T08:54:12.296Z| vcpu-1| Beginning monitor coredump
2012-05-21T08:54:13.684Z| vcpu-1| End monitor coredump
2012-05-21T08:54:13.685Z| vcpu-1| Dumping extended monitor data
2012-05-21T08:54:16.071Z| vmx| GuestRpcSendTimedOut: message to toolbox timed out.
2012-05-21T08:54:20.554Z| vcpu-1| Msg_Post: Error
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic] *** VMware ESX internal monitor error ***
2012-05-21T08:54:20.554Z| vcpu-1| --> vcpu-0:VMM fault 13: src=MONITOR rip=0xfffffffffc283c2a regs=0xfffffffffc011ba0
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic.report] You can report this problem by selecting menu item Help > VMware on the Web > Request Support, or by going to "http://vmware.com/info?id=8&logFile=%2fvmfs%2fvolumes%2f4de67ffa%2d513d052a%2d4f8a%2d001b21ad8a94%2f...". Provide the log file (/vmfs/volumes/4de67ffa-513d052a-4f8a-001b21ad8a94/CDC01/vmware.log) and the core file(s) (/vmfs/volumes/4de67ffa-513d052a-4f8a-001b21ad8a94/CDC01/vmmcores.gz, /vmfs/volumes/4de67ffa-513d052a-4f8a-001b21ad8a94/CDC01/vmx-zdump.001).
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic.serverdebug] If the problem is repeatable, set 'Use Debug Monitor' to 'Yes' in the 'Misc' section of the Configure Virtual Machine Web page. Then reproduce the incident and file it according to the instructions.
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic.vmSupport.vmx86] To collect data to submit to VMware support, run "vm-support".
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic.entitlement] We will respond on the basis of your support entitlement.
2012-05-21T08:54:20.554Z| vcpu-1| [msg.log.monpanic.finish] We appreciate your feedback,
2012-05-21T08:54:20.554Z| vcpu-1| --> -- the VMware ESX team.
2012-05-21T08:54:20.554Z| vcpu-1| ----------------------------------------

admin · ‎05-21-2012

Hi Dalsia

Welcome to the communites

If problem comes after upgrading please check the You have fixed all those RAM in proper slots .

and if possiable please remove old RAM and check it through new RAM to make sure

there is no compatibility issuse.

"Life is never easy for those who dream"