VMware Cloud Community
tilman201110141
Contributor
Contributor

ESXi 4.1 Update 3 panics CentOS 6 guests

I'm running VMware vSphere 4 Essentials on three almost identically configured hosts with a mix of guests including Windows 2003 and 2008 as well as CentOS 5 and 6.

Recently I installed Update 3 on one of the hosts. The updated host runs Windows and CentOS 5 (kernel 2.6.18-308.13.1.el5 i686) VMs fine, but two CentOS 6 VMs (kernel 2.6.32-279.5.2.el6.x86_64) I tried refuse to come up. They display the GRUB menu just fine, but the kernel panics immediately with "early exception 0d". After moving the VM back to the host without update 3 it comes up fine again.

Any ideas how to fix or even start debugging this?

Btw, all my CentOS 6 VMs are 64 bit. Windows and CentOS 5 VMs come in both 32 and 64 bit flavours. Both 32 and 64 bit Windows VMs run fine on the updated host. So far I haven't tried a 64 bit CentOS 5 VM.

Reply
0 Kudos
11 Replies
tilman201110141
Contributor
Contributor

Here are the actual builds of the two involved hosts:

- the one where the CentOS 6 guests are running fine is advertising VMware ESXi, 4.1.0, 702113

- the one where they are early panicking, ESXi, 4.1.0, 800380

Attached is the vmware.log from an attempt to start one of the guests ending in panic, with debugging enabled. Perhaps someone can see something in it because I can't. Well, of course I see the VGA screenshot with my PANIC message:

Sep 11 17:21:31.498: vmx| PANIC: early exception 0d rip 10:ffffffff81038879 error 0 cr2 0

and also what appears to be the actual crash report immediately before:

Sep 11 17:21:19.628: vcpu-0| RDMSR: unknown MSR[0x1a0] (read as zero): rip=0xffffffff810388db count=1
Sep 11 17:21:19.628: vcpu-0| RDMSR: unknown MSR[0x1a0] (read as zero): rip=0xffffffff810388db count=2
Sep 11 17:21:19.629: vcpu-0| X86Fault_Warning: vmcore/vmm64/cpu/interp.c:427: cs:eip=0x10:0xffffffff81038879 fault=13
Sep 11 17:21:19.632: vcpu-0| Vix: [1125838 vmxCommands.c:9609]: VMAutomation_HandleCLIHLTEvent. Do nothing.
Sep 11 17:21:19.632: vcpu-0| MsgHint: msg.monitorevent.halt (sent)
Sep 11 17:21:19.632: vcpu-0| The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

But why this would happen, why only with a CentOS 6 guest, and why only on ESXi 4.1.0 build 800380

and not on build 702113, I have no idea.

Reply
0 Kudos
tilman201110141
Contributor
Contributor

New findings:

1. I updated another host and this one runs CentOS 6 guests fine. The key difference seems to be the processor. The host which runs CentOS 6 guests fine has a Xeon E3-1230 while the one where they panic has a Xeon E3-1270V2.

2. The affected guests are all fully updated CentOS 6.3 with kernel 2.6.32-279.2.1.el6.x86_64. An older VM that was still at CentOS 6.2 with kernel 2.6.32-220.7.1.el6.x86_64 comes up fine but the keyboard displays a panic blink pattern (CapsLock and ScrollLock blinking in unison) even though there is no actual kernel panic.

So the trigger for the early exception panic appears to be:

ESXi build 800380 *and* Xeon E3-1270V2 *and* CentOS 6.3

Reply
0 Kudos
admin
Immortal
Immortal

Hi   tilman2011101,

Welcome to the communities .

This is interesting which is vary person to person .

I will suggest to install one by one hotfix which will give exact idea which one is the exact cause .

"Life is never easy for those who dream"
Reply
0 Kudos
tilman201110141
Contributor
Contributor

Hi aarav,

thanks for your reply and advice.

I'm afraid I don't see how I can install hotfixes one by one.

I have used Update Manager to install ESXi410-Update03.

Now I see neither a way to uninstall that again, nor to install or uninstall parts of it individually.

Regards,

Tilman

Reply
0 Kudos
TilmanS
Contributor
Contributor

(Please don't be confused by the changed name. I am the original poster, but for some reason the forum software has forced me to choose a new screen name.)

I have now updated all three of my hosts to ESXi 4.3 Update 3 (build 800380) and only the one with the E3-1270V2 processor panics CentOS 6 guests. (I'm not even 100% sure anymore if they ran fine there before the update, and of course it's impossible to check now.)

The following matrix sums up the results of my research:

ProcessorE5620E3-1230E3-1270V2
Windows (all versions)okokok
CentOS 5.8 (kernel 2.6.18-308.13.1.el5)okokok
CentOS 6.2 (kernel 2.6.32-220.7.1.el6.x86_64)
okokok(*)
CentOS 6.3 (kernel 2.6.32-279.2.1.el6.x86_64)
okokpanic

Note for CentOS 6.2 on E3-1270V2: The VM works fine but when its vSphere console gets the keyboard focus the keyboard LEDs show the typical blinking pattern of a kernel panic. (Shift Lock and Caps Lock blinking in unison.)

Right now I avoid moving CentOS 6 VMs to the host with the E3-1270V2 processor, but that cannot be the last word in the matter.

Reply
0 Kudos
TilmanS
Contributor
Contributor

Any ideas on this?

It seems there's no support for Essentials, so I'm totally depending on the community to resolve this problem.

TIA

T.

Reply
0 Kudos
Joshwaa
Contributor
Contributor

I am having the same problem with with my 4.1 ESXi with Tyan S5512 and E3-1230V2 Ivy. Can not install CentOS 6.3 or Windows 8 64-bit.

Reply
0 Kudos
TilmanS
Contributor
Contributor

Nice to know I'm not the only one.

It would be even nicer if VMware would provide some sort of solution, or indeed any response at all.

Reply
0 Kudos
TilmanS
Contributor
Contributor

Just for the record:

The problem persists with the latest Update ESXi410-201211401-SG.

(Not that I had any hopes ...)

Reply
0 Kudos
TilmanS
Contributor
Contributor

Just for the record again: problem persists with CentOS 6.4.

(This time I did have hopes, as RHEL 6.4 announced new VMware drivers. Alas, no improvement.)

Should I switch to KVM? They seem to care, at least.

Reply
0 Kudos
TilmanS
Contributor
Contributor

By the way, before this I had also installed update ESXi410-201301401-SG, updating ESXi to build 988178. Didn't change anything about this problem either.

Reply
0 Kudos