I can crash any ESXi 4.1 server from guest OS

[ViennaAustria]

About two months ago we upgraded from ESXi 3.5 to 4.1. All Servers are build 320173, all VMs have the latest VMware tools installed. Among several hundred other guest OSes we have 4 very old VMs using SuSE Linux 8.x or 9.x with 2.4 kernel. They're leftovers from our stone age, but they work well and do their jobs.

When I shut down one of these VMs, everything ist fine.

When I boot these VMs up again, everything is fine.

But when I reboot any of these VMs, the ESXi host crashes with all the other VMs on it!  :smileyshocked:

The usual picture on the console stays the same, but it won't respond to keystrokes any more. It also doesn't respond to ping any more, you can't log in by SSH or vSphere Client - nothing. It's completely blocked. The only options are the reset key or power button.

It won't make a difference, if you issue the reboot of the VM inside the os by, like, /sbin/reboot, or if you issue it from the vSphere Client with "Restart Guest". The reult is always the same: a total crash of the entire host - and of course of all the other VMs on it.

We're not definitely sure, if the crashes came from the fact, that these VMs use the old 2.4 kernel or some other issue. But these 4 VMs are the only ones that could crash the VMware host and they're also the only ones we have with 2.4 kernel. +100 other Linux guests with 2.6 kernel work as well as before with ESXi 3.5.

Meanwhile we could upgrade (reinstall) 2 of those VMs with a modern Linux with 2.6 kernel. No issue with rebooting them any more. But the other 2 are using ancient software which isn't available for moden Linuxes any more. We have no workaround yet. One of the remaining VMs felt like reboot last night and crashed the server [it's 7am right now in Vienna] and the SMS alerts woke me from bed about two hours ago. :smileyangry:

Does anybody else have such problems?

Does anybody maybe have a solution other than "install a modern OS"?

Thanks!

[Dave_Mishchenko]

I haven't heard of this issue myself.  I would suggest checking the setting syslog.local.datastorepath and ensure that it's set to a datastore or /scratch for ESXi installable.  That way you can get the log files from ESXi when the problem occurs again.

[bulletprooffool]

I'd suggest getting your hardware checked - get a vendor diagnostic tool and get the physical host (particularly CPU and Memory) checked out.

Do you get any crash logs you could share?

Also, just out of curiosity, could you verify that the Virtual Hardware that you present to your VMs is for the correct version of Linux?

[J1mbo]

Can the OP or anyone else put up a 2.4 kernel susie distro installable ISO?

[ViennaAustria]

We already tried other ESX servers in our datacenters. It behaves identically on Dell Servers (2950, R300) and on self-built machines with Supermicro+Xeon or Asus+AMD.

I can definitely rule out any hardware issue. When I put the VMs back to our last ESXi 3.5 host, they won't crash.

So it's definitely some wierd interaction between these VMs [most likely the old 2.4 kernel] and ESXi 4.1.

[piaroa · Accepted solution]

A shot in the dark, but have you tried updating your VMware tools and the upgrade to vHardware 7 on those VMs?

[J1mbo]

The concern here is that this could represents a DoS attack vector against ESXi hosts - I wonder how long this thread will stay up....

[piaroa]

What kernel version are those VMs running, and what distro ?

I'd like to reproduce the issue with a lab environment.

[ViennaAustria]

@ piaroa

> A shot in the dark, but have you tried updating your VMware tools and the upgrade to vHardware 7 on those VMs?

No, we haven't tried that. Could be a simple solution!

I'll migrate the VMs to an unused server and try that in the evening.

> What kernel version are those VMs running, and what distro ?

As said before, there are only two of those strange VMs left. Both are

# SuSE-release
SuSE-Release: 9.0 90 i386
# uname -a
Linux kunde1 2.4.21-303-default #1 Tue Dec 6 12:33:58 UTC 2005 i686 athlon i386 GNU/Linux

The others, which we could upgrade/reinstall, were SuSE 8.2 with the latest kernel update and another SuSE 9.0. All had between 1 and 4 GiB RAM.

[piaroa]

I just installed CentOS 3 and SUSE 9 on our 4.1 lab, installed VMware tools and issued a /sbin/reboot from the guest and had no crash. I then used the Restart VM option from vSphere client and didn't have an issue.

Can't seem to reproduce this on the lab.

[ViennaAustria]

Which virtual hardware version did you use? 4 or 7?

If you like, I can send you an image of our VMs...   [joking]

But first let's see, if the virtual hardware upgrade to v7 solves our problem.

[piaroa]

I used both v4 and v7, running kernel 2.4.21.

[DSTAVERT]

Before fixing things I would give VMware Support a call. If it is indead an issue having a broken instance would be helpful in fixing the problem in code.

[ViennaAustria]

@piaora:

> A shot in the dark, but have you tried updating your VMware tools and the upgrade to vHardware 7 on those VMs?

Yeah! Great! That solved the problem [at least on one of the two remaining VMs]! I can now reboot the guest OS without crashing the ESX host,

I'll try that on the other VM in a minute.

THANK YOU!

[ViennaAustria]

That was a chear too early. A second reboot attempt crashed the ESXi host again.

So: NO solution with upgrading the virtual hardware from v4 to v7.

:smileycry:

[DSTAVERT]

VM support?

[piaroa]

Now try to get the host log files from before the crash.

You can easily browse your hosts logs by going to https://yourhostip/host and logging in.

See what you get out of those.

[DSTAVERT]

ESXi logs are written to a RAM disk so it won't help unless you point logs to a datastore or a syslog server. Use the Configuration TAB / Software / Advanced Settings / Syslog

[piaroa]

How about if you recreate the virtual machine and point it to the existing vmdk.

Just create a new vm with any name, then attach the virtual disks of the vms that crash your host, and try.

It could be something in the vmx file.

Also check your hosts logs, go to https://yourhostip/host and you'll find most interesting logs there.