After updating software on a RedHat-6 system (yum update), workstation-10.0.1 will 'crash' the system.
The update included:
The 'Bug Tracking' tool reports problems with glib2 and kernel 'hang'.
Below is the latest news from RedHat.
Hello Lewis, I have opened up this support case addressing the kernel crash. From my primary analysis, I could find that VMware have already opened a bug with us for this issue. From current status on BugZilla, the bug still needs to be reviewed by our Engineering Team. I am also seeking help from our Kernel Team into this issue. I shall keep you posted once I get any updates on this case.
Thanks for the info. The bug happened to me today with a CentOs 6.5 VM, which after updating to latest kernel generated a kernel panic. Booting previous kernel solved it. Hope that RH and VMware will solve the problem.
Thanks all for reporting this issue and posting tips to work around the issue. We are working together with Redhat to find a solution.
Here are steps to work around the problem:
If the system does not boot without panic, start with step 1.
If the system still boots, start with step 8.
Steps 1-7 are to make the system bootable again
by removing the vmware modules:
1) at boot time, invoke grub menu by pressing any key
2) in the grub menu, press 'a'
3) add 'init=/bin/sh' to kernel args
4) mount -o remount,rw /
5) rm /lib/modules/2.6.32-431.el6.i686/misc/v*.ko
6) depmod -a
7) reboot the system
Remove broken modules from system:
😎 remove prebuilt modules:
sudo rm -rf /usr/lib/vmware/modules/binary/bld-*71*
sudo rm -rf /usr/lib/vmware/modules/binary/bld-*131*
9) sudo /etc/init.d/vmware stop
10) sudo rm /lib/modules/2.6.32-431.el6.i686/misc/v*.ko
11) sudo depmod -a
Make workstation build native modules:
12) sudo yum install make gcc kernel-headers-$(uname -r)
13) start vmware - at start, you will be asked for the root password, and modules will be automatically built from source.
I have run into the same problem, with the same symptoms:
VMware Workstation 10.0.1
OS: CentOS 6.5 amd64
Linux VMs seem fine
Windows 7 VM freezes the entire system (the host) shortly after the Windows desktop loads
Windows XP VM does the same
Recent update to kernel 2.6.32-4126.96.36.199.1.el6.x86_64 did NOT fix the issue
Everything below is done using the 2.6.32-4188.8.131.52.1.el6.x86_64 kernel
Followed steps 8-13 of the workaround suggested by okurth1
What I got was the VMware Kernel Module Updater compiling all the modules without problems (or so it seemed), but gave an error at the end saying:
"Unable to start services. See log file /tmp/vmware-root/vmware-modconfig-3659.log for details."
Running "sudo service vmware start" gives:
Starting VMware services:
Virtual machine monitor [FAILED]
Virtual machine communication interface [FAILED]
VM communication interface socket family [FAILED]
Blocking file system [ OK ]
Virtual ethernet [FAILED]
VMware Authentication Daemon [ OK ]
Looking at this log file, either I'm not looking closely enough, can't see it because I don't know what I'm looking for, or there's nothing there, which is the reason it finishes compiling and actually tries to start the vmware service.
Hello, I am experiencing the exact same problem. RHEL 6.5 x86-64 Workstation on an AMD Opteron 4386, running VMware Workstation 9.0.3. I upgraded to VMware Workstation 10.0.1 and it did not resolve the problem. glib2 crash and whole machine crash when booting up Windows XP virtual machine.
I hope Red Hat can fix this bug soon. To the person who opened the Red Hat support case, it may be beneficial to give them the url to this discussion thread.
I have chosen a different workaround: Use the old kernel (2.6.32-358.23.2 instead of 2.6.32-431). The Windows VM runs fine on the old kernel.
Just edit /boot/grub/menu.lst and change the default so you don't have to remember to select the old kernel after every reboot.
That worked for me too, thanks very much.
I run into the same problem as described by Jack10z using the method okurth1 posted. I am running 64-bit 2.6.32-431.1.2.el6 kernel (instead of 32-bit 2.6.32-43.el6 one used in okurth1's example). The kernel modules seems to have been built fine but had trouble in loading:
# cd /lib/modules/2.6.32-431.1.2.el6.x86_64
# file misc/*
misc/vmblock.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
misc/vmci.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
misc/vmmon.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
misc/vmnet.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
misc/vsock.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
# lsmod | grep vm
kvm_intel 54285 0
kvm 332980 1 kvm_intel
# modprobe vmblock
FATAL: Error inserting vmblock (/lib/modules/2.6.32-431.1.2.el6.x86_64/misc/vmblock.ko): Invalid module format
It failed on 2.6.32-431.el6.x86_64 as well. Here is part of the log:
2013-12-16T16:35:24.287-05:00| vthread-3| I120: Obtaining info using the running kernel.
2013-12-16T16:35:24.287-05:00| vthread-3| I120: Setting header path for 2.6.32-431.el6.x86_64 to "/lib/modules/2.6.32-431.el6.x86_64/build/include".
2013-12-16T16:35:24.287-05:00| vthread-3| I120: Validating path "/lib/modules/2.6.32-431.el6.x86_64/build/include" for kernel release "2.6.32-431.el6.x86_64".
2013-12-16T16:35:24.288-05:00| vthread-3| I120: Failed to find /lib/modules/2.6.32-431.el6.x86_64/build/include/generated/utsrelease.h
2013-12-16T16:35:24.288-05:00| vthread-3| I120: Created new pathsHash.
2013-12-16T16:35:24.300-05:00| vthread-3| I120: Preprocessed UTS_RELEASE, got value "2.6.32-431.el6.x86_64".
2013-12-16T16:35:24.300-05:00| vthread-3| I120: The header path "/lib/modules/2.6.32-431.el6.x86_64/build/include" for the kernel "2.6.32-431.el6.x86_64" is valid. Whoohoo!
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Reading in info for the vmmon module.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Reading in info for the vmnet module.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Reading in info for the vmblock module.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Reading in info for the vmci module.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Reading in info for the vsock module.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Setting vsock to depend on vmci.
2013-12-16T16:35:24.404-05:00| vthread-3| I120: Invoking modinfo on "vmmon".
2013-12-16T16:35:24.407-05:00| vthread-3| I120: "/sbin/modinfo" exited with status 256.
2013-12-16T16:35:24.407-05:00| vthread-3| I120: Invoking modinfo on "vmnet".
2013-12-16T16:35:24.410-05:00| vthread-3| I120: "/sbin/modinfo" exited with status 256.
2013-12-16T16:35:24.410-05:00| vthread-3| I120: Invoking modinfo on "vmblock".
2013-12-16T16:35:24.414-05:00| vthread-3| I120: "/sbin/modinfo" exited with status 256.
2013-12-16T16:35:24.414-05:00| vthread-3| I120: Invoking modinfo on "vmci".
2013-12-16T16:35:24.418-05:00| vthread-3| I120: "/sbin/modinfo" exited with status 256.
2013-12-16T16:35:24.418-05:00| vthread-3| I120: Invoking modinfo on "vsock".
2013-12-16T16:35:24.422-05:00| vthread-3| I120: "/sbin/modinfo" exited with status 0.
2013-12-16T16:35:24.435-05:00| vthread-3| I120: to be installed: vmmon status: 2
2013-12-16T16:35:24.435-05:00| vthread-3| I120: to be installed: vmnet status: 2
2013-12-16T16:35:24.435-05:00| vthread-3| I120: to be installed: vmblock status: 2
2013-12-16T16:35:24.435-05:00| vthread-3| I120: to be installed: vmci status: 2
thanks for providing the log files. This gave me the clue what's going on:
The PBM set "bld-2.6.32-100.28.5.el6-x86_64-Oracle6.0" is the best match for kernel "2.6.32-4184.108.40.206.1.el6.x86_64".
So the solution is to remove this set of precompiled modules from /usr/lib/vmware/modules/binary/ as well, so modconfig won't find them. You can remove all directories in /usr/lib/vmware/modules/binary/ starting with bld-2.6.32*, you won't need them. Alternatively, just rename the whole directory /usr/lib/vmware/modules/binary/ to something else (like /usr/lib/vmware/modules/binary.away), or remove the whole directory. Then go to step 9 in the instructions that I posted.
Worked great! No problems in XP or Windows 7 VMs anymore.
Summary of what I did:
Had no problem booting the host, so skipped steps 1-7.
$ sudo service vmware stop
$ sudo mv -v /usr/lib/vmware/modules/binary /usr/lib/vmware/modules/binary.orig
$ sudo rm -v /lib/modules/$(uname -r)/misc/v*.ko
$ sudo depmod -a
#I already had gcc, make, and the kernel headers, but if anyone is following this, they should probably make sure
$ sudo yum install make gcc kernel-headers-$(uname -r)
Very nice, works for me too on kernel 2.6.32-4220.127.116.11.1.el6.x86_64, VMware Workstation 8.0.6 build-1035888, and a Windows 7 guest OS.
In addition to Jack10z's instructions I had to
yum install kernel-devel
before VMware was able to build the modules.
Thanks to okurth1 and Jack10z for sharing your great workaround!
That's by far the best so far.
> ... but this looks like you didn't call 'depmod -a' after removing the modules from /lib/modules/2.6.32-431.el6.x86_64/misc/
Indeed. It works fine for me now. Thank you very much, okurth1.
Sigh. Doesn't work for me. I do get all of the kernel modules rebuilt and loaded and
all of the vmware services start just fine. But vmware exits immediately after rebuilding
and starting the services. Running vmware again results in it rebuilding the modules again
and exiting after successfully loading them). This is VMware Workstation 9.0.3 build-1410761
and CentOS 6.5 with kernel 2.6.32-418.104.22.168.1.el6.x86_64. I cannot find any errors in
It might be that your modules.dep is not world readable. If it's not readable by you, you will end up in the infinite loop trying to rebuild the kernel module again and again:
-rw-r--r-- 1 root root 196904 Dec 14 09:18 /lib/modules/2.6.32-358.23.2.el6.x86_64/modules.dep