VMware Cloud Community
gregsh
Contributor
Contributor

kernel panic in VM Red Hat when upgrading from RHEL54 to RHEL55

At my company, we have many Red Hat Linux (v5.4) VMs running in ESX with vCenter 4.1. We wanted to upgrade from RHEL54 to RHEL55 and used "yum update" to upgrade all RPMs. All RPMs upgraded without error, even the kernel RPM (to v2.6.18-194). The problem is when we reboot the VM using the new kernel, it PANIC's like this:

Red hat nash Version 5.1.19.6 starting

Unable to access resume device ( /dev/sda7)

mount: Could not find the filesystem '/dev/root'

setuproot : moving /dev/ failed : No such file or directory

setuproot : error mounting /proc : No such file or directory

setuproot : error mounting /sys : No such file or directory

switchroot: mount failed : No such file or directory

kernel panic - not sysncing: Attempted to kill init

If you select the older kernel via grub, the VM boots just fine. The VM disk controller is pvscsi and the ethernet driver is vmxnet3. This seems to be a problem with the initrd of the newer kernel. Do I need to have the pvscsi kernel object installed into the initrd? If so how do I tell vmtools to install itself into a different kernel than is booted (I can boot 2.6.18-164 but I want to install it into 2.6.18-194)?

Does this mean we cannot utilize "yum update" for Red Hat guest OS upgrades?

Thanks in advance for any help or advice.

Greg Shepherd

Reply
0 Kudos
5 Replies
gaspipe
Enthusiast
Enthusiast

Did you run vmware-config-tools.pl after the upgrade? If not, try booting the old kernel and running vmware-config-tools.pl with the --kernel-version or -k option (and your new kernel version after that), which could be used to "Build/install modules for the given kernel version instead of the running one, implies modules-only, skip-stop-start and compile."

vmware-config-tools --help

gregsh
Contributor
Contributor

I'm not sure how I missed the vmware-config-tools.pl utility, thank you for pointing that out. It did not work quite right but it started me towards the solution.

I ran it "vmware-config-tool.pl --kernel-version 2.6.18-195.el5" but it did not work correctly. It did build a new initrd file but it named it the same version as the running kernel (/boot/initrd-2.5.18-164.el5.img), not the specified version given with --kernel-version flag. I even renamed this new initrd to /boot/initrd-2.5.18-194.el5.img and tried booting with it, but it seems to be a mix of the two kernel environments and it still PANIC'ed at boot.

I did get the new kernel to work in this way:

cd /lib/modules

cp -pr 2.6.18-164.el5/misc 2.6.18-194.el5/

cat 2.6.18-164.el5/modules.alias | egrep 'vmci|vmx|pvscsi' >> 2.6.18-194.el5/modules.alias

cat 2.6.18-164.el5/modules.dep | egrep 'vmci|vmx|pvscsi' >> 2.6.18-194.el5/modules.dep

cat 2.6.18-164.el5/modules.pcimap | egrep 'vmci|vmx|pvscsi' >> 2.6.18-194.el5/modules.pcimap

I also had an original copy of the /boot/initrd-2.5.18-164.el5.img file because vmware-config-tool.pl kept over-writing the good one.

Thank you,

Greg Shepherd

Reply
0 Kudos
gregsh
Contributor
Contributor

Just a point of clarification. After copying the kernel modules into /lib/modules/ and modifying the three module files, you have to manually create the initrd like this:

/sbin/mkinitrd -f --with=vmxnet3 --with=vmxnet --with=pvscsi /boot/initrd-2.6.18-194.el5 2.6.18-194.el5

reboot and the new kernel should be booted.

then run vmware-config-tools.pl and reboot again

Reply
0 Kudos
josesa
Contributor
Contributor

It got me a bit of trouble getting it to work on a VM previously configured to use PVSCSI bus, but finally got it working with kernel update and tools reconfiguration without any reboots.

vmware-config-tools.pl is buggy because it always assumes you are installing/compiling the modules for the current kernel, independently of you passing the -k argument overriding the kernel version.

The internal compilation is also buggy because it installs the compiled modules in the current kernel directories instead of the new one. That's why I backup some files before module compilation and restore it later so the old kernel is still bootable.

Following is the full procedure I used (working and tested) without typos or partial information. Enjoy and let's hope VMWare fixes their buggy scripts so we don't have to do this manuallly like this:

# Assuming 'gcc' and 'make' were already installed with system
# or install before kernel update (assuming yum is configured properly)
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
yum -y install gcc make

-----
# going to some folder with new RHEL version mounted
cd /net/nfs_filer/mnt/rhel5u6_x64/Server
old_kern=$(uname -r)
new_kern=$(rpm -qp kernel-2.6.18-*.el5.x86_64.rpm | cut -f2- -d-)

-----
# Install required kernel packages
rpm -Uvh kernel-headers-2.6.18-*.rpm kernel-devel-2.6.18-*.el5*.rpm
rpm -ivh --force kernel-2.6.18-*.el5*.rpm

-----
# backup current kernel modules
rsync -a /lib/modules/${old_kern}/misc /tmp/${old_kern}

-----
# Build new kernel modules
for module in $(cd /usr/lib/vmware-tools/modules/source && ls -1 *.tar | cut -f1 -d.); do
    /usr/lib/vmware-tools/sbin64/vmware-modconfig-console --build-mod ${module} \
    /usr/bin/gcc /lib/modules/${new_kern}/build/include misc ${module} -- -l /usr/lib/vmware-tools
done

-----
# move new modules to their proper place
cd /lib/modules
mkdir ${new_kern}/misc
cp -p ${old_kern}/misc/*.o ${new_kern}/misc
cd /lib/modules/${new_kern}/misc
for mod in *.o; do ln -s $(pwd)/${mod} ${mod%.o}.ko; done

-----
# Copy settings from current kernel
cd /lib/modules
egrep 'vmci|vmx|pvscsi' ${old_kern}/modules.alias  >> ${new_kern}/modules.alias
egrep 'vmci|vmx|pvscsi' ${old_kern}/modules.dep    >> ${new_kern}/modules.dep
egrep 'vmci|vmx|pvscsi' ${old_kern}/modules.pcimap >> ${new_kern}/modules.pcimap

-----
# create initrd
/sbin/mkinitrd -f --with=vmxnet3 --with=vmxnet --with=pvscsi /boot/initrd-${new_kern}.img ${new_kern}

-----
# restore current kernel modules
rsync -a /tmp/${old_kern}/misc /lib/modules/${old_kern}
rm -rf /tmp/${old_kern}


-----

# Reboot into the new kernel without any VM configuration changes needed

Reply
0 Kudos
khushal
Contributor
Contributor

Here is more detailed article.with thorough explaination on the issue

http://www.rdoxenham.com/?p=218

Hope this helps...

Reply
0 Kudos