Upgrading to vsphere 6 from 5.5 causes issues when upgrading vmware tools

[scottly]

Hello,

I have upgraded our vCenter from 5.5 to 6, and am currently in the process of upgrading our 6 host cluster to 6.0 u2 as well.  However, I have run into some issues, and since this is in production I need to be careful with this.  I've upgraded 3 hosts already, and 2 of them went by fine.  However, on the third one, after I migrated the machines from it to one of the new 6.0 hosts, I had issues installing/upgrading vmware tools.  For some of the VMs that are running RHEL 6, I am able to automatically install the tools via the vSphere client GUI.  On others however, I do not get the option to automatically install the tools, and it just says "Installing the VMware Toold package will greatly enhance graphics and mouse performance in your virtual machine."  When I click "OK", it can take a very long time to install the tools, and sometimes it seems to never finish (or I don't wait long enough, I've waited around 30 minutes before).  I'm not sure why some are different, the OS versions are all exactly the same as they've been deployed using the same template.

Also, some of the RHEL machines are throwing kernel panics.  I'm trying to pinpoint the cause, but we get "BUG:soft lockup - CPU# stuck for Xs!" messages.  Has anybody seen this before?  Is this due to the installation of vmware tools? 

[vHaridas]

Do you have existing VM tools running on VMs where you are not able to mount tools ISO?

for BUG:soft lockup - CPU# stuck for Xs - "Many Linux kernels have a soft lockup watchdog thread, and report soft lockup messages if that watchdog thread does not get scheduled for more than 10 seconds. On a physical host, a soft lockup message generally indicates a kernel bug or hardware bug. When running in a virtual machine, this might instead indicate high levels of overcommitment (especially memory overcommitment) or other virtualization overheads"

VMware KB: Soft lockup messages from Linux kernel running in an SMP-enabled virtual machine

Make sure RHEL Vms get enough CPU, memory. Migrate VMs to another ESXi Host and try to install VM tools and also look for soft lockup message.

also it could be the Kernel Bug also, so make sure you have same/working kernel on all systems. find your kernel version and search for possible issues.

[Muthur85]

Hello,

The soft lockup messages are not kernel panics, and generally appears when the virtual machine is using a large amount of its resources.

To stop the error messages from appearing as frequently:

Some kernels allow you to adjust the soft lockup threshold by running the command:

echo time > /proc/sys/kernel/softlockup_thresh

Where time is the number of seconds after which a soft lockup is reported. The default is generally 10 seconds.

[scottly]

Thanks, yes that appears to be why I could not install the vmware tools automatically on a couple of the VMs.  I didn't check before upgrading the hosts, but those VMs appear to not have been running vmware tools for some reason.  I need to wait for our Unix guys to run the vmware tools installation manually.

For the CPU lockup thing, the kernel versions are the same between some of the ones that gave the error, and others that did not appear to have any issue with the installation.  I did notice that some machines appeared to slow down considerably when I was upgrading vmware tools automatically (this wasn't a scientific test, I just pinged the server and saw that the response times would spike).  This didn't happen on all the servers however.  These servers aren't really highly utilized, either.  Is there any known cases where the installation/removal of vmware tools can cause server slowdown?  We aren't using the VMXNET3 driver, which I believe is updated in this tools version.

[scottly]

Thanks for the clarification!

[vHaridas]

There shouldn't be any performance issues.

as @Muthur85 said, you increase the softlockup_thresh values in Linux server. I had to do this my one my Production Oracle Linux VM.

Linux server will continue to work without any issue.