Post ESX 4.1 Update 2 DRS Issue

[woodycollins]

I hope this is in the correct community thread, if not feel free to move it.

We are experiencing an issue that is fairly odd.  In our environment we have 3 seperate clusters consisting of anywere from 2 hosts up to 5 hosts.  Each host is configured identically in regards to CPU model. (AMD 6234 2socket 12 core). Each clustered host was also running identical versioning of ESX 4.1 update 1 as well as patch level's.  In this state everything works great.  DRS, VMotions etc.

As of last night we began deploying out 4.1 update 2. Everything was going smooth with the deployment of the update by vmotioning machines off each host, applying the update and patches and bringing the host back into its associated cluster and continuing on with the next host in the same fashion.  That is until we get to the last cluster.  After vMotioning the VM's off the first host and applying the updates we recieve the following error when trying to migrate the VM's off the second host and back onto the first:

"A general system error occurred: The product version of the destination host does not support one of more CPU features currently in use by the virtual machine.  Such features from CPUID level 0x1 register 'ecx' are indicated with a '1' bit: x000:0000:x000:0000:00x0:0010:000x:000x"

The only time I have seen this issue is when you do not have compatible CPU flag's between the hosts however this is not the case with these two hosts as clearly the vMotions were working prior to update 2.  One thing I did notice was the CPU description under the summary tab does change post the upgrade.  But what is really throwing me is that we did this to 7 machines prior to this with the same configuration and did not run into this error.

Any idea's on what might be causing this and what can be done to get these VM's to vMotion live once again?

[peetz]

This may be caused by different BIOS settings (specifically the CPU virtualization and NX features).

Did you by chance update the BIOS on the machines and unintentionally changed some BIOS settings?

For details on diagnosing this see here: http://kb.vmware.com/kb/1029785

- Andreas

[woodycollins]

This was my thoughts at first as well.  We did not upgrade the bios or change any settings and just for sanity checks I booted into the bios and double checked the configuration.  Obviously I can not boot the host that is running the VM's at the moment but the host that we performed the upgrades on the bios configuration is configured as we expect and nothing was changed.  Plus if this were the issue I would have expected that even during the operation of the host on update 1 vMotion would not work properly.

[peetz]

The KB article describes a method for checking the settings without rebooting the host (using the CPUID tool inside a VM).

Have you tried that to check if there are any differences?

[woodycollins]

The article say:

     Boot a virtual machine from the CPUID Utility ISO Image on each host and check for any differences in the ECX register.

I don't have th ability at the moment to reboot one of the hosts, but will later tonight.  I can tell you that the CPU is incorrectly identified as a 4 socket 6 core (model 19 ...) processor and with update 2 the processor gets correctly identified as a 2 socket 12 core (model 21 ...) processor.  This was one of the things update 2 corrected was proper support for the model of processor we are using.  But once again even if this was the issue, its odd that we didnt see it on any other host prior to this cluster.  One other thing to mention is we are not running EVC anywere in our environment, so that shouldn't affect why it worked in one cluster and not the other.

[woodycollins]

Here is another interesting twist.  I just tested powering down a VM and migrating it offline to the host running 4.1 update 2 and after starting the VM it is able to v-Motion between both hosts in the cluster without problem.  Were as all VM's still running on the host with 4.1 update 1 shows the original error posted above when trying to v-Motion.

[peetz]

That probably means that the not yet updated 4.1 update 1 host exposes a CPU feature that is not available on the other host with 4.1 update 2.

You should really check the BIOS settings of the updated host if there is anything that is inadvertently disabled.

[woodycollins]

I will check the bios when I can.  But this logic doesn't make since.  Reason being, If I have 6 VM's running on a host that doesnt show the flags and I migrate the VM's to a host that does, I would expect to be able to move the VM's back to the host with the flag that is not presented to the VM since the VM's were started with the flag not being exposed to them. But I couldn't.  I would also expect the VM's on the host that does have the flag exposed to not allow vMotion to the host that does not have the flag exposed, however prior to the upgrade I could.

When a VM is powered on the VM may begin using the processor functionality presented to it by the host.  It doesnt suddenly change functionality if a VM is vMotioned to a host that does or does not have functionality being presented to a VM.

[woodycollins]

Alright, finally  moved all the VM's off and rebooted the host.  Bios configuration in regards to the NX and VT settings are identical.  After updating the host to update 2 I now see the same information dumped when I cat /proc/cpuinfo.  A very odd issue we have experienced.