VMotion: Unable to migrate virtual machines between two ESX hosts

[agriesser]

Hi folks!

I'm trying to migrate VMs via VMotion from one ESX host to the other.

When I want to do this, the error message in VC2 looks like this:

Unable to migration from "host A" to "host B": The CPU of the host is incompatible with the CPU feature requirements of virtual machine; problem detected at CPUID level 0x1 register 'ecx'.

Well, I found two postings here in the community regarding this topic, but none led me to a

conclusion that helped me out here.

host A's /proc/vmware/cpuinfo looks like this:

================================

pcpu 00 01 02 03

family 15 15 15 15

model 04 04 04 04

type 00 00 00 00

stepping 03 03 03 03

cpuKhz 3200113 3200113 3200113 3200113

busKhz 200007 200007 200007 200007

name GenuineIntel GenuineIntel GenuineIntel GenuineIntel

ebx 0x00020800 0x01020800 0x06020800 0x07020800

ecxFeat 0x0000641d 0x0000641d 0x0000641d 0x0000641d

edxFeat 0xbfebfbff 0xbfebfbff 0xbfebfbff 0xbfebfbff

initApic 0x00000000 0x00000001 0x00000006 0x00000007

apicID 0x00000000 0x00000001 0x00000006 0x00000007

host B's /proc/vmware/cpuinfo looks like this:

==============================

pcpu 00 01 02 03

family 15 15 15 15

model 02 02 02 02

type 00 00 00 00

stepping 09 09 09 09

cpuKhz 3059962 3059962 3059962 3059962

busKhz 133041 133041 133041 133041

name GenuineIntel GenuineIntel GenuineIntel GenuineIntel

ebx 0x0002080b 0x0102080b 0x0602080b 0x0702080b

ecxFeat 0x00004400 0x00004400 0x00004400 0x00004400

edxFeat 0xbfebfbff 0xbfebfbff 0xbfebfbff 0xbfebfbff

initApic 0x00000000 0x00000001 0x00000006 0x00000007

apicID 0x00000000 0x00000001 0x00000006 0x00000007

Which attribute \_EXACTLY_ makes the problem here? All are different...

Is it ecxFeat?

Additionally, I don't want to mask some CPU attributes out on a per-VM basis if possible,

is there any way to tell each ESX host that they should provide all the same CPU?

Will this decrease performance on the "best" CPU if it presents itself as the "worst" one to all

virtual machines just for letting VMotion work?

[sbeaver]

The stepping and model is what is holding you back

[agriesser]

And is there anything I can do against this?

Are those two hosts simply incompatible to each other in terms of VMotion?

[mstahl75]

They shouldn't be, you just need to figure out what mask to use. I don't know what setup you have in your environment but you might be able to do a cold migrate and VC might create the mask for you.

I know in the process of migrating from 2.5.x and 3.0 the cold migrate did this for me. I haven't yet tested if it does it with all cold migrates or just the change from the previous versions.

[agriesser]

OK, I'll try this, wait a few minutes.

BTW: Is it usually a good idea to hide the exposure of the NX flag from the guest OS?

[mstahl75]

Most of what I have done at this point has been automated in regards to the CPU masking. My VMs have the option to keep the the settings for that are defined in the advanced section.

[agriesser]

OK, now I did a cold migrate of one machine from "host A" to "host B" and that (of course) succeeded.

But VC didn't do anything to the CPU masking settings of this virtual machine.

Needless to mention, migrating it back from "host B" to "host A" using VMotion doesn't work too.

I guess, I'll have a look at some KB-articles tomorrow morning during the morning coffee... 😕

[mstahl75]

Yeah, I wasn't sure if that would work or not. I know it did for the migrate from 2.5.x but it might only work in that case. If I get a chance I'll test it and see if it works on cold migrations cross LUN. That really isn't a solution though.

This topic[/url] has a link to some Intel documention that might be of help and there is some more discussion on the issue. I don't think there are very many KB articles about this issue yet or details on masking.

[agriesser]

I found this article, but it is all about VC 1.3.

http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=1993&sliceId=SAL_Public&di...

Is there any similar way to make the CPU check more relaxed in VC2 too?

[mstahl75]

If there is I haven't seen anything about it yet.

[agriesser]

Well, then, what to do now?

VMotion between two IBM xSeries 345 works flawlessly, but with the 346 I got these problems...

[agriesser]

My question is not answered by now, but my problem is solved.

I did some testing on the CPU masking myself in trial-and-error mode and succeeded with the

following CPU mask for CPUID Level 0x1, register ecx:

FFFF FFFF FFFF FFFF F0FF F0FF FFFF FFFF

This value lets me happily VMotion VMs from my originally mentioned "host A" to "host B" and vice

versa.

It took me about 20 to 30 minutes to find out the correct masking. Isn't there any easier way or

a description what the registers of the CPUid value are for so that one can easily find out such

masking values?

[mstahl75]

Yes, there has to be an easier way. In fact, there is some mechanism within VC, or possibly the ESX servers, to do some of the work already. That is what created my masks for me. Maybe there will be a way to get that in a future release.

[einthuber]

Hi,

i had the same problems with my ESX 3 / VC 2 environment.

When i tried to migrate my Guest from a HP DL 380 G4 to a HP Blade I got the error: CPUID level 0x1 register ecx

My CPUs are:

VMHost1:

pcpu 00 01 02 03

family 15 15 15 15

model 02 02 02 02

type 00 00 00 00

stepping 05 05 05 05

cpuKhz 3189374 3189374 3189374 3189374

busKhz 132890 132890 132890 132890

name GenuineIntel GenuineIntel GenuineIntel GenuineIntel

ebx 0x0002080b 0x0102080b 0x0602080b 0x0702080b

ecxFeat 0x00004400 0x00004400 0x00004400 0x00004400

edxFeat 0xbfebfbff 0xbfebfbff 0xbfebfbff 0xbfebfbff

initApic 0x00000000 0x00000001 0x00000006 0x00000007

apicID 0x00000000 0x00000001 0x00000006 0x00000007

VMHost2:

pcpu 00 01 02 03

family 15 15 15 15

model 04 04 04 04

type 00 00 00 00

stepping 03 03 03 03

cpuKhz 3600128 3600128 3600128 3600128

busKhz 200007 200007 200007 200007

name GenuineIntel GenuineIntel GenuineIntel GenuineIntel

ebx 0x00020800 0x01020800 0x06020800 0x07020800

ecxFeat 0x0000659d 0x0000659d 0x0000659d 0x0000659d

edxFeat 0xbfebfbff 0xbfebfbff 0xbfebfbff 0xbfebfbff

initApic 0x00000000 0x00000001 0x00000006 0x00000007

apicID 0x00000000 0x00000001 0x00000006 0x00000007

You can see that the CPU in the VMHost1 is an Intel Xeon 3.2, model 2, stepping 5, 512 KB Cache and the CPU in the VMHost2 is an Intel Xeon 3.6, model 4, stepping 3, 2048 KB Cache.

With VMWare EXS 2.5.x there was no problem switching between this CPUs.

With the information from agriesser (thank you!!!) it was no problem for me to find the correct mask:

I used: "FFFF FFFF FFFF FFFF F00F F0F0 00F0 00F0" to mask my CPUs, and VMotion worked well. I don't know if there is a performance bottleneck when i switch to the slower CPU.

Do anybody knows if this is by design from VMWare?

[dwcrist]

If you scroll down in the article you mentioned, you will find a section that shows how to mask things for VC 2.x/ESX 3.x.

http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=1993&sliceId=SAL_Public&di...

Unfortunately for me, I'm in the middle, with VC 2.x/ESX 2.5.3. I used to be able to Vmotion using the config.ini file. But since upgrading to VC 2.0.1, it fails again.

Doug

[biekee]

take a look at the following link:

http://www.vmware.com/community/thread.jspa?messageID=543079&#543079

bk

[gshields]

If anyone's interested, I've posted a screencast of this bit-masking process to my blog at:

http://www.realtime-windowsserver.com/podcast/2007/06/extending_the_reach_of_vmotion.htm.

All the info that makes that screencast possible came from this and other threads right here on VMTN. I just figured I'd make it a little easier to figure out for people who're having troubles understanding the process.

Thanks all!