VMware Cloud Community
delainej
Contributor
Contributor

Vmotion and CPU compatibility

Can you find out if we can use vmotion on an ESX server with 2 single processors with hyperthreading (3.6 Ghz) and an ESX server with dual processors (2.88 Ghz)?

0 Kudos
58 Replies
Quotient
Expert
Expert

There is a legend in the VI Client which explains the different characters.

VM -> Edit Settings -> Options -> Advanced -> Advanced -> Legend

Smiley Happy

0 Kudos
mstahl75
Virtuoso
Virtuoso

I guess I need to open my eyes :smileygrin:

Thanks.

0 Kudos
ServersAndStora
Contributor
Contributor

For example, here is what VC 2.0 creatd during cold

migrations:

cpuid.1.eax =

"--xxxxxxxx----xxxx------"

cpuid.1.ecx = "--------------R-0--R----"

cpuid.1.ecx.amd = "--------------------------------"

cpuid.80000001.ecx =

"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0"

cpuid.80000001.edx =

"xx0xxxxxxxx0xxxxxxxxxxxxxxxxxxxx"

Ah - this could be most useful to me - I'm having no luck migrating from Opteron 200 series (HP BL35p) to Opteron 800 (HP BL45p)

Where does VC2 write this information during a cold migrate?

ta

tobi

0 Kudos
mstahl75
Virtuoso
Virtuoso

It sticks it in the vmx file.

0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

Community Initiative[/b]

Guys, whilst I was at VMworld last week I attended a session entitled:

"Migrating between Apples and Oranges with VMware VMotion in VMware Infrastructure 3"

http://www.rtfm-ed.co.uk/?p=309

I was a good session - but I felt it still left some questions unanswered. After the session I was approached by some delegates asking if "RTFM Education" would be writting a whitepaper on the topic.

This is something I would like to do, but I feel quite out of my depth. Also I think such an undertaking would require very[/i] rigious hardware testing. Something that one-man-shop like me would struggle to do properly - through a lack of equipment. In fact I think in general such a whitepaper could only be[/i] created by a joint community effort.

So that's why I am adding this post today - to propose something. In the abscence of any "idiots" guide or cook-book on using the advanced CPU masking feature, I propose that we as a Community produce our own whitepaper. I wouldn't take any credit for the work - that would go to you guys who contributed (either in an authorial or testing manner). I would be happy to host the whitepaper on my website - and hopefully that would help disseminate this knowledge to the wider community. It would join the library of free guides and documents I host at RTFM Education already.

This is the way I see my site going. As I become more busy with paid-for-work - the amount I give back to community is under-pressure. I would like to do more - but time is not endless resource - and the book I am writing with Ron & Scott must take priority in the short/medium term. I would however, like to kick-start some Community driven projects - using my website as a platform to get them out to the Community at large. Remember my site has completely free content - and contains no ads - so its about as "neutral" and not-for-profit a space there can be that still gets a reasonable amount of hits. I would hope this project would not go unnoticed - and would benefit the reputation of those who choose to contribute.

Anyway. If this is particular topic is off interest to you please respond. I think once the "Apples & Oranges" PPT's are released this will provide a good framework for documenting the CPU mask feature. There was a few tidbits in there which I thought the community at large would benefit from.

Kind Regards

Mike Laverick

RTFM Education

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

oh i should add - something i hadn't noticed - that KB1993 has been updated to add some addition info about advanced CPU masks - I imagine you guys knew this already - but it had slipped under my radar...

http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=1993

Regards

Mike

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
keitha
Enthusiast
Enthusiast

Hendrick, becareful masking "all" new features. I was at the session Mike spoke of and the best tidbit I walked away with is the fact that ESX performas Binary translation on the OS only. Not the application execution. This makes sense (now) as the application will run at ring level 3 and can't make illegal call on the CPU. What it does means though is that the masks that you create do not apply to the application. VMotioning an application from a CPU with SSSD to one with out could cause your application to do something bad if it calls that instruction set. VMware supports masking the NX bit but everything else you are on your own for.

0 Kudos
Hendrik_Van_Nev
Contributor
Contributor

If I mask all the new features per VM , it should not make a difference if I am running it on DL380G4 or on a DL580G2. Or am I wrong here ? The application is not expecting the new feature , so it is not getting the new features on either system (masked on both systems).

I have been using the following masks for some months now ... didn't have any problems with them yet...

Level 1

eax -


XXXX XXXX -


-


XXXX -


-


ecx -


-


-


-


R-0- -


0R-0 00-0

Level 80000001

ecx XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXX0

edx XX0X XXXX XXX0 XXXX XXXX XXXX XXXX XXXX

0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

This is precisely the kind of information - that needs putting into a single white paper on the issue & tools...

Keith is right on the money on this one...

Do we have any takers to write a whitepaper... I've already had Scott Lowe of Jumpbox express an interested. He's taken stuff from this thread and blogged about it. But I think we all know it needs cooking down - drawing all these tidbits in a single document that become the kind of "bible" on this topic generally... the information/knowledge is out there in the community it just needs consolidating & co-ordinating... if we don't do it - who will???

Regards

Mike

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
keitha
Enthusiast
Enthusiast

Hendrik, you are correct with regard to the OS but what we learned at VMworld is that the applications are not masked. That is it can "see" the CPUs true nature despite the VMware mask. I don't expect many applications to have a problem, most will never directly use SSSD instructions and their for never have a problem. The problem is we don't know which ones do or when they might decide to call them.

Mike, I am definitely willing to help however I can. Count me in.

0 Kudos
bwb
Contributor
Contributor

Does anyone has a mask for these two processor types ?

XEON 5030 (SSE3) and XEON 5160 (SSE4)

I read the VMware Docs 1991 and 1993 to ignore SSE4 instructions (and also SSE3 combinded with SSE4) but i get everytime the same error when i want to migrate or use vmotion:

unable to migrate from xyz to zyx:

The CPU of the host is incompatible with CPU feature requirements of virtual machine; problem detected at CPUID level 0x1 register 'eax'

Servers are Dell 2950, same hardware except of the XEON CPUs

0 Kudos
doomdevice
Enthusiast
Enthusiast

I´ve got another information about migrating between ESX Hosts.

If Intel VT is activated on only one of the ESX Server used for migration you get a message like 0x80000001 at edx.

Turn on or turn off VT on all ESX Servers used for VMotion and you´re fine.

VI PowerScripter [http://www.powerscripter.net] Every Click can be a customized function within VI client
0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

I´ve got another information about migrating between

ESX Hosts.

If Intel VT is activated on only one of the ESX

Server used for migration you get a message like

0x80000001 at edx.

Turn on or turn off VT on all ESX Servers used for

VMotion and you´re fine.

Hi there,

You seem to know what you are doing - have thought of contributing to writing a whitepaper on this feature?

Regards

Mike

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

Something that you might be all interested in:

Richard at Anykey has written a VC SDK application which reports cpu compatiability information:

http://www.run-virtual.com/?p=153

Regards

Mike

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
larstr
Champion
Champion

The presentation is available now:

VMotion between Apples and Oranges, Understanding CPU Compatibility Constraints for VMware®VMotionTM[/url]

Lars

0 Kudos
Michelle_Laveri
Virtuoso
Virtuoso

The presentation is available now:

[url=http://download3.vmware.com/vmworld/2006/tac1356.

pdf]VMotion between Apples and Oranges, Understanding

CPU Compatibility Constraints for

VMware®VMotionTM[/url]

Lars

This was a great presentation - and I am glad its been released - I was hoping that I could inspire a community based whitepaper - which would take this PPT as a starting point - creating a cookbook for anyone want to use the "advanced" CPU masking features...

Regards

Mike

Regards
Michelle Laverick
@m_laverick
http://www.michellelaverick.com
0 Kudos
admin
Immortal
Immortal

Hi Mike,

This is a good idea, I'd like to help however I can. With some coordinate with the autor of the vmotionInfo program on http://www.run-virtual.com we could get some auto-mask generation for VM migrations which would be very useful.

Obviously testing is limited by what kit I have to test things out on. I've been playing around a bit vmotioning VMs between 8832 HS20 IBM Blades and 8843 HS20's. Cpu info as per below.

8832[/b]

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 2

model name : Intel(R) Xeon(TM) CPU 2.80GHz

stepping : 9

cpu MHz : 2799.302

cache size : 512 KB

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 2

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm

bogomips : 5583.66

8843[/b]

processor : 0

vendor_id : GenuineIntel

cpu family : 15

model : 4

model name : Intel(R) Xeon(TM) CPU 2.80GHz

stepping : 3

cpu MHz : 2800.238

cache size : 2048 KB

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 5

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm lm

bogomips : 5596.77

As you can see the newer processors have the lm bit. I've found that the following mask on the level 0x1 ecx bit allows vmotion between them.

FFFF FFFF FFFF FFFF F00F F0F0 00F0 00F0[/b]

0 Kudos
scott_lowe1
Contributor
Contributor

Hi all,

I didn't have the opportunity to attend the session at VMworld 2006 but had already experimented with CPU masks before the conference and had written up some brief information here:

http://blog.scottlowe.org/2006/09/25/sneaking-around-vmotion-limitations/

After seeing the release of the VMotionInfo tool by Richard Garsthagen, I downloaded his tool and then expanded upon my original article to include more information. The expanded information is here:

http://blog.scottlowe.org/2006/11/23/vmotion-compatibility/

I'd love for the technical geniuses here to pick apart what I've written and contribute additional information based on their own experience. Perhaps this can form the basis of a community-written whitepaper? I'm willing to help in any way that I can.

Regards,

Scott

0 Kudos
ForgeFlakshack
Enthusiast
Enthusiast

Thanks for the articles Scott, they helped out. Here's how I managed to get Vmotion working on my test system.

Trying to perform a Vmotion from a Dell 2950 to a Dell 2850 (fyi: according to the Dell chart here: http://download3.vmware.com/vmworld/2006/tac1356.pdf they are incompatible with VMotion).

Dell 2950 with 2 Intel(R) Xeon(R) CPU 5160 @ 3.00Ghz

Dell 2850 with 2 Intel(R) Xeon(TM) CPU 3.60Ghz

When attempting to perform the VMotion, I would see the following message:

The CPU of the host is incompatible with the CPU feature requirements of virtual machine; problem detected at CPUID level 0x1 register 'eax'.[/b]

First, I downloaded the VMotion Info utility (http://www.run-virtual.com/?page_id=155). The trick is to run it in debug mode (there is a option on the login page for this) so you can see the actual registers:

For the 2850 CPU 0 Level 1, eax register:

0000:0000:0000:0000:0000:1111:0100:0011

For the 2950 CPU 0 Level 1, eax register:

0000:0000:0000:0000:0000:0110:1111:0110

The different bits are:

0000:0000:0000:0000:0000:\*11*:*1**:0*1*

Then on the VM, Edit Settings...Options Tab...Advanced...Advanced Button. Click on Level 1 eax. The default mask for the EAX row (shown in the Row details section):

XXXX HHHH HHHH XXXX XXXX HHHH XXXX XXXX

X bits are not used by the VM, the H bits are shown to the VM, so basically we can ignore any different bits that have an X. To disable the feature, we want to set the conflicting bits to 0. The resulting bit mask is:

\-- -


-
-
-
00 -


-


XXXX HHHH HHHH XXXX XXXX 0HH0 XXXX XXXX

After saving the changes and attempting to VMotion again, I would see the following message:

The CPU of the host is incompatible with the CPU feature requirements of virtual machine; problem detected at CPUID level 0x1 register 'ecx'.[/b]

Checking the VMotion Info Utility for the ecx register differences:

For the 2850 CPU 0 Level 1, ecx register:

0000:0000:0000:0000:0110:0101:1001:1101

For the 2950 CPU 0 Level 1, ecx register:

0000:0000:0000:0100:1110:0011:1011:1101

The different bits are:

0000:0000:0000:0*00:*110:0**1:1011:1101

The default mask for the ECX register (again, click on the register and see the default in the Row details section below):

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH

R bits are already hidden from the OS, X bits are not used by the VM, and the 0 bits are already hidden, so we only need to mask out the conflicting H bits with a zero:

\-- -


-
-0 -


--0- -


-


RRRR RRRR RRRR R0R0 00XR R000 000H 0RRH

After saving the changes and attempting to VMotion again, it works! Well it seems to work anyway. The trick now is to figure out if I'm worried that making VMotion ignore those bits is going to cause me problems later on. I'm probably guessing that it will cause me problems, so I won't do this on my production environment. Fun to do in testing though.

For the EAX register, I changed bits 8 & 11 (the processor family identifier bits).

For the ECX register, I changed bits:

9 - SSSE3 (Which is what VMotion Info found as incompatible)

18 - DCA (Direct Cache Access)

0 Kudos
scott_lowe1
Contributor
Contributor

Forge,

Good information, thanks for sharing that with everyone. Programmers/developers out there: What could be the potential implications of hiding these bits (and thus masking the CPU's true functionality)? I'm thinking that in most cases it would simply cause a slight decrease in performance, but does anyone know if serious stability issues will result? That seems to be the #1 question these days, and one I'm not prepared to answer.

Regards,

Scott

0 Kudos