VMware Cloud Community
rob_nixon
Contributor
Contributor

VMotion compatibility between DL585G1 dual-core and DL585G5 quad-core

We are looking to add one or more of the new HP DL585G5 servers with quad-core Opteron processors to an ESX 3.0.2 server farm with several DL585G2 dual-core Opteron servers. I looked at the compatibility document, but have not yet been able to make any sense out of it.

Question is, are the dual-core and quad-core Opteron processors VMotion compatible?

Rob Nixon

Reply
0 Kudos
20 Replies
Troy_Clavell
Immortal
Immortal

I just got a bunch of information on this, I hope it helps.

Link to latest vmotion compatibility matrix:

Link to VMware server/options compatibility matrix:

LInk to ProLiant OS Support Matrix:

Reply
0 Kudos
jfields
Enthusiast
Enthusiast

I cannot speak personally to the Opterons, but I can tell you that we have two ESX hosts with different core counts. One is dual-core and the other quad-core. VMotion works great and I have never had any problem with it.

Reply
0 Kudos
eldee201110141
Enthusiast
Enthusiast

If you go from a VM created on DL585G1 to the DL585G5 you'll have no problems. You might have problems from any VM created on the DL585G5 going to the DL585G1.

Reply
0 Kudos
rob_nixon
Contributor
Contributor

I'm still investigating this question. I found, on on the HP web site, the list of HP servers and processor groups that are VMotion compatible. Our existing servers are listed, DL585G2 with Opteron 8220SE processors, but not the newer DL585G5 with Opteron 8356 processors.

I have a case open with VMware support. Hopefully they will come up with the answer.

Thanks to everyone,

Rob Nixon

Reply
0 Kudos
kimono
Expert
Expert

Definitely keen to have this compatibility confirmed...

/kimono/

/kimono/
Reply
0 Kudos
bggb29
Expert
Expert

We have had some masking issues between dl585g1's and dl385 g2's 585's are 880 and the 385's are 2200.

Which got worse after 3.5 starting adding lines into the vmx files. I had this all running under 3.0 with one

simple mask. Have yet to fully get it run under 3.5

Reply
0 Kudos
BenConrad
Expert
Expert

vMotion between socket E and Socket F works fine in 3.x.x using this mask:

<<VM DEFAULT TAB>>

cpuid.1.eax = -


xxxxxxxx----


cpuid.80000001.ecx = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0

cpuid.80000001.edx = xx0xxxxxxxx0xxxxxxxxxxxxxxxxxxxx

<<AMD OVERIDE TAB>>

cpuid.80000001.edx.amd = -


0----0--


Can somebody post their socket E/F & quad-core mask when it's figured out?

Reply
0 Kudos
Thorsten_Schnei
Hot Shot
Hot Shot

Hi Rob,

have you got any informations from VMware on this topic ? I'm currently in the same situation as we are about to buy new DL385G5. We already have 385G1 and G2 and VMotion works with the CPU-mask applied. It is pointless to buy a G5 if we cannot VMotion between the hosts.

It is a pitty that the informations in the compatibility matrices haven't been updated yet.

Thanks

Thorsten

Reply
0 Kudos
rob_nixon
Contributor
Contributor

This issue is still not completely clear, but one of our guys was at the HP Conference last week and talked with an AMD rep. The word he got is that as of now the DL585G5 - "Barcelona" Opteron quad-core is not VMotion compatible with DL585G2 - Opteron dual-core. He was told that in the reasonably near future some changes in ESX would make it possible to VMotion between these two processors. We don't know a time frame for those changes.

That's the best I've been able to come up with so far.

We have just about decided that our 8 x DL585G2 ESX farm with 374 VMs is about big enough, so we are planning toward building a new farm with several new DL585G5s

Reply
0 Kudos
mreferre
Champion
Champion

Rob,

he might be referring to a long waited technlogy that would allow you to vmotion across different processor generations. In order to make this happen support must be built into the processor and into the processor.

I have been looking more at the Intel house of the things for a number of reasons lately (i.e. there was not so much to look at the AMD house and also it looks like Intel was more fashionable than AMD for more customers ... but I am sure that as all fashions ... this will change again). For Intel in order for this to happen you('ll) have to have at least Penryn based CPU's cores (i.e. the lastest 52xx and 54xx QC cpu's for the 2 socket space or Dunnington which is the new 6 cores CPU for the 4 socket and above space). These processors have built-in morphing features that allows them to present themselves as "old kits" when it is required. That way you'll be able to VMotion between one of these beasts and say ... a Tulsa based box (like the IBM 3950/3850 ... which i think would be a DL580G4 if I get it right). This will only be possible with VMware software support which should happen shortly anyway (sorry can't post here details as you know VMware is pretty paranoid about people disclosing stuff).

So .. back to the point.... I think that Barcelona has this hardware built-in support (to be confirmed as I am not sure) and they are waiting for VMware to come out with the update to intercept it. I am not even sure if the same VMware update will work for both Intel and AMD cpu's....

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
Reply
0 Kudos
sheetsb
Enthusiast
Enthusiast

In kb 1992 there is a discussion about how to make the Barcelona chips vmotion compatible with earlier processors. It specifically states "...apply 3DNPREFETCH mask (not supported)." I haven't tried it yet but I may. I do feel a bit squeamish about using an unsupported configuration in production.

Bill S.

Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso

Found this interesting whitepaper on the AMD web site: http://developer.amd.com/assets/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf. Page 16 has a very nice diagram indicating VMotion support.

It would seem the technology is definitely in the processor, even allowing original single-core (Rev. C) Opterons to VMotion to the newer Barcelona, but as Massimo said it's dependent on a software update from VMware.

Reply
0 Kudos
Thorsten_Schnei
Hot Shot
Hot Shot

A very similar whitepaper can be downloaded from The Register: http://whitepapers.theregister.co.uk/paper/download/375/live-virtual-machine-migration-on-amd-proces...

It contains exactly the same information and was also created by AMD. The layout looks different and the very intersting part is at the end where they show a way how to migrate VMs between 4-core CPUs and older ones.This is still beta code but it proves that there is a way to do it.

Regards

Thorsten

Reply
0 Kudos
xylog
Enthusiast
Enthusiast

I have gotten vmotion to work from a DL585 G1 with the Dual Core Denmark Opteron (model 880) to a DL585 G5 with the Quad Core Barcelona Opteron (model 8356) with this mask:

cpuid.1.eax = ""
cpuid.1.ecx = "--


0
"
cpuid.1.edx = ""
cpuid.80000001.eax.amd = ""
cpuid.80000001.ecx.amd = "

00000000000"
cpuid.80000001.edx = "

0

"
cpuid.80000001.edx.amd = "
0-00---0
0-0

"
cpuid.80000001.ecx = ""
cpuid.1.ecx.amd = "

0

0

0
-"
cpuid.1.eax.amd = "--
00000
00----00000000"

but it fails when I try to vmotion the vm back from the G5 to the G1, with the message,"a general system error occurred: invalid fault."

Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso

A more reliable and supported way would be to just upgrade the farm to 3.5u2 (with the licensing patch, of course Smiley Happy ) and enable EVC.

Reply
0 Kudos
ThompsG
Virtuoso
Virtuoso

We tried this but cannot add G5 to a G1/G2 cluster and vice-versa. They must be using diffent masks between them.

I thought that EVC was going to help in this regard however it just makes it worse (or at least no better).

Glen

Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso

That's odd, this is what EVC is supposed to be able to do.

Is the cluster you're trying to move the hosts into enabled for EVC? Can you provide some details about the steps you're following / messages you're getting?

Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso

You may want to take a look at this thread, where the problem was due to the fact that some of the hosts did not have the NX (no-execute) feature enabled in the BIOS: http://communities.vmware.com/thread/159379

Incidentally, the G1/G2 hosts vary considerably, but the G2's seem to be identical to the G5's - they use the same BIOS. Also, the "G5" stickers on the front of our hosts fell off the other day, revealing a G2 label underneath Smiley Happy

Reply
0 Kudos
sheetsb
Enthusiast
Enthusiast

I finally have some clusters up and running with ESX 3.5 Update 2 and VC 2.5 update 2. One cluster consists of BL465c G5 servers (quad core 2356 Opteron) and DL385 G1 servers (dual core 285 Opteron). I created the cluster without any VMs and activated EVC. I've since been able to move many of our VMs to the cluster with only messages that the CPU mask will be modified to maintain CPU functionality for the VM. We have a few VMs with incompatible masks which will require a shutdown and reset to the default mask. Otherwise it seems to be working well.

Unfortunately, you can't just upgrade the cluster in place because it requires a complete shutdown of all the VMs in the cluster to activate EVC. That's why I created an additional cluster with some new and some old machines first. Once I had all the machines at the same ESX version I turned on EVC and started migrating VMs.

Bill S.

Reply
0 Kudos