We are looking to add one or more of the new HP DL585G5 servers with quad-core Opteron processors to an ESX 3.0.2 server farm with several DL585G2 dual-core Opteron servers. I looked at the compatibility document, but have not yet been able to make any sense out of it.
Question is, are the dual-core and quad-core Opteron processors VMotion compatible?
Rob Nixon
I cannot speak personally to the Opterons, but I can tell you that we have two ESX hosts with different core counts. One is dual-core and the other quad-core. VMotion works great and I have never had any problem with it.
If you go from a VM created on DL585G1 to the DL585G5 you'll have no problems. You might have problems from any VM created on the DL585G5 going to the DL585G1.
I'm still investigating this question. I found, on on the HP web site, the list of HP servers and processor groups that are VMotion compatible. Our existing servers are listed, DL585G2 with Opteron 8220SE processors, but not the newer DL585G5 with Opteron 8356 processors.
I have a case open with VMware support. Hopefully they will come up with the answer.
Thanks to everyone,
Rob Nixon
Definitely keen to have this compatibility confirmed...
/kimono/
We have had some masking issues between dl585g1's and dl385 g2's 585's are 880 and the 385's are 2200.
Which got worse after 3.5 starting adding lines into the vmx files. I had this all running under 3.0 with one
simple mask. Have yet to fully get it run under 3.5
vMotion between socket E and Socket F works fine in 3.x.x using this mask:
<<VM DEFAULT TAB>>
cpuid.1.eax = -
xxxxxxxx----
cpuid.80000001.ecx = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0
cpuid.80000001.edx = xx0xxxxxxxx0xxxxxxxxxxxxxxxxxxxx
<<AMD OVERIDE TAB>>
cpuid.80000001.edx.amd = -
0----0--
Can somebody post their socket E/F & quad-core mask when it's figured out?
Hi Rob,
have you got any informations from VMware on this topic ? I'm currently in the same situation as we are about to buy new DL385G5. We already have 385G1 and G2 and VMotion works with the CPU-mask applied. It is pointless to buy a G5 if we cannot VMotion between the hosts.
It is a pitty that the informations in the compatibility matrices haven't been updated yet.
Thanks
Thorsten
This issue is still not completely clear, but one of our guys was at the HP Conference last week and talked with an AMD rep. The word he got is that as of now the DL585G5 - "Barcelona" Opteron quad-core is not VMotion compatible with DL585G2 - Opteron dual-core. He was told that in the reasonably near future some changes in ESX would make it possible to VMotion between these two processors. We don't know a time frame for those changes.
That's the best I've been able to come up with so far.
We have just about decided that our 8 x DL585G2 ESX farm with 374 VMs is about big enough, so we are planning toward building a new farm with several new DL585G5s
Rob,
he might be referring to a long waited technlogy that would allow you to vmotion across different processor generations. In order to make this happen support must be built into the processor and into the processor.
I have been looking more at the Intel house of the things for a number of reasons lately (i.e. there was not so much to look at the AMD house and also it looks like Intel was more fashionable than AMD for more customers ... but I am sure that as all fashions ... this will change again). For Intel in order for this to happen you('ll) have to have at least Penryn based CPU's cores (i.e. the lastest 52xx and 54xx QC cpu's for the 2 socket space or Dunnington which is the new 6 cores CPU for the 4 socket and above space). These processors have built-in morphing features that allows them to present themselves as "old kits" when it is required. That way you'll be able to VMotion between one of these beasts and say ... a Tulsa based box (like the IBM 3950/3850 ... which i think would be a DL580G4 if I get it right). This will only be possible with VMware software support which should happen shortly anyway (sorry can't post here details as you know VMware is pretty paranoid about people disclosing stuff).
So .. back to the point.... I think that Barcelona has this hardware built-in support (to be confirmed as I am not sure) and they are waiting for VMware to come out with the update to intercept it. I am not even sure if the same VMware update will work for both Intel and AMD cpu's....
Massimo.
In kb 1992 there is a discussion about how to make the Barcelona chips vmotion compatible with earlier processors. It specifically states "...apply 3DNPREFETCH mask (not supported)." I haven't tried it yet but I may. I do feel a bit squeamish about using an unsupported configuration in production.
Bill S.
Found this interesting whitepaper on the AMD web site: http://developer.amd.com/assets/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf. Page 16 has a very nice diagram indicating VMotion support.
It would seem the technology is definitely in the processor, even allowing original single-core (Rev. C) Opterons to VMotion to the newer Barcelona, but as Massimo said it's dependent on a software update from VMware.
A very similar whitepaper can be downloaded from The Register: http://whitepapers.theregister.co.uk/paper/download/375/live-virtual-machine-migration-on-amd-proces...
It contains exactly the same information and was also created by AMD. The layout looks different and the very intersting part is at the end where they show a way how to migrate VMs between 4-core CPUs and older ones.This is still beta code but it proves that there is a way to do it.
Regards
Thorsten
I have gotten vmotion to work from a DL585 G1 with the Dual Core Denmark Opteron (model 880) to a DL585 G5 with the Quad Core Barcelona Opteron (model 8356) with this mask:
cpuid.1.eax = ""
cpuid.1.ecx = "--
0
"
cpuid.1.edx = ""
cpuid.80000001.eax.amd = ""
cpuid.80000001.ecx.amd = "
00000000000"
cpuid.80000001.edx = "
00-00--
"
cpuid.80000001.edx.amd = "-0
0-0
"
cpuid.80000001.ecx = ""
cpuid.1.ecx.amd = "
0
0-"
0
cpuid.1.eax.amd = "--00----00000000"
00000
but it fails when I try to vmotion the vm back from the G5 to the G1, with the message,"a general system error occurred: invalid fault."
A more reliable and supported way would be to just upgrade the farm to 3.5u2 (with the licensing patch, of course ) and enable EVC.
We tried this but cannot add G5 to a G1/G2 cluster and vice-versa. They must be using diffent masks between them.
I thought that EVC was going to help in this regard however it just makes it worse (or at least no better).
Glen
That's odd, this is what EVC is supposed to be able to do.
Is the cluster you're trying to move the hosts into enabled for EVC? Can you provide some details about the steps you're following / messages you're getting?
You may want to take a look at this thread, where the problem was due to the fact that some of the hosts did not have the NX (no-execute) feature enabled in the BIOS: http://communities.vmware.com/thread/159379
Incidentally, the G1/G2 hosts vary considerably, but the G2's seem to be identical to the G5's - they use the same BIOS. Also, the "G5" stickers on the front of our hosts fell off the other day, revealing a G2 label underneath
I finally have some clusters up and running with ESX 3.5 Update 2 and VC 2.5 update 2. One cluster consists of BL465c G5 servers (quad core 2356 Opteron) and DL385 G1 servers (dual core 285 Opteron). I created the cluster without any VMs and activated EVC. I've since been able to move many of our VMs to the cluster with only messages that the CPU mask will be modified to maintain CPU functionality for the VM. We have a few VMs with incompatible masks which will require a shutdown and reset to the default mask. Otherwise it seems to be working well.
Unfortunately, you can't just upgrade the cluster in place because it requires a complete shutdown of all the VMs in the cluster to activate EVC. That's why I created an additional cluster with some new and some old machines first. Once I had all the machines at the same ESX version I turned on EVC and started migrating VMs.
Bill S.