Does Fault Tolerance or vMotion fail between Sandy Bridge Core i7-2600 and Xeon E3-1200 models?

[wgillick]

I have searched the Intel and VMware documentation for this answer...

Do the advanced VMware features such as Fault Tolerance (FT), High Availability (HA) and vMotion even work on non-Xeons like the 2nd generation Sandy Bridge Core i7 models such as i7-2600 and i7-2600K? Per the latest Intel "Software Developer’s Manual, Volume 3B", the FamilyName_ModelName is the same (06_2AH) for both the Xeon E3-1200 series and the Core i7 series. However, nearly all VMware documentation describes only Xeon implementations:

FT (Fault Tolerance) requires the highest level of CPU compatibility in order to implement VMware's "vLockStep" method of CPU synchronization. Five levels are taken from a recent VMware compatibility guide note:

- Intel Xeon Core 2
- Intel Xeon 45nm Core 2
- Intel Xeon Core i7
- Intel Xeon 32nm Core i7
- Intel Sandy-Bridge Generation

If there is a ESXi enthusiast who has successfully implemented vMotion across Sandy Bridge systems, your detailed configuration information would be greatly appreciated.

Note, this is a build-in-progress query. The CPU will be used in an ASUS P8B WS.server motherboard that supports either Xeon or Core i7 Sandy Bridge models.

Regards,

Will Gillick

[Troy_Clavell]

thread moved from VCP: VMware Certified Professional to Availability: HA & FT

[DSTAVERT]

Check to see whether the two processors are in the same EVC class. http://kb.vmware.com/kb/1003212

[wgillick]

Thanks for the insta-reply!

Yes, I have been poring over the VMware KB's, including 3212, but even the detailed "Table 3.1: EVC Baselines on Intel CPUs" is inconclusive. The light blue section of the table should hold the answer, but it possibly erroneously states:

"Intel® 32nm Core™ i7 (Westmere)"

Intel refers to Westmere as their Xeon 5600 E7-series. Sandy Bridge is the newer version of both desktop (Core i7) and server (Xeon) products. Also there are 2 generations of Sandy Bridge, so the confusion comes down to these questions:

- Is the latest (2nd-Gen) Sandy Bridge supported as a subset of Westmere in VMware ?

- Does vMotion, HA and FT work on non-Xeons, especially the newest i7-2600?

Thanks,
Will Gillick

DSTAVERT wrote:

Check to see whether the two processors are in the same EVC class. http://kb.vmware.com/kb/1003212

[DSTAVERT]

Try the HCL which may be more helpful comparing CPUs for FT. You can refine your search by FT and EVC http://vmware.com/go/hcl

[wgillick]

Thanks DStavert.

But, I am still looking for the magic info. However, I decided to build my servers anyway to see if I could make it work. I just finished the build on a Sandy Bridge Core i7 server with 8GB of non-ECC DDR3. However, my first test with CPUID was not successful: "No FlexMigration support"

That doesn't sound very good. However, I will try to update the motherboard firmware to make it work. I hope this means that the motherboard manufacturers are simply slow to implement the VMware-ready features. Otherwise, it would mean that Intel has decided to disallow vMotion support on their Core i7 CPUs. That would be terrible move. So, I'm hoping to test this new firmware in the next few days.

By the way, I am using an ASUS server-class motherboard P8BWS (similar to the SuperMicro X9 series).

--Will

DSTAVERT wrote:

Try the HCL which may be more helpful comparing CPUs for FT. You can refine your search by FT and EVC http://vmware.com/go/hcl

[WillL]

Did you get a chance to actually test them out?

According to current HCL, among non-Xeon SB CPUs only i3-2100 support FT (Fault Tolerant Compatible Set: FT-SandyBridge).

vMotion should work, EVC Modes: Intel® Sandy-Bridge Generation. CPUID might not be up to date.

Good luck!

[wgillick]

Thanks WillL,

I finally built a Sandy Bridge motherboard up into a decent desktop case for testing, and I have some partial results. The system runs fine, it boots directly into ESXi 4.1 from a USB drive with no complaints. That is because I am using a server-quality motherboard from Asus with the newest 0605 firmware:

   Asus P8B WS

and CPU:

   Intel Core i7-2600

This motherboard has (2) highly-compatible Intel NICs and very plain SATA features, so ESXi compatibility should be great.

However, when I ran a VMware test tool called "CPUID" in this downloaded file: "VMware-CPU-Compatibility-e.x.p-160658.zip", it reported that both FlexMigration and EVC modes are not supported. Huge disappointment.(I will post the screenshots when I get home tonight.)

However, there's a possibility that the tool is simply outdated and didn't recognize the CPU. I could not find a newer tool either on VMware or elsewhere. So at this point I have a dilemma to test vMotion and FT:

a) Do I ignore the CPUID result, buy a 2nd i7-2600, and build another ASUS board to test the pair?

b) Do I buy (2) new Sandy Bridge Xeons and build up a system to test those? (now I have 3 expensive CPUs.)

c) Do I buy (1) Sandy Bridge Xeon and test vMotion and FT against the i7-2600 as I originally intended?

d) Or, wait for someone else to do it.

If I had CPUs laying around, then maybe I could just give it a try. But it would be nice if VMware kept their tools and documentation updated on one of the most popular CPUs in a long time. Or maybe I could just go and buy AMD processors and have it "just work".

Hmmm, decisions...

--Will

William wrote:

Did you get a chance to actually test them out?

According to current HCL, among non-Xeon SB CPUs only i3-2100 support FT (Fault Tolerant Compatible Set: FT-SandyBridge).

vMotion should work, EVC Modes: Intel® Sandy-Bridge Generation. CPUID might not be up to date.

Good luck!

[wgillick]

I did a CPUID compatibility test with the new Intel Sandy Bridge CPU: Core i7-2600. However, even with the latest motherboard (Asus P8B WS) and latest BIOS (0605), according to the CPUID tool, Intel's FlexMigration does not work on this processor! Using CPUID from a recent download (VMware-CPU-Compatibility-e.x.p-160658.zip), I get this result when booting my Sandy Bridge host:

Does this mean that VMware only supports server-specific Xeon processors? In that case, what special feature of Xeons does FlexMigration and vMotion require? Or, is it possible that the CPUID utility has not yet been updated for 2nd-gen Intel CPUs like the Sandy Bridge Core i7-2600? My CPU has "Family and Model Number" = 06_2Ah. This chart comes from the Intel System Programming Guide, Volume 3B:

After a few hours with the Intel docs, it looks like VMware *can* determine a specific model of "Xeon" Server versus "Core i7" Desktop processor based on a few differences between the Core i7 / E3 Xeon class processor:

- model-specific registers

- ECC vs. non-ECC support

- number of bytelanes

However, this brings up three questions:

- Why does VMware report no FlexMigration support when mere VT-x is the essential flag?

- Why hasn't VMware updated their CPUID tool for over 2 years.

and most importantly:

- How can any individual user of limited means know in advance if a processor will be truly incompatible or merely flagged incompatible?

If there are any members who have a more recent CPUID.iso tool than April 2009, I would appreciate that. I believe there is another tool out there, perhaps it is "SiteSurvey", but it doesn't run on my system.

Any other VMware explorers out there willing to try a new 2nd-gen Sandy Bridge CPU (Xeon or i7) in a LGA1155 socket?

Note, see this thread for a similar problem by another member:

http://communities.vmware.com/message/1722563#1722563

__Will

[WillL]

SiteSurvey is a vCenter plug-in, needs to connect to a vCenter instance to run, it checks for FT compatiblity.

I'm guessing all desktop Nehalem/SB CPUs technically support FT, but majority of them are not added to VMware's compatibitliy database only if a hardware vendor used the CPU in their enterprise servers and requested VMware to certify the server including the CPU.

FT does not use a specific CPU feature but requires specific CPU families to function. vLockstep is more of a software solution that relies on some of the underlying functionality of the processors. The software level records the CPU instructions at the VM level and relies on the processor to do so; it has to be very accurate in terms of timing and VMware needed the processors to be modified by Intel and AMD to ensure complete accuracy. The SiteSurvey utility simply looks for certain CPU models and families, but not specific CPU features, to determine if a CPU is compatible with FT. In the future, VMware may update its CPU ID utility to also report if a CPU is FT capable.

I'm wondering if there is a list of supported models on ESXi, and if we can add to it ...

[WillL]

I "borrowed" an i5-2500k and added to a cluster running 4.1 U1 with 1st gen i7 hosts (bloomfield). After setting up EVC to "Intel® Xeon® Core™ i7", the CPU compatibility error went away, so I assume vMotion would work after shared storage is setup properly. But I don't see "Intel® Sandy-Bridge Generation" in 4.1, maybe it will come up in 5.0.0

Site Survey says i5-2500k is not FT capable.

[wgillick]

Thanks, William!

It's terrible news for me, I'm afraid.

But thank you so much for testing for me (and everyone else considering using the latest Intel CPUs). I still find it incredible that VMware has not made it crystal clear what modern CPUs will work with their best features. Of course there's the HCL, but that only works for some. If all the new owners of HP and Dell small business computers with the newest Sandy Bridge i5 and i7 CPUs discovered their systems are not compabible, that could be a big PR headache for Intel & VMware. I only "wasted" $850 on a hand-build Sandy Bridge server--but what about those who bought the newest $2000 Dell boxes?

Unfortunately, it's too late for me to get an early copy of vSphere 5.0 for testing. I've only been working on this VMware project for about 7 months, so I don'tknow how to get "invited" to the party at this point. Still, I'm hoping that VMware will make some concessions--I just can't believe a huge product release like Sandy Bridge is not supported--this is Intel's forte!

Anyway, I apologize for the tone of this message. It is just a huge frustration. And I do thank you again for testing the Sandy Bridge.

__Will (San Jose)

update:

I just realized that you tested the "K" model of the 2nd-gen Sandy Bridge. The i5-2500K is the unlocked enthusiast version of the Intel Sandy Bridge without some key features like VT-d and TXT (Trusted eXecution Technology). So, there's still hope that the standard i5-2500 or i7-2600 models will support FT. My dilemma is that I will have to build or buy TWO Sandy Bridge servers in order to verify that a key VMware feature will work. Not sure if I can afford it on my household budget.

Intel CPU reference pages for the i5-2500K, i5-2500, and i7-2600:

------------------------------------ ------------------------------------ ------------------------------------

http://ark.intel.com/products/52210?wapkw=i5-2500k

http://ark.intel.com/products/52209?wapkw=i5-2500

http://ark.intel.com/products/52213?wapkw=i7-2600

------------------------------------ ------------------------------------ ------------------------------------

William wrote:

I "borrowed" an i5-2500k and added to a cluster running 4.1 U1 with 1st gen i7 hosts (bloomfield). After setting up EVC to "Intel® Xeon® Core™ i7", the CPU compatibility error went away, so I assume vMotion would work after shared storage is setup properly. But I don't see "Intel® Sandy-Bridge Generation" in 4.1, maybe it will come up in 5.0.0

Site Survey says i5-2500k is not FT capable.

Message was edited by: wgillick, (added i5-2500K details)

[WillL]

I'm not sure if non-K will make any difference in term of FT capability. VT-d is for VMDirectPath and TXT is more security/TPM.

The funny thing is i3-2100 is FT compatible while i5-2500/i7-2600 are not, the same thing happened to first gen: i3/i5 clarkdale is good with FT, but my i7-960 is not. VMware must think dual core is better than quad core. The reason is probably as I posted earlier.

I know your pain because I'm on the boat, maybe you can sell i7-2600, and replace with Xeon E3-12xx about the same price.

[DSTAVERT]

I still find it incredible that VMware has not made it crystal clear what modern CPUs will work with their best features. Of course there's the HCL, but that only works for some. If all the new owners of HP and Dell small business computers with the newest Sandy Bridge i5 and i7 CPUs discovered their systems are not compabible, that could be a big PR headache for Intel & VMware. I only "wasted" $850 on a hand-build Sandy Bridge server--but what about those who bought the newest $2000 Dell boxes?

While I understand your frustration do understand that vSphere ESX(i) is an enterprise operating system and expects to find enterprise grade hardware. While a gaming machine may benefit from "the latest CPU" a server is as much about stability as anything else. In a production environment skimping on hardware is false economy. You want the highest level of stability possible. You want the basic hardware to provide the stability so that you don't need rely on HA and FT to maintain your production workloads. Given that Fault Tolerance requires at least Enterprise you would be paying as much or more for your VMware licenses as for a mainstream server. Even HA requires a substantial investment in licensing.

In my opinion, building a production server from components it is no longer a viable option. There are just too many problems trying to integrate components and keep them updated and working well together. With a mainstream server, firmware is updated as a whole. Slight differences in component timing etc can be and are corrected with firmware. I use HP servers and the firmware updates are applied to all components --BIOS, disk controller, network, baseboard and go so far as updating hard drive firmware. That isn't something a self builder can do. Most motherboard manufacturers don't update BIOS software much beyond the manufacturing cycle and other components may never get updates. For hobby and testing a self built host machine may be OK but production requires a mainstream server that conforms to the VMware HCL. Anything less is risking the financial health of a business.

[WillL]

In terms of stability, is the following true?

• i3-550 > i7-960 ?
  
• i3-2100 > i5-2500 or i7-2600 ?
  

Dual core i3 are in HCL but not the quad core i5/i7, makes no sense to me. Is this VMware's fault? not entirely I think.

Let's see what Intel's Sandy Bridge server chipset C202/C204 support:

• One Quad-Core Intel® Xeon® processor E3-1200 product family
  
• One Dual-Core Intel® Core™ processor i3-2100 product family
  

Looks to me Intel wants you buy quad core Xeon instead of i5/i7 which are left unsupported. VMware isn't going to against Intel.

Interestingly, Intel didn't produce a C206 based board itself and let ASUS debut P8B WS which has i5/i7 support "added", if a server vendor like Dell create a server line based on C206 and support desktop i3/i5 processors, VMware would add FT support accordingly. But I doubt this would happen, in the end C206 is workstation not server chipset.

By the way, the offical spec of P8B WS:

• Intel® Socket 1155 for 2^nd Generation Core™ i3 Processors
  
• Intel® Socket 1155 for Intel® E3-1200 processors
  

With no mention of i5/i7, go figure.

[wgillick]

Hmmm, good research on that, WillL.

I suspect the whole Sandy Bridge segmentation really confused everyone, including Intel and VMware. Not to mention, the motherboard vendors. The Cougar Point recall in Jan/Feb probably muddled the nice clean re-release of Sandy Bridge. I had been searching for a good server board from SuperMicro that would handle both ECC and non-ECC when I encountered the ASUS P8B WS wunderboard! Both ECC and non-ECC DDR3, both Xeon and i7 support, and finally a board that enabled the VT-d flag.

I am still hoping for a reprieve from Intel or VMware. The Xeon's are made from the Core i7 technology, after all. Even VMware mentions it, but not quite clearly.

__Will (San Jose)

William wrote:

In terms of stability, is the following true?

• i3-550 > i7-960 ?
• i3-2100 > i5-2500 or i7-2600 ?

Dual core i3 are in HCL but not the quad core i5/i7, makes no sense to me. Is this VMware's fault? not entirely I think.

Let's see what Intel's Sandy Bridge server chipset C202/C204 support:

• One Quad-Core Intel® Xeon® processor E3-1200 product family
• One Dual-Core Intel® Core™ processor i3-2100 product family

Looks to me Intel wants you buy quad core Xeon instead of i5/i7 which are left unsupported. VMware isn't going to go against Intel.

Interestingly, Intel didn't produce a C206 based board itself and let ASUS debut P8B WS which has i5/i7 support "added", if a server vendor like Dell create a server line based on C206 and support desktop i3/i5 processors, VMware would add FT support accordingly. But I doubt this would happen, in the end C206 is workstation not server chipset.

By the way, the offical spec of P8B WS:

• Intel® Socket 1155 for 2^nd Generation Core™ i3 Processors
• Intel® Socket 1155 for Intel® E3-1200 processors

With no mention of i5/i7, go figure.

[wgillick]

Hi DStavert,

You are correct on all points. And I appreciate your very professional response. (some of my posts are not very subtle due to my frustration.)

In a sense, I am getting what I deserve--as a test engineer this is what I do, so I really should know better. By testing with un-proven CPUs and self-built hardware, I'm on my own, with only the scraps of bloggers and the denizens of forums to provide assistance. And as a former Intel technical marketing engineer, I really shouldn't be surprised about product segmentation between Xeon and Desktop. :>}

However, since this VMware test lab I'm building is for my own education, it's a bit more personal, and I'm just incredibly curious how it will turn out. I wish I could afford to lay out 4 SuperMicro X9-series boards and 4 ASUS motherboards (P8 and Z68), grab 4 processors from each family segment, and build my test matrix. (Which I would be very happy to do so, gratis, if I could get some samples from that nice company in Palo Alto. Waiting for that email...)

I'm seeking knowledge through bootstrapping myself into vSphere. I hope to obtain a VCP, and I hope to demonstrate FT and HA with 3 clusters of server classes: AMD 3rd generation, HCL-based Xeon X3400 Lynnfield, and 2nd-gen Sandy Bridge. I have 5 of the 6 nodes ready, but this 6th server is causing me headaches because I don't know what will work, and I can't just return Intel processors that don't work. And it just seems incredible to me that I still haven't found if a Xeon can talk to an i7 during vMotion or if an i7 can FT to another i7, and so on.

So, please excuse much of my rhetoric. It is simply the Lament of the Underfunded Test Engineer--one who seeks, but still has to eat.

__Will

D>STAVERT wrote:

I still find it incredible that VMware has not made it crystal clear what modern CPUs will work with their best features. Of course there's the HCL, but that only works for some. If all the new owners of HP and Dell small business computers with the newest Sandy Bridge i5 and i7 CPUs discovered their systems are not compabible, that could be a big PR headache for Intel & VMware. I only "wasted" $850 on a hand-build Sandy Bridge server--but what about those who bought the newest $2000 Dell boxes?

While I understand your frustration do understand that vSphere ESX(i) is an enterprise operating system and expects to find enterprise grade hardware. While a gaming machine may benefit from "the latest CPU" a server is as much about stability as anything else. In a production environment skimping on hardware is false economy. You want the highest level of stability possible. You want the basic hardware to provide the stability so that you don't need rely on HA and FT to maintain your production workloads. Given that Fault Tolerance requires at least Enterprise you would be paying as much or more for your VMware licenses as for a mainstream server. Even HA requires a substantial investment in licensing.

In my opinion, building a production server from components it is no longer a viable option. There are just too many problems trying to integrate components and keep them updated and working well together. With a mainstream server, firmware is updated as a whole. Slight differences in component timing etc can be and are corrected with firmware. I use HP servers and the firmware updates are applied to all components --BIOS, disk controller, network, baseboard and go so far as updating hard drive firmware. That isn't something a self builder can do. Most motherboard manufacturers don't update BIOS software much beyond the manufacturing cycle and other components may never get updates. For hobby and testing a self built host machine may be OK but production requires a mainstream server that conforms to the VMware HCL. Anything less is risking the financial health of a business.