8-Way Opteron Server Support, HELP PLEASE!

[softpointmedia]

Okay, so here is my overly complicated scenario, and BTW, I am a VMWare n00b, so bare with me:

Right now I have an 8-way Opteron server with 8x Opteron 850 (2.4GHz single core) CPUs and 32GB of memory running Windows Server 2008 x64 and our Exchange 2007 server. The server is an I-WILL H8501 for those interested () that I got over a year ago at an auction (it was a lot auction and I had no clue what I had gotten until we got it all back to the shop). What I want to do is convert the current server install to a VM and install ESX or ESXi 4.0 onto the box and virtualize all of our other various servers as well since as you can tell, this box is WAY overkill for what it is currently doing. To be honest, originally I was going to go Hyper-V, however since this is an older server and Hyper-V in all its glory only works with the newest latest and greatest hardware, it was a no-go. So I turned to VMWare which I have actually found I like better anyways since the hypervisor is so much lighter then Hyper-V anyways and the client is awesome.

So I had the great idea of pulling our dual processor Xeon workstation from the lab, and installed ESX 3.5 on there in eval (since 4.0 was giving me the CPUID error). The next step was to convert the physical server I want ESX on finally to a VM on the temp host server, then load ESX onto the final server and move the VM over to the new host server as a guest. Sounded awesome in theory.

The problem I am having is when I try and run the converter it tells me the temporary host does not support 64-bit guests and will not let me convert it. I tried a 32-bit server for fun and that worked perfectly. So I know the temp host is x64 compatible as we have had various x64 OS's on there when running different lab scenarios previously, and if I setup a new VM on there, it lets me install Server 2008 x64 no problem at all.

So then I converted another server over to the temp host, and tried the same thing on that server, which again I know is a fully x64 compatible server, even though it isn't Intel VT. Same problem.

I am totally stumped and could really use some guidance here as to what to do. We don't have the budget for any new servers right now, so I have to make due with that I have. I know I can setup a new Server 2008 x64 VM on the temp host, load Exchange 2007 and move the mailboxes over, and then uninstall it from the 8-way server and demote it out, but that is a ton of work that i would rather avoid if possible. And then I still don't know if after all that I will be able to move the guest over to the final server or if I will get stuck with the same error.

PLEASE HELP!

-Brian

[dburgess]

What I think is happening here is that the server h/w on your interim system is Intel based and that will require VT features in order for the x64 based guest operating system to work.

In ESX we use features of Intel VT to make the monitor code (the core part of the hypervisor specific to each guest) safe from the VM.

The reason for this is a bit complex but simply - in 64 bit mode Intel dropped a feature called Segment Limit Offset check. This basically means on Intel machines we require the system to support VT in order to run x64 guests; even though they are capable of running x64 natively. On AMD we use the older binary translation technique because a) its a bit faster and b) it will work on older systems. I'm pretty sure you could confirm this with a cpu utility we provide that you can boot up, I'll try and dig out the link to this.

Cheers,

dB

[dburgess]

Have not found the cpu util - but the KB article 1901 is worth looking at.

dB

[softpointmedia]

Thanks for the info. I did find some more info and did find the aforementioned utility. That also confirmed that the CPUs in the H8501 (1st gen Opteron 850's) were NOT capable of running 64-bit guests. I ordered 8 Opteron 852's after confirming they were gen E Opterons that would support 64-bit guests (I am praying). I successfully migrated to a different swing server that had an Intel VT CPU, and then when we upgraded the CPUs in the main server, we had to update the BIOS and then BAM, the new BIOS that we have to run to support the 852 CPUs does not support the RAID card. So right now we are running on the swing server until we can get a new RAID controller in hopefully by Tuesday. Also, this is the first time we have ever run 8 CPUs in this server (only had 4 previously) and it is not seeing the second motherboard and 4 CPUs. I have call in to Flextronics who bought out IWILL for that, but I doubt I will get very far with them. So far this project has been a nightmare, but hopefully it will be over soon and we can move forward.

[azn2kew]

Some basics to understand when running VMware hypervisors.

1. Your BIOS must have VT enable and your systems capable of support x64 bit to run x64 OS/applications such as Exchange 2007.

2. Your hardware must be on the Hardware Compatability List

3. Upgrade your BIOS/Firmware and drivers to the latest to prevent from issues and bottleneck.

4. Stress test it for 48 hours before deploy ESX 4.0 and load VMs.

I suggest you look at your vendor website and check your hardware specifications for details or use CPU Compatiblity tool from VMware to validate it if it can runs with ESX 4.0

Once you have the CPU requirements in placed, you should be fine with P2V converter. Onced you configured the cluster, if you have VMotion CPU issue, then you can use advance settings to hide the CPU NX flag or use Enhance VMotion Compatability (EVC) feature if allow.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[TomHowarth]

Stefan, this thread is about Opteron ie AMD why is VT Extensions of any interest at all as it is an Intel function?

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert

VMware Communities User Moderator

Blog: www.planetvm.net

Contributing author for the upcoming book "[VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment|http://my.safaribooksonline.com/9780136083214]”. Currently available on roughcuts

[azn2kew]

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[softpointmedia]

Thank you for the replies.

The physical former Exchange server has already been migrated to a 'swing server' running ESX 4.0 on an eval license since it is only temporary. The swing server is Intel-VT and runs the VM fine, albeit slowly. I have pretty much figured out all of the issues as far as VMWare is concerned I believe. I now know a number of things I didn't when I started this nigthmare project, such as that in order to run 64-bit guests the processors must be VT if they are Intel (which the swing server is, but it is only a single processor machine), or AMD-V OR at least a generation E Opteron if it is an AMD chip. Just being a 64-bit CPU is not enough. I purchased 8 Opteron 852's to replace the previous 850's since the 852's are generation E chips. Just because a processor is 64-bit compatible, does not mean it supports 64-bit guests. Case in point are my Opteron 850's which have been running Server 2008 x64 with Exchange 2007 for over a year now without any problems, yet those same 64-bit CPUs CANNOT run 64-bit guests. I have also found the compatibility tool to be pretty much useless in my case since no one has ever heard of this IWILL server, or its brothers the H8502 and H8503. In fact most people don't even know 8-way Opteron servers exist, and even out of those who do, few have ever seen one in person. I know I didn't and hadn't until I came across this box at an auction (got it for $400 if you can believe tha because the guy selling it didn't even know what it was).

Now of course the problems I am having are hardware with the server itself. I upgraded the BIOS to support the new 852 CPUs (well upgrade being a relative term since I went from a BIOS dated in 2005 to one from 2006, the latest they ever made), and then it decided it didn't like the Promise RAID controller i had in there anymore, so I had to purchase a new Adaptec 2820SA which should be compatible (hopefully will be in today). It has only ever had one motherboard installed with 4 cpus. and now it is not recognizing the second motherboard of 4 CPUs so I only have 4 CPUs being recognized out of 8. All of this on a server that was made by a company that no longer exists. I feel like I am the only person in the world who owns one of these behemoths.

I will keep this thread going for education's sake. If anyone out there has one of these servers, even if you aren't running VMWare on it, please post and help on the hardware issues. Thanks!

[dewbieZ]

Having the same exact problem DL585G1 Proliant AMD Opteron 850 2.4 4 way configuration.

livecd ~ # cat /proc/cpuinfo

processor : 0

vendor_id : AuthenticAMD

cpu family : 15

model : 5

model name : AMD Opteron (tm) Processor 850

stepping : 10

cpu MHz : 2396.927

cache size : 1024 KB

fpu : yes

fpu_exception : yes

cpuid level : 1

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow rep_good

bogomips : 4793.85

TLB size : 1024 4K pages

clflush size : 64

cache_alignment : 64

address sizes : 40 bits physical, 48 bits virtual

power management: ts fid vid ttp

When Virtualized CPU Flags Go to This

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext 3dnowext 3dnow constant_tsc up tsc_reliable

Can only boot 32bit guests.

[softpointmedia]

dewbiez;

There were 2 series of Opteron 850's made, the Sledgehammers which were rev CG, model number OSA850CEP5AV, and then the Athens series which were rev E4, model number OSA850FAA5BM. If you have Sledgehammers, you are out of luck, as I was. If you have the Athens, you should be okay. I was able to order 8 of the 852 Athens series off eBay for $175 shipped, which is pretty good considering that was for 8 cpus. Please look at the following resources that I found extremely helpful:

List of all Opteron series CPUs, with Revision Codes:

VMWare 64-Bit Processor Compatibility Check Tool:

VMWare KB Article:

[dewbieZ]

Yeah I just ran the GuestCheck tool on a 64bit Native LiveCD and it said I couldn't run 64bit Guests. Kind of makes me angry, but w/e. I don't really NEED 64bit Guests ATM. If I really need 64bit Linux guests in the future I will install Xen (i tried and tested this already and it will run 64bit Linux guests for me fine). At this moment I need this box to be able to run Windows Guests and Xen of course can't run Windows guests 32bit or 64bit if you don't have HVA (HVM) so its not an option for me. I'll just buy a new box in the future if I want 64bit Linux/Windows guests in ESXi/Xen and make sure to get an AMD-V/Intel-VT box now that I know all the quirks. I didn't really want to spend that much money and I got this DL585G1 for $410 with shipping included.

[softpointmedia]

Well, I agree, but for roughly $90 you can support x64 guests with THIS server on ESX/i with Opteron 852's. I just pulled this from eBay. This is the same seller I just got mine from, and he is pretty good, and very responsive. They might also negotiate on the price since you are buying 4. Just trying to be helpful.

[dewbieZ]

Nice. . . . but this still won't give me AMD-V (Pacifica) am I wrong? If I go through all this trouble again, I want to make sure I get something that can run Linux 64bit Guests and Windows 64bit Guests

[dewbieZ]

Thanks though for all your feedback its really appreciated

[softpointmedia]

Now I am DEFINTIELY NOT an expert, but from everything I have read, and if you read the KB article I sent you above carefully, VMWare clearly states all AMD Opteron processors that are rev E or above (which includes the Athens Opteron 852 which is Rev E4), should be compatible with 64-bit VMs. I am going to be rolling out 64-bit Windows and Linux guests under ESX 4.0 on our 8-way Opteron server running 852's this Saturday. If you can wait until after then, I will post to confirm it all works, but like I said, from everything I have read I am gambling I am right. I don't think AMD-V is 100% necessary to run 64-bit guests, but I believe that anything AMD-V is guaranteed to be compatible and it is strongly recommended, where as the others you have to do your homework. That is the theory at least....

[dewbieZ]

852's aren't AMD-V enabled I don't think. . . . HVA/HVM is NEEDED to run 64bit Windows Guests on Xen+ESXi and Needed to run 32bit WIndows Guests on Xen period.

[softpointmedia]

The Opteron 852's ARE AMD64, but ARE NOTE AMD-V compatible. I don't know anything about Xen, but in regards to ESX/ESXi, I believe you do not have to have AMD-V. One more time, from the KB Article I posted before:

KB Article: 1901

Updated: Jan 14, 2009

-

---

Products:

VMware Fusion

VMware Workstation

VMware ESX

VMware Converter

VMware ESXi

VMware ACE

VMware Player

VMware Server

Product Versions:

VMware ACE 2.5.x

VMware ESXi 3.5.x Installable

VMware Server 1.0.x (Linux Hosts)

VMware Workstation 6.5.x (Linux Hosts)

VMware ESXi 3.5.x Embedded

VMware Workstation 6.0.x (Linux Hosts)

VMware Workstation 5.5.x (Windows hosts)

VMware Workstation 6.0.x (Windows Hosts)

VMware Server 1.0.x (Windows Hosts)

VMware ACE 2.0.x

VMware Server 2.0 (Windows Hosts)

VMware Workstation 5.5.x (Linux hosts)

VMware Workstation 6.5.x (Windows Hosts)

VMware Converter 3.0.x

AMD

Segmentation support not present in the initial AMD64 processors (Revision C and earlier) while running in long mode. As a result, AMD64 processors prior to revision D do not have an efficient mechanism for isolating the virtual machine monitor from 64-bit guest operating systems.

A limited form of segmentation was reintroduced in long mode, in revision D AMD64 processors. As a result, AMD64 processors must be revision D or later to run 64-bit guest operating systems.

Note: Because AMD Opteron and Turion processors do not ship in revision D, AMD Opteron and Turion 64 processors must be revision E or later to run 64-bit guest operating systems.

Note: 64-bit guests are not supported for ESX versions 2.5.x and earlier. For AMD Opteron-based systems, the processors must be Opteron Rev E and later. To ensure your processors support AMD64, please reference the following link -

Intel

Intel CPUs require EM64T and VT support in the chip and in the BIOS.

I don't know what more to say other than I am going to try it this weekend and will let you know the outcome. But according to everything I have read, the 852's should work since they are Rev E, and they are AMD64. I know the 850's which are AMD64 as well but not Rev E ran Server 2008 x64 fine for over a year without an issue. I have learned that running a 64-bit OS and running 64-bit guests are two VERY different things though because of the processor segmentation.

[softpointmedia]

Anyone know if I can get points for answering my own question? Or do I just get the "Use the Search Feature, Stupid" award for that? :smileylaugh:

[TomHowarth]

As Softpointmedia states all AMD chips from Athlon rev. E are capable of supporting 64 bit guests.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert

VMware Communities User Moderator

Blog: www.planetvm.net

Contributing author for the upcoming book "[VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment|http://my.safaribooksonline.com/9780136083214]”. Currently available on roughcuts

[softpointmedia]

And I am only quoting the KB article I found on VMWare's site. Like I said before, I am going to test this for sure on Saturday and will update everyone of the results, although I am fairly confident that it will work. At least confident enough that I have spent a lot of money on new processors and more memory for this server, so we will see. Otherwise the whole lot is going on eBay if I fail.