clarkwayne
Enthusiast
Enthusiast

AMD FX 8150 nested virtual ESXi black screens on power up.

Jump to solution

Background:

-I have followed all guildelines and suggestions when running ESXi as a virtual machine, and then running 64-bit operating systems in it.

-SVM is turned on in the bios.

-As you can see here (http://support.amd.com/us/kbarticles/Pages/GPU120AMDRVICPUsHyperVWin8.aspx) the AMD FX 8150 (Zambezi) supports AMD-V and RVI.

-For some reason Windows 7 64-bit powers on perfectly fine, but when powering on Windows Server 2008 R2 and CentOS 64-bit, all I get is a black screen.

It would be great if someone could help me or atleast explain to me why Windows 7 64-bit works but the other two don't.

Thanks.

My Blog: http://vflent.com
0 Kudos
1 Solution

Accepted Solutions
admin
Immortal
Immortal

clarkwayne wrote:

I figured it was something like that... only if I knew before I bought it. Would you mind providing a link to an article or something that goes over that?

Thanks.

I don't believe this issue is documented anywhere except this thread.

View solution in original post

0 Kudos
29 Replies
clarkwayne
Enthusiast
Enthusiast

I forgot to mention I'm running ESXi 5 update 1 on both the physical host, and the virtual host.  Running the latest BIOS revision for my mother board also (Asus M5A99x EVO).

My Blog: http://vflent.com
0 Kudos
kooltechies
Expert
Expert

You can look at this KB there can be multiple reasons why this would be the case. Have a look at this.

http://kb.vmware.com/kb/749640

Blog : http://thinkingloudoncloud.com || Twitter : @kooltechies || P.S : If you think that the answer is correct/helpful please consider rewarding points.
0 Kudos
clarkwayne
Enthusiast
Enthusiast

I'm sorry, but the console does come up. The VMware BIOS logo shows, and then it just goes black. If I hard boot it, I get the options to start windows normally, last known good settings, etc. After I select "Start Windows Normally" the console gets bigger for about 10 seconds, and then get's smaller.

Thanks.

My Blog: http://vflent.com
0 Kudos
admin
Immortal
Immortal

There is a bug in the virtualized AMD-V implementation of ESXi 5.0 with respect to Bulldozer CPUs.  I do not believe there is a workaround.

clarkwayne
Enthusiast
Enthusiast

I figured it was something like that... only if I knew before I bought it. Would you mind providing a link to an article or something that goes over that?

Thanks.

My Blog: http://vflent.com
0 Kudos
admin
Immortal
Immortal

clarkwayne wrote:

I figured it was something like that... only if I knew before I bought it. Would you mind providing a link to an article or something that goes over that?

Thanks.

I don't believe this issue is documented anywhere except this thread.

View solution in original post

0 Kudos
admin
Immortal
Immortal

The bug I spoke if is supposed to be fixed in ESXi 5.0 U1, so perhaps this is a different issue.

0 Kudos
clarkwayne
Enthusiast
Enthusiast

aw man. I upgraded ESXi 5.0 to 5.0 Update 1. Should I try reinstalling it completely? or any suggestions?

My Blog: http://vflent.com
0 Kudos
admin
Immortal
Immortal

Jim Mattson wrote:

The bug I spoke if is supposed to be fixed in ESXi 5.0 U1, so perhaps this is a different issue.

It turns out that the bug is only partially fixed in ESXi 5.0 U1, so what you are experiencing is probably the same bug.  I'm afraid you'll have to wait for the next release of ESXi.

0 Kudos
clarkwayne
Enthusiast
Enthusiast

The problem is all fixed in ESXI 5.1.

Everything works, if anyone is looking into buying the FX8150 over an i7, do it. It's worth every penny. After owning an i7 2600k whitebox for about a year, and an 8150 whitebox for 6 months, I wish I had bought an 8150 instead of an i7 before.

My Blog: http://vflent.com
0 Kudos
Najtsob
Enthusiast
Enthusiast

Can you please explain why is FX-8150 better than i7 ?

I am just buying new system and I cant decide wheter to choose FX-8150 or i7 3770.

FX-8150 - 4 modules, 8 integer cores, shared l2 cache and FPU

i7 3770 - 4 cores, 8 threads

I am guessing that you are better off with 8 integer cores than with combination of 4 "real" cores and 4 "virtual" (hyperthreaded) cores. But this is just my guess and I really like to hear some other opinions and arguments.

I will use workstation 9 and will run the following VM's simoultaneosly:

- esxi 5.1 4vCPU

- NexentaStore 2vCPU

- Domain Controler 1vCPU

Best regards, Primoz

0 Kudos
clarkwayne
Enthusiast
Enthusiast

If you are going to run ESXi on workstation (and I assume your underlying OS is windows 7), I would get the i7, just because Window's really doesn't know how to effectively use 8 cores... so the hyperthreading on Windows 7 will help. Also, get the K model of the i73770 so you can over clock it, the only difference VMware wise between the 2 is the non-K supports VD-d (direct passthrough), and the K doesn't, but if you're using it with Workstation 9 on Windows 7 it doesn't matter anyways.

If you want to see why Windows 7 sucks on AMD's compared to Intel: http://www.agner.org/optimize/blog/read.php?i=49 (basically Windows 7 uses the Intel compiler, and business wise, why would Intel write the compiler to be optimized with anything else other than their own processors). Key thing to take is AMD's compiler support's Linux, which ESXi is based off of. Thats why I believe the FX beat's the i7 in ESXi but not workstation.

The FX8150 system will cost you way less. If you run ESXi, and get a motherboard that supports VT-d (like the one I have, ASUS m5A99x), you can passthrough an AMD/ATI GPU to a virtual machine, hook up a monitor, and use that as your "desktop". The performance is flawless.

On a side note, the CPU isn't going to be your bottleneck, get 32gb ram (1600 PC12800 32gb ram costs $150 on newegg), and get an SSD ($130 for an OCZ Vertex 3 (500mb read and write, ~20k disk i/o)).

Let me know if you have anymore questions, or if any of that didn't make sense,

-Clark.

My Blog: http://vflent.com
0 Kudos
Najtsob
Enthusiast
Enthusiast

Thank you very much for you awesome post.

My first plan was to build ESXi whitebox with esxi 5.1 which could also be used as a workstation, but after few days of googling I realized that very few consumer grade motherboards has proper (working) implementation of IOMMU.

I made the list of AM3+ motherbords that should have working IOMMU support, but there was always some doubts.

- ASUS M5A99X EVO

- Gigabyte GA-970A-UD3

- asrock 970 extreme4

- asrock 990fx extreme 3

- asrock 990fx extreme 4

Now when you say that ASUS M5A99X EVO really supports IOMMU I'm certain that I will build my system around this motherboard and with FX-8150 cpu. Can you please tell me which exact components did you use to build your system ?

Also M5A99X EVO motherboard should support ECC ram, did you maybe try this ? How important is ECC when you have 32GB of ram and your box runs almost 24/7 ?

Best regards, Primoz

P.S. Does Workstation 9 also supports device passthrough (VMDirectPath) ?

0 Kudos
clarkwayne
Enthusiast
Enthusiast

-Here's the list of what I have:

-On the m5A99x EVO, I have not tried ECC ram. Didn't see a need for it (plus I was in a rush and didn't really look into it). The m5a99x was down clocking my ram to 1333 automatically. It took me a while to get it right without having ESXi freezing because the ram wasn't stable. I'd make sure to get something that's on the compatibility list for the m5a99x, and then you shouldn't have a problem. My box run's 24/7 now and I haven't had any problems. It draws about 275 to 300 watt's of power on average, and peaks out around 350 to 400, compared to the i7 drawing about 200 watts on average and peaking out at 300.

-I got most of these parts at my local MicroCenter, the FX8150 was $200 and the m5a99x EVO was $80 with a discount.

-Originally I had not planned on passing my GPU through to a vm so I picked up a decent gpu since the m5a99x doesn't have an onboard one.

-If you plan on passing through a GPU to a vm, I'd make sure to get an ATI/AMD card. After tons of researching, people have difficulty getting consumer grade nVIDIA card's to work. The GPU's that are officially supported are high end nVIDIA ones, and can be used with multiple virtual machines are EXPENSIVE. I'm talking about the cost of your whole whitebox + some: http://communities.vmware.com/thread/415887.

And, if you go to the first comment it shows you how to get a consumer grade gpu to work perfectly (again, ATI/AMD, not nVIDIA): http://tinkertry.com/gpu-pass-future-test/

-I have not had a chance to play with workstation 9 yet so I can't tell you and be a 100% sure, but after some quick googling, it doesn't seem like it supports passthrough...

Writing all this has inspired me to start a blog that describles my whole setup, how much it cost, what works, what doesn't, what I would've done differently, and all the cool stuff I'm doing with it soon.

Regards,

-Clark.

My Blog: http://vflent.com
0 Kudos
Najtsob
Enthusiast
Enthusiast

What is the whole experiance usign VM as workstation ?

So you first install esxi, then create win7 VM, enable automatic VM startup, configure graphics card passthrough, reboot system and after reboot you first see esxi console and when Win7 VM boots up you get W7 on your monitor ?

How about soud card, usb and other peripherals ? do they work in such setup ?

Best regards, Primoz

0 Kudos
clarkwayne
Enthusiast
Enthusiast

-What you described should theoritically work, but I wouldn't recommend it at all. I'd make sure you have a 2nd machine to manage the ESXi box through the vSphere client.

-If the graphics card has HDMI, the HDMI will work for Audio and Video.

-(See attached Pic) ESXi will not allow you to pass through all peripherals, only supported ones. On the m5a99x, the on board NIC, USB slots, and Sata Controller are able to be passed through. The other Gigabit ethernet controller's you see there are my added Intel/Broadcom Nics.

-If you only have 1 GPU installed, you will lose access to your ESXi local console once passthrough is enabled on it.

The way I have it (you need a 2nd box to open up the vSphere console for the virtual machine, until you attach a wireles mouse and keyboard usb reciever to the VM, you have to use the vSphere console as your mouse and keyboard):

1) Install ESXi.

2) Create new VM, install Windows 7, patch it and install all software. Make sure to give it no more than 2gb of ram.

3) Shutdown the VM, take a snapshot.

4) Enable passthrough for the GPU.

5) Reboot the ESXi box.

6) Add the PCI device(s) on the VM.

7) Add a reservation for the amount of ram allocated to it.

😎 Power on the VM (Make sure a monitor is plugged in on the GPU).

9) Installed the drivers.

10) Reboot, and then shutdown again.

11) Change the Ram to above 2gb if desired, add the reservation for the amount of ram allocated.

12) Add a USB controller.

13) Edit the .vmx file to add the "pciHole.start = "1200"" and "pciHole.end = "2200"".

14) Plugin a wireless mouse and keyboard usb reciever.

15) Power on the VM and attach the usb reciever to it.

16) Enjoy Smiley Happy

My Blog: http://vflent.com
0 Kudos
Najtsob
Enthusiast
Enthusiast

But theoretically it should be possible to passtrough USB controller or maybe separate USB card so the mouse and keboard would work right afer Windows VM boots up ?

If i remeber corectlly you can passtrough 6 devices per VM when using esxi 5 or 5.1, so should be possible to use graphics card, sound card and USB Expansion Card.

0 Kudos
clarkwayne
Enthusiast
Enthusiast

Yes it should be possible. I haven't tried this because I am running ESXi off a 2gb USB stick and if I passthrough the USB, it might create a blackhole...

If you aren't in a hurry to build this, I can test the setup the way you want it on Thursday and let you know.

My Blog: http://vflent.com
0 Kudos
Najtsob
Enthusiast
Enthusiast

That would be great if you can do this.

What VMs do you run on your setup and how high is CPU utilization ? I can't decide between AMD FX-6100 and FX-8150, because of 30W lower TDP of the first cpu.

0 Kudos