Solved: Install ESX on top of ESX - can it be done?

AnnTelope · ‎11-24-2008

I have an ESX host which quite happily runs VMs - however ...when I try to install an ESX host as a VM

I get as far as the following .......

Mounting Local filesystems:

Savin boot logs:

Unmounting initrd:

Enabling swap space

INIT: Entering runlevel: 3

Entering non-interactive startup

Applying Intel IA32 Microcode update:

Starting vmkstart:

it then jumps to a black screen .....

If I start it in console only mode then I can log in and ping other machines and they can ping the console. Does anyone have any thoughts?

Has anyone managed to run ESX on top of ESX ?

admin · ‎11-26-2008

Tom is correct. This configuration only works for AMD64 chips with AMD-V and RVI. I misstated the configuration options in my previous post. They should be:

monitor.virtual_mmu = "hardware"
monitor_control.restrict_backdoor = "true"

Neither "monitor_control.vt32" nor "monitor_control.enable_svm" has any effect under ESX 3.5.

View solution in original post

manfred_w · ‎11-24-2008

ESX on top of ESX doesn't work.

Not supported and not functional.

AnnTelope · ‎11-24-2008

Im trying to run esx on top of esx: - it seems to install OK and I get as far as ......

Mounting Local filesystems:

Saving boot logs:

Unmounting initrd:

Enabling swap space

INIT: Entering runlevel: 3

Entering non-interactive startup

Applying Intel IA32 Microcode update:

Starting vmkstart:

then a blank screen .....

If I boot into service console only mode then I can ping other machines and they can ping the console but try to boot into ESX it only goes so far and is un pingable. There are numersoud posts on running ESX on top of Workstation 6 ( eg ) but I want to run ESX on top of ESX - Someone must have done it ?

AnnTelope · ‎11-24-2008

Why doesn't it work ?

Texiwill · ‎11-24-2008

Hello,

Not sure it will work. It is not designed to work that way. Workstation is designed to work as you describe. The key is how the host OS (in this case ESX) handles INtel VT/AMD-V for VMs.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

manfred_w · ‎11-24-2008

it's technically not possible to run the vmware hypervisor in a virtual machine because the hypervisor needs to have direct access to the underlying hardware. and in a vm the guest os (in your case: esx) has no direct access to the hardware.

Texiwill · ‎11-24-2008

Hello,

Not exactly true since it is possible to run ESX within a VMware Workstation VM. It is more about what the host presents to the VM.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

JDLangdon · ‎11-24-2008

While it is possible to have the COS load on top of ESX, I have yet to hear of anyone who has been able to get ESX to load on top.

The upside is that ESX will load on top of Vmware Workstation 6.5.

________________________________

Jason D. Langdon

AnnTelope · ‎11-24-2008

if ESX can sit on top of workstation then surely it does not have direct access to the hardware.

what is the difference between sitting ESX on workstation and sitting ESX on top of ESX ?

Texiwill · ‎11-24-2008

Hello,

Quite a bit actually. ESX as a host does not allow as much of the physical system to get to the VMs as Workstation does for one. THe other is workstation interprets and users INtel-VT/AMD-V quite a bit differently.

The hypervisors are very different and meet different goals. While both fruit it is like comparing an apple to an orange.

If you would like a much more detailed answer you should open a Support Request with your VMware Support Representative or even contact your VMware Sales Representative.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

RParker · ‎11-24-2008

Has anyone managed to run ESX on top of ESX ?

The reason has already been explained. It's not possible, nor is it advisable to run a VM within a VM. It's recursive logic. ESX is a bare metal install. VM Workstation is not. VM Workstation is a product that runs as a service under Windows (and it doesn't have direct access to the hardware either). It can vritualize an environment.

ESX is meant to be the ONLY OS on a hardware VM host, therefore installing ESX within ESX will not work.

What are you trying to do? Obviously you are running ESX, so why do you need a sub-instance of ESX? Can't you do everything you need using a Stand Alone Virtual host with Workstation or Server, to accomplish the same thing?

ESX is a unique product, and you shouldn't treat it like a simple Redhat or Linux install that will host VM's. ESX is very restrictive with regards to hardware and it's designed to be the top level hosting product.

So running ESX is designed to be hardware level, not a software install. Meaning you can't run it inside a Virtual Machine.

Texiwill · ‎11-24-2008

Hello,

So running ESX is designed to be hardware level, not a software install. Meaning you can't run it inside a Virtual Machine.

within ESX. but it is possible to run it within a VM within Workstation.Workstation was specifically enhanced to allow this capability. ESX does not contain the same enhancements.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

TomHowarth · ‎11-24-2008

It does not work because the VMware 3x development team specifically removed the code that would allow it to work, if you want to run nested guests, then you can utilised Workstations 6.5 and run ESX 3.5 as a guest and then run guests under the 3.5 instance, but it is slow and of no uses other than training and demo purposes

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410

RParker · ‎11-24-2008

Yes, but only because Workstation in itself is NOT an OS, it's a service host. Since it doesn't have bare metal installs, you can do more with Software than you can with an OS. Same reason you can't run Hyper Visor inside a Hyper Visor.. It's specific of the HCL..

Workstation is a rudimentary hosting program, but it's not really a hosting OS, it just allows some layer of virtualization, so I am not surprised it allows more latitude for other OS to run (even when they aren't supported)

I don't think ESX is supported when it's NOT bare metal, right? So just because it works, doesn't mean that's a real solution.

admin · ‎11-24-2008

Actually, you can install ESX 3.5 on top of ESX 3.5, but only if you have an AMD64 processor with AMD-V (aka SVM) and RVI. Your ESX guest should have the following configuration options:

monitor.virtual_mmu = "hardware"
monitor_control.restrict_backdoor = "TRUE"

You will also have to use the following command to enable the ESX VM's NICs to operate in promiscuous mode:

vsish -e set /net/portCfgs/portgroup[yournumber]/l2secPolicy/options 0

Finally, any nested VMs run under the ESX VM are limited to 32 bits.

Message was edited by: jmattson. Corrected the configuration options.

RParker · ‎11-24-2008

you can install ESX 3.5 on top of ESX 3.5, but only if you have an AMD64 processor with AMD-V (aka SVM)

So are you saying this is a supported configuration?

admin · ‎11-24-2008

So are you saying this is a supported configuration?

Sorry; I didn't mean to imply that this is a supported configuration. Running ESX in a VM is not supported, regardless of which product you are using to run the VM.

AnnTelope · ‎11-26-2008

You are the first to indicate that it is indeed possible. The point where I am failing seems to be down to the hyperthreaded core sharing. I'll try again with this disabled.

AnnTelope · ‎11-26-2008

or maybe

monitor_control.vt32 = TRUE

TomHowarth · ‎11-26-2008

Ann

he has indeed indecated that it is posible, he also stated that it is only on Hosts that are utilising AMD processors. I have never used AMD for ESX so that is why I was unaware of it working under AMD. I am pretty certain that as AMD chipsets do not used VT32 then the line you have mentioned will do absolutely nothing at all.

I again repeat for clarity IT ONLY WORKS WITH AMD CHIPS, is you have a INTEL machine just give up

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410

All

Install ESX on top of ESX - can it be done?