VMware Cloud Community
scotte
Enthusiast
Enthusiast
‎01-22-2013 05:54 AM

Nested ESXi 5.1 with Intel Nehalem L5520 CPU's

Hi,

I am having difficulty setting up the nesting of ESXi 5.1, hope you can help please.

I have a physical DL350 G6 with Intel L5520 CPU's, which support both VT-x & EPT.

I have followed the guide in this blog post

http://www.virtuallyghetto.com/2012/09/having-difficulties-enabling-nested.html

I have enabled vhv on the physical host.

I have enabled hardware virtulisation on each nested host.

I have also followed the procedure for enabling EVC on the cluster for Nehalem CPU's and then copying out the CPUID settings, but once I do this I cannot power on the ESXi nested hosts, I get this error "The processor does not support a required feature for this virtual machine. The virtual machine cannot be powered on."

This is the Intel CPUID settings, that I copied from the cluster and entered into each of the nested ESXi hosts

Nehalem_CPUID.jpg

Without amending the CPUID settings I have had a 64-Bit VM running but I cannot vMotion this between the two nested ESXi 5.1 hosts.

Any help would be appreciated.

Thanks

Scott

0 Kudos
10 Replies
admin
Immortal
Immortal
‎01-22-2013 07:37 AM

scotte wrote:


I have also followed the procedure for enabling EVC on the cluster for Nehalem CPU's and then copying out the CPUID settings, but once I do this I cannot power on the ESXi nested hosts, I get this error "The processor does not support a required feature for this virtual machine. The virtual machine cannot be powered on."

What procedure did you follow?  Can you post your .vmx file?

0 Kudos
scotte
Enthusiast
Enthusiast
‎01-22-2013 07:53 AM

Hi,

I basically copied the fields from the EVC settings in the image to the identical fields on each hosts under CPUID Mask > Advanced

This is my .vmx config of each esxi 5.1 host

ESXi1

.encoding = "UTF-8"

config.version = "8"

virtualHW.version = "9"

pciBridge0.present = "true"

pciBridge4.present = "true"

pciBridge4.virtualDev = "pcieRootPort"

pciBridge4.functions = "8"

pciBridge5.present = "true"

pciBridge5.virtualDev = "pcieRootPort"

pciBridge5.functions = "8"

pciBridge6.present = "true"

pciBridge6.virtualDev = "pcieRootPort"

pciBridge6.functions = "8"

pciBridge7.present = "true"

pciBridge7.virtualDev = "pcieRootPort"

pciBridge7.functions = "8"

vmci0.present = "true"

hpet0.present = "true"

nvram = "ESXi5-1_1.nvram"

virtualHW.productCompatibility = "hosted"

powerType.powerOff = "soft"

powerType.powerOn = "hard"

powerType.suspend = "hard"

powerType.reset = "soft"

displayName = "ESXi5-1"

extendedConfigFile = "ESXi5-1_1.vmxf"

floppy0.present = "true"

scsi0.present = "true"

scsi0.sharedBus = "none"

scsi0.virtualDev = "lsilogic"

memsize = "8192"

scsi0:0.present = "true"

scsi0:0.fileName = "ESXi5-1_1.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

ide1:0.present = "true"

ide1:0.clientDevice = "true"

ide1:0.deviceType = "cdrom-raw"

ide1:0.startConnected = "false"

floppy0.startConnected = "false"

floppy0.fileName = ""

floppy0.clientDevice = "true"

ethernet0.present = "true"

ethernet0.virtualDev = "e1000"

ethernet0.networkName = "VM Network - 192.168.3.x"

ethernet0.addressType = "vpx"

ethernet1.present = "true"

ethernet1.virtualDev = "e1000"

ethernet1.networkName = "iSCSI VM Network - 192.168.2.x"

ethernet1.addressType = "vpx"

ethernet2.present = "true"

ethernet2.virtualDev = "e1000"

ethernet2.networkName = "VMotion Network - 172.16.1.x"

ethernet2.addressType = "vpx"

ethernet3.present = "true"

ethernet3.virtualDev = "e1000"

ethernet3.networkName = "VM Network - 192.168.0.x"

ethernet3.addressType = "vpx"

guestOS = "vmkernel5"

uuid.location = "56 4d 22 4f e9 c6 4c a8-be da c0 c2 2a 97 05 fa"

uuid.bios = "56 4d 22 4f e9 c6 4c a8-be da c0 c2 2a 97 05 fa"

vc.uuid = "52 2d b4 25 89 64 7a 2d-26 c2 63 c4 c4 6f 3f ef"

ethernet0.generatedAddress = "00:50:56:b9:65:1d"

ethernet1.generatedAddress = "00:50:56:b9:65:1e"

ethernet2.generatedAddress = "00:50:56:b9:65:1f"

ethernet3.generatedAddress = "00:50:56:b9:65:20"

numvcpus = "8"

cpuid.coresPerSocket = "4"

monitor_control.restrict_backdoor = "True"

vmci0.id = "1667260645"

cleanShutdown = "FALSE"

replay.supported = "FALSE"

sched.swap.derivedName = "/vmfs/volumes/37e990a4-5ec929a3/ESXi5-1_1/ESXi5-1_1-a7fb9aac.vswp"

replay.filename = ""

scsi0:0.redo = ""

pciBridge0.pciSlotNumber = "17"

pciBridge4.pciSlotNumber = "21"

pciBridge5.pciSlotNumber = "22"

pciBridge6.pciSlotNumber = "23"

pciBridge7.pciSlotNumber = "24"

scsi0.pciSlotNumber = "16"

ethernet0.pciSlotNumber = "32"

ethernet1.pciSlotNumber = "33"

ethernet2.pciSlotNumber = "34"

ethernet3.pciSlotNumber = "35"

vmci0.pciSlotNumber = "36"

hostCPUID.0 = "0000000b756e65476c65746e49656e69"

hostCPUID.1 = "000106a500100800009ce3bdbfebfbff"

hostCPUID.80000001 = "00000000000000000000000128100800"

guestCPUID.0 = "0000000b756e65476c65746e49656e69"

guestCPUID.1 = "000106a400040800809822091fabfbff"

guestCPUID.80000001 = "00000000000000000000000128100800"

userCPUID.0 = "0000000b756e65476c65746e49656e69"

userCPUID.1 = "000106a500100800009822091fabfbff"

userCPUID.80000001 = "00000000000000000000000128100800"

evcCompatibilityMode = "TRUE"

vmotion.checkpointFBSize = "4194304"

tools.syncTime = "FALSE"

sched.cpu.min = "0"

sched.cpu.units = "mhz"

sched.cpu.shares = "normal"

sched.mem.min = "0"

sched.mem.shares = "normal"

softPowerOff = "FALSE"

monitor.virtual_mmu = "hardware"

monitor.virtual_exec = "hardware"

cpuid.0.ecx = ""

cpuid.0.edx = ""

cpuid.1.eax = ""

cpuid.1.ecx = ""

cpuid.1.edx = ""

cpuid.80000001.eax = ""

cpuid.80000001.eax.amd = ""

cpuid.80000001.ebx = ""

cpuid.80000001.ebx.amd = ""

cpuid.80000001.ecx = ""

cpuid.80000001.ecx.amd = ""

cpuid.80000001.edx = ""

cpuid.80000001.edx.amd = ""

ESXi2

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "9"
pciBridge0.present = "true"
pciBridge4.present = "true"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "true"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "true"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "true"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "true"
hpet0.present = "true"
nvram = "ESXi5-2.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
displayName = "ESXi5-2"
extendedConfigFile = "ESXi5-2.vmxf"
floppy0.present = "true"
scsi0.present = "true"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsilogic"
memsize = "8192"
scsi0:0.present = "true"
scsi0:0.fileName = "ESXi5-2.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "true"
ide1:0.clientDevice = "true"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "false"
floppy0.startConnected = "false"
floppy0.fileName = ""
floppy0.clientDevice = "true"
ethernet0.present = "true"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "VM Network - 192.168.3.x"
ethernet0.addressType = "vpx"
ethernet1.present = "true"
ethernet1.virtualDev = "e1000"
ethernet1.networkName = "iSCSI VM Network - 192.168.2.x"
ethernet1.addressType = "vpx"
ethernet2.present = "true"
ethernet2.virtualDev = "e1000"
ethernet2.networkName = "VMotion Network - 172.16.1.x"
ethernet2.addressType = "vpx"
ethernet3.present = "true"
ethernet3.virtualDev = "e1000"
ethernet3.networkName = "VM Network - 192.168.0.x"
ethernet3.addressType = "vpx"
guestOS = "vmkernel5"
uuid.location = "56 4d 65 c3 4a 22 c2 75-a9 30 81 89 d1 52 a0 43"
uuid.bios = "56 4d 65 c3 4a 22 c2 75-a9 30 81 89 d1 52 a0 43"
vc.uuid = "52 61 28 4f 51 83 05 d2-7d 0b a2 be 44 0a 78 ca"
ethernet0.generatedAddress = "00:50:56:b9:65:21"
ethernet1.generatedAddress = "00:50:56:b9:65:22"
ethernet2.generatedAddress = "00:50:56:b9:65:23"
ethernet3.generatedAddress = "00:50:56:b9:65:24"
numvcpus = "8"
cpuid.coresPerSocket = "4"
monitor_control.restrict_backdoor = "True"
vmci0.id = "957882373"
cleanShutdown = "FALSE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/37e990a4-5ec929a3/ESXi5-2/ESXi5-2-fa0acf3d.vswp"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
ethernet1.pciSlotNumber = "33"
ethernet2.pciSlotNumber = "34"
ethernet3.pciSlotNumber = "35"
vmci0.pciSlotNumber = "36"
hostCPUID.0 = "0000000b756e65476c65746e49656e69"
hostCPUID.1 = "000106a500100800009ce3bdbfebfbff"
hostCPUID.80000001 = "00000000000000000000000128100800"
guestCPUID.0 = "0000000b756e65476c65746e49656e69"
guestCPUID.1 = "000106a400040800809822091fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000b756e65476c65746e49656e69"
userCPUID.1 = "000106a500100800009822091fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "TRUE"
vmotion.checkpointFBSize = "4194304"
tools.syncTime = "FALSE"
sched.cpu.min = "0"
sched.cpu.units = "mhz"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.shares = "normal"
softPowerOff = "FALSE"
monitor.virtual_mmu = "hardware"
monitor.virtual_exec = "hardware"
featMask.vm.hv.capable = "Min:1"
vhv.enable = "TRUE"
cpuid.0.ecx = ""
cpuid.0.edx = ""
cpuid.1.eax = ""
cpuid.1.ecx = ""
cpuid.1.edx = ""
cpuid.80000001.eax = ""
cpuid.80000001.eax.amd = ""
cpuid.80000001.ebx = ""
cpuid.80000001.ebx.amd = ""
cpuid.80000001.ecx = ""
cpuid.80000001.ecx.amd = ""
cpuid.80000001.edx = ""
cpuid.80000001.edx.amd = ""
0 Kudos
admin
Immortal
Immortal
‎01-22-2013 08:04 AM

Something went wrong, because your CPUID masks are all empty.

Can we take a step back?  What are you trying to accomplish by this?

0 Kudos
scotte
Enthusiast
Enthusiast
‎01-22-2013 09:18 AM

Hi,

Sorry forgot to mention I reset them to default so that I could power them back on.

The reason I want to do this is to test features such as FT, I only have one physical ESXi as this is a lab environment.

Thanks

Scott

0 Kudos
admin
Immortal
Immortal
‎01-22-2013 10:34 AM

Under 5.1, I would expect BT-based FT to work without mucking with CPUID masks (I think).  HV-based FT isn't going to work, because we don't virtualize enough of the PMU.

0 Kudos
scotte
Enthusiast
Enthusiast
‎01-22-2013 12:01 PM

According to this blog you can do it

http://www.virtuallyghetto.com/2011/07/how-to-enable-nested-vft-virtual-fault.html

0 Kudos
admin
Immortal
Immortal
‎01-22-2013 12:15 PM

Yes, you can do BT-based FT, as I said above.

0 Kudos
scotte
Enthusiast
Enthusiast
‎01-22-2013 01:10 PM

What is  BT-based FT &  HV-based FT?

What's the difference?

Thanks

Scott

0 Kudos
admin
Immortal
Immortal
‎01-22-2013 01:20 PM

BT = binary translation

HV = hardware-assisted virtualization

With BT-based FT, every guest instruction is emulated via binary translation.  This is not practical and unsupported, but it is available.  This is the mode described in the blog.

With HV-based FT, hardware-assisted virtualization is used to execute the guest instructions.  This is the mode used in production, but it isn't available for nested VMs due to hardware CPU features that are not virtualized.

0 Kudos
scotte
Enthusiast
Enthusiast
‎01-23-2013 01:46 AM

Thanks for explanation.

All the above is supposed to help get 64-Bit VM's running on the nested ESXi hosts, I am even having a few problems running 32-Bit VM's.

When I inistitally power on a VM on ESXi-1 the VM & the host momentarily become disconncted in the VI client, they eventually return.

I can then vMotion from ESXi-1 to ESXi-2 successfully but when I try the other way I get this error 'An error occurred whilst communicating with the remote host' ESXi-1 then states (Not responding) in the VI client.

Any idea what's happening here?

Thanks

0 Kudos