orangetangca
Contributor
Contributor

Home Lab - Nested VM Requirements?

Looking to put together a home lab to be able to do testing of VMware as well as other solutions. I'm struggling to find something that will fit my needs without blowing the budget - I know many of you have probably spent countless hours putting together your home labs so I thought I'd start here for advice. I'd done a fair amount of reading but still haven't really found an ideal solution. I'm back and forth between older used hardware, Intel NUCs and other solutions.

What do I need to pay special attention to when trying to find hardware to meet these requirements?

1. Nested VMware ESXi - I'd like the hardware to run nested ESXi instances, that can also run VMs. Performance of those nested VMs isn't a huge deal, but need to be able to do configurations, test VMotion etc. This doesn't have to be supported by VMware, but it needs to be able to work. Most other VMs (VCSA, and other solutions would just run within the physical ESXi host hypervisor)

2. Would like to run ESXi 6.7 - No point buying something that can't run the latest and greatest. This doesn't have to run 6.7 at the physical level, but the nested ESXi Hosts need to support it.

3. I have no need to test vSAN at this time.

4. Price is a factor, but not the only factor.

Anyone that has been down this path able to offer some advice?

6 Replies
kenbshinn
Enthusiast
Enthusiast

I am in the same boat as you right now. I was going to go down the Intel NUC route or even Super Micro, however the price really starts to add up after a while.

I just took a VMware Online Design Class and the Instructor showed us his "Lab" which ended up being a Single PC with a Ton of Memory and Storage. He used his VMUG Advantage Subscription to Install VMWare Workstation Player on this desktop and then installed Nested ESXi on there with a VCSA Appliance running as a VM. For storage he was running FreeNAS and carving off storage from one of his several 4 TB drives.

He was able to run at least 2 VCSA appliance and 9 ESXi Hosts (running WIndows Server and WIndows 7 VMs).

I think this will be the route I go, just to dip my toes in.

0 Kudos
orangetangca
Contributor
Contributor

Interesting point. I thought about doing something similar, since today I run my test hosts on my laptop using VMware Workstation Player. Any idea what compatibilities I'd need to take into account (ie CPU) to be able to support nested VMs running in a virtual ESXi instance through Player?

Did he happen to provide you with a parts list for his environment by chance?

0 Kudos
sjesse
Leadership
Leadership

Keep an eye on Ryzen processors if you have one, they have trouble with nested virtualization, there a few posts out there. I have a quad core laptop with 32 gb of memory that run the applicans and about 13 other servers using vmware workstation. With quad core your likely to run into problems with more then 8 or 9 vms, I'd make sure you try to keep 6:1 ratio of virtual cpus to the number of cores you have, running the bare minimum I can for a horizon view environment I'm hitting 100% often. To test networking and such you can get a virtual router like sophos utm which will do vlan tagging. Freenas has worked well for me for testing iscsi, but windows iscsi role works as well.

It really depends on your plans for testing. If you want to test things like horizon and nsx, make sure if your using workstation to have a 6 core proccessor I thin, or 2 nucs with plenty of memory. Another thing you can look at is Ravello | Oracle Cloud, you can start your lab on demand and then pay only storage costs when its not on.

0 Kudos
mc1903cae
Enthusiast
Enthusiast

I run a single HP Z800 workstation which has 2 x Intel Xeon X5650 (6 Cores each), 96GB Memory, 2 x 2TB SATA (RAID1), 1 x 500GB SSD, 1 x Fusion IO 320GB (Gen1) and 1 x Fusion Io 640GB (Gen1) PCI Flash Cards. The networking is 6 x 1Gbps NICs (2 x 1Gbps Onboard + 1 x Quad Port 1Gbps PCIe slot) with a Cisco 3750; which has IP routing for 6-10 VLANs enabled.

I boot ESXi 6.5 from the SATA RAID 1 and the rest of it is a VMFS ISO/Template Datastore. I run my test VMs on the SSD & Fusion IO's.

Overall performance is not blistering fast; but it is very acceptable.

The biggest issue I face is memory, as I regularly max out 96GB; whereas the CPU rarely goes over 60%!

I run a Windows 2012 R2 DC/DHCP/DNS VM, a VCSA 6.5 which manages the physical (allows snaps/clones/templates) and I also use a Win 10 VM as a jumpbox as  I RDC in from my PC.

I can run upto 4 x VCSA (Tiny/Thin) with 4 x Ext PSC and a couple of ESXi 6.7 VM's; or a Single VCSA/Emb PSC with about 6 ESXi vSAN nodes (2 vCPU/16Gb). vSphere ESXi 6.7 VM's can be installed but there is a 'nag' warning of future deprecation with the Intel Xeon X5650 CPUs.

I am looking to upgrade to a HP Z820 or Z840 later this year; my minimum spec will be 256GB Memory and 2 x 6/8 core Intel E5-2600 (v2 or v3) CPU's. I am looking in the region of £1500 for a Z820 or £4500 for a Z840. Trying to find one on eBay that doesn't have a crazy high spec graphics card is difficult;

You need the following CPU features for nested virtualisation to be made available in the BIOS. Also, just because the CPU has the features it doesn't automatically mean the ability to switch them on is exposed in the BIOS. Found this out when I was researching the Z800!

  • Intel VT-x or AMD-V is required for running nested 32-bit VMs.
  • Intel EPT or AMD RVI is required for running nested 64-bit VMs.

If you work for a VMware partner, you can get access to the ~90 day Partner Evaluation (test/PoC/non-production) licenses for most vSphere products (and Workstation Player/Pro) on Partner Central. Also, get your My VMware account linked to the whichever company My VMware account has been designated with Partner download rights; so you can download the vSphere install code & patches.

Good luck and enjoy building your lab.

M

0 Kudos
sjesse
Leadership
Leadership

If your not a vmware partner, you can get yearly licenses from vmug advantage, its 200 a year. That includes vmware workstation as well

kenbshinn
Enthusiast
Enthusiast

I think he said he was running an Intel i7 (not sure the model), 64 GB of RAM, and 3x 4 TB HDD and 1 TB SSD Storage. He said it was a custom build so I am sure he either had it build or built it off of Newegg.com

0 Kudos