VMware Communities
Al_Koch
Enthusiast
Enthusiast

Confusion about Threadripper, Workstation Pro and Windows 10

Hello,

My workload has grown to the point where I need to simultaneously run 7 - 8 VMs. Some run Windows 7 & 10 and others run Ubuntu. Each VM runs one or two of: MS Office, a browser, database server, Java Server, Eclipse, VS Studio, VS Code and video editing & rendering. My current system (i7-4770 Quad Core, 32GB)  is maxed out at 4 cores. Ideally, I'd like to upgrade to a system running the Threadripper 3950X with 16 cores with Workstation running on Win 10. I have several questions:

1) Will Workstation Pro on Win 10 even run on the Threadripper 3950X or higher models? I am confused because on the one hand https://www.vmware.com/resources/compatibility/search.php?deviceCategory=cpu doesn't show ANY Threadripper CPUs suggesting there is no support but https://communities.vmware.com/thread/622377?start=30&tstart=0 has a long thread about the 3960X running but taking a long time to boot.

2) Does this CPU seem appropriate for this workload? I'm thinking 2 cores for each VM with possibly 4 cores for the VM running the video software. I realize that this # of cores exceeds 16 (plus Windows needs some as well!) so this would require some swapping. It's a "worst" case to need all 8 VMs - I probably would normally run with 6.

3) Is it possible to make a recommendation for a minimum amount of RAM for these 16 cores?

Thank you.

32 Replies
bluefirestorm
Champion
Champion

Quite frankly, a lot of positive I read about AMD chips is mostly its cost. And a lot of Intel bashing. There is not a lot of objective proof of technical superiority other than with synthetic benchmarks which I tend to ignore as they are synthetic.

What current AMD chips don't have

Process Context Identifier (PCID) in the TLB cache to distinguish entries between processes (available since Westmere)

INVPCID instruction (available since Haswell) - an instruction to invalidate cache entries specific to a PCID, this is to avoid clearing the entire cache

These two features are vital for Meltdown patch and reducing the negative impact on performance the patch brings. I think Zen3 will include PCID and INVPCID.

For virtualisation, starting from Haswell generation, there are features that make for more efficient running of VMs. So there are less VM transitions (VM-EXIT handover to hypervisor and then back with VM-ENTRY) compared to earlier generations. There are also features that make running nested VMs more efficient (VMCS shadowing). I don't know the AMD equivalent.

Another thing is neither Intel nor AMD seem to publicly disclose the clock cycle costs for VM transitions. I have only seen a PowerPoint slide deck from Intel that claims Intel Haswell VM transition costs around 500 clock cycles down from around 2000 clock cycles with Westmere chips.

Going to the specific thread that you refer to, one thing I found strange in the vmware.log attached in the original post is that it shows VMware Workstation (and perhaps also Windows OS) sees two NUMA nodes but the other node seem to have zero CPUs/cores. For a comparison of active virtualisation features, I also pasted here comparison between the Threadripper 3960X 24-core and an Intel i9-9880H and you'd see there is a longer list of "Active HV Capabilities" for the Intel chip although some are specific to Intel.

2019-11-29T08:48:48.437+01:00| vmx| I125: Log for VMware Workstation pid=15916 version=15.5.0 build=build-14665864 option=Release

2019-11-29T08:48:48.543+01:00| vmx| I125: hostCPUID name: AMD Ryzen Threadripper 3960X 24-Core Processor

2019-11-29T08:48:48.594+01:00| vmx| I125: HostWin32: WIN32 NUMA node 0, CPU mask 0x0000ffffffffffff

2019-11-29T08:48:48.594+01:00| vmx| I125: HostWin32: WIN32 NUMA node 1, CPU mask 0x0000000000000000

2019-11-29T08:55:28.096+01:00| vmx| I125: Active HV capabilities

2019-11-29T08:55:28.096+01:00| vmx| I125:    Address space identifiers

2019-11-29T08:55:28.096+01:00| vmx| I125:    Flush by ASID

2019-11-29T08:55:28.096+01:00| vmx| I125:    Full decode

2019-11-29T08:55:28.096+01:00| vmx| I125:    MSR permission bitmap

2019-11-29T08:55:28.096+01:00| vmx| I125:    Nested paging A/D bits

2019-11-29T08:55:28.096+01:00| vmx| I125:    Next RIP

2019-11-29T08:55:28.096+01:00| vmx| I125:    Guest physical width

2019-11-29T08:55:28.096+01:00| vmx| I125:    Real-address mode

2019-11-29T08:55:28.096+01:00| vmx| I125:    TSC scaling

2019-11-29T08:55:28.096+01:00| vmx| I125:    Mode-based execute control for nested paging

2019-11-29T08:55:28.096+01:00| vmx| I125:    Virtualized VMSAVE/VMLOAD

2020-10-24T09:38:24.044-04:00| vmx| I005: Log for VMware Workstation pid=3744 version=15.5.6 build=build-16341506 option=Release

2020-10-24T09:38:24.214-04:00| vmx| I005: hostCPUID name: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

2020-10-24T09:38:24.373-04:00| vmx| I005:   Use VMCS shadowing                       {0,1}

2020-10-24T09:38:26.497-04:00| vmx| I005: Active HV capabilities

2020-10-24T09:38:26.497-04:00| vmx| I005:    Address space identifiers

2020-10-24T09:38:26.497-04:00| vmx| I005:    Flush by ASID

2020-10-24T09:38:26.497-04:00| vmx| I005:    Full decode

2020-10-24T09:38:26.497-04:00| vmx| I005:    MSR permission bitmap

2020-10-24T09:38:26.497-04:00| vmx| I005:    Nested paging A/D bits

2020-10-24T09:38:26.497-04:00| vmx| I005:    Next RIP

2020-10-24T09:38:26.497-04:00| vmx| I005:    Real-address mode

2020-10-24T09:38:26.497-04:00| vmx| I005:    Skip debug state

2020-10-24T09:38:26.497-04:00| vmx| I005:    X2APIC virtualization

2020-10-24T09:38:26.497-04:00| vmx| I005:    BNDCFGS/MPX

2020-10-24T09:38:26.497-04:00| vmx| I005:    INVEPT single-context

2020-10-24T09:38:26.497-04:00| vmx| I005:    Page-modification logging

2020-10-24T09:38:26.497-04:00| vmx| I005:    ENCLS exiting

2020-10-24T09:38:26.497-04:00| vmx| I005:    PAUSE-loop exiting

2020-10-24T09:38:26.497-04:00| vmx| I005:    Advanced exit information for EPT violations

2020-10-24T09:38:26.497-04:00| vmx| I005:    Mode-based execute control for nested paging

2020-10-24T09:38:26.497-04:00| vmx| I005:    EPT-violation virtualization exception

2020-10-24T09:38:26.497-04:00| vmx| I005:    Event injection with instruction length zero

Maybe I'm biased, but If I were in your position, I'd stick to Intel. The VMCS Shadowing feature seem to be indicated by the "vPro Eligibility" indicated as Yes in the Intel ARK website. The Skylake-X/Cascade Lake-X series doesn't have vPro and I have seen a vmware.log of one Skylake-X chip that confirms it doesn't have VMCS Shadowing.

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm.  Thanks very much for the detailed comparison. I note that you used the i9-9880 in the comparison which only has 8 cores (Intel® Core™ i9-9880H Processor (16M Cache, up to 4.80 GHz) Product Specifications ). My expected workload is closer to 2 cores x 8 VMs = 16 cores. What would you suggest if 16 cores (8 VMs) are needed?

Also, I am still confused by https://www.vmware.com/resources/compatibility/search.php?deviceCategory=cpu. It doesn't show the i9 series at all. Can you explain what this page is showing?

Thanks again!

Reply
0 Kudos
bluefirestorm
Champion
Champion

I haven't come across a 10th gen Intel desktop CPU vmware.log yet in this forum. Otherwise I'd used that. I used the i9-9880H (yes, it is a laptop chip) as that is the closest to its desktop equivalent i9-9900K and it has VMCS shadowing.

The compatibility guide link you refer to is for ESXi which is moving towards to a server chips only direction.

Even if all VMs are running simultaneously, are some VMs more busy/more idle than others? So you may not need that many cores in the host CPU. But you should know the load profile of the VMs.

And again it goes back to the virtualisation features, if the VM(s) is(are) making more VM-EXITS, it doesn't matter if the host CPU has 24, 16 cores or 8 cores, that just means the VM is not executing and more busy switching back and forth between VM and hypervisor.

Windows 7 OS itself cannot take advantage of PCID/INVPCID (its OS kernel wasn't written to use PCID in the first place) so a Windows 7 VM will already be inherently inefficient with the Meltdown patch. And perhaps a way to counter this is to disable the Meltdown patch in the Windows 7 registry.

From a core count view, the current generation of Intel chips can't compete especially in terms of cost. But the Threadripper problem thread to me is already a red flag to avoid buying it for virtualisation use. And I wouldn't advise buying an expensive Cascade Lake-X either just to satisfy the core count (that is if you can find them in stock to buy). With the way things are moving, nested virtualisation will probably become more of a requirement (e.g. running a WSL2 inside a Windows 10VM, running docker in a VM, etc); hence the VMCS shadowing feature will likely become more important. Otherwise the highest end desktop chip now from Intel i9-10900K (10 core/20 threads) and it has vPro Eligibility as "Yes". It seems so flimsy that a feature is down to marketing jargon of "vPro Eligibility" but so far that has born out, those with "Yes" has VMCS shadowing indicated in the vmware.log and those that are "No" does not.

https://ark.intel.com/content/www/us/en/ark/products/199332/intel-core-i9-10900k-processor-20m-cache...

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm.

Again, thanks for your thoughtful analysis. I am learning a great deal about current processor and virtualization tech from your responses!

- In response to your question about VM "activity", this workstation is for software development and the usual usage is to work in, say a "Java Dev VM" for a while, push the changes to a "Java server VM" which restarts with new changes, switch to a "Frontend/Angular VM" to mod some code and then run a browser to test all of this. Then I may have to jump to a "C++ Dev VM" for similar activities with occasional jumps to a VM to check email or to browser based web lookups. Meanwhile a "database server VM" has been running and responds as needed. So, there is a material amount of "switching VMs" and not all VMs would be "running hard" all the time and I suspect I could get by with less than 16 cores but 1) of course, a couple of cores are consumed by the Win 10 host OS and 2) I want to have some room for unplanned growth. I'm very curious to hear your observations.

- So far, I am finding it hard to make a selection. On the one hand I understand your arguments for Intel related to the PCID and INVPCID and additional "Active HV Capabilities" commands and the anticipated nested virtualization. However, after a fair amount of research on the 109xx processors I am really concerned about 1) the massive heat that these chips generate with the attendant cooling issues/risk and 2) investing in a new (and expensive) system that is built on the old 14nm technology. I don't currently need nested virtualization but of course, it would be good to have that since there will be unexpected future needs and the thread I posted about Threadrunner issues does make me a bit nervous, but the heat issue and investing a large amount of money in a new system based on tech that is clearly "on the way out" makes me really nervous.

- AMD is releasing Zen3 (which you suggested would have PCID and INVPCID) next week so it makes sense for me to keep "limping along" at least until some testing and reviews become available for those processors. In the meantime I would really appreciate your thoughts.

Thanks!

Reply
0 Kudos
bluefirestorm
Champion
Champion

I doubt VM CPU usage will generate much heat unless your development tasks involve doing simulated load/stress test of a server application.

The 11th Gen Rocket Lake desktop CPUs should also be coming soon. It is another 14nm tech but it is suppose to come with AVX-512, deep learning instructions and PCIe 4.0 support. At least there are more future-forward looking features than a 10th Gen Comet Lake and I hope Intel doesn't fudge around with the VMCS shadowing with Rocket Lake.

Yes, you could chug along with the current i7-4770 for a while longer, maybe for another 6 to 9 months.

On the other hand, unless you have promised letting somebody else to have the i7-4770 desktop or constraint such as physical space, nothing stops you from spreading the VM load across two machines (implies the VMs have to be on bridged networking). Perhaps the i7-4770 desktop can run with the database server VM and other VMs that have infrequent direct interaction and have the more modern system with the more interactive VMs. It may not be the most ideal setup as it does not have all VMs in one physical unit. Or you could keep all VMs in the new unit but keep the i7-4770 to alleviate any concern for the core count not being enough or serve as a fallback in situations where the VMs are expected to be under heavy load such as during a load/stress test.

I have never used a C++ compiler on an AMD CPU. I don't know what specific issues a C++ developer would have to be aware of if compiling code on an AMD CPU while the target is the much larger Intel CPU market. For one thing, some libraries will be optimised for Intel (example: libraries that use AVX512 obviously would not work in AMD CPUs).

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm,

Well, AMDs Zen3 was released last week and none of the reviews I've found so far raise any issues. However, none of them are running VMware in their tests so I'm not sure what to make of all that.

Thanks for your thoughtful response. I've been posting the question for "me" but there is actually a team of 7 that need to upgrade.

Based on your last response I researched Rocket Lake but what I found is really discouraging. I was hoping Intel might address my concerns with this but it doesn't look that way. The problems are:
- Projected for Q1 2021 but there's always a risk of delay.
- Will only offer a max of 8 cores so we're at risk of eventually overwhelming the system again.
- This doesn't change the current schedule that delays Intel 7nm until late 2022 or early 2023 and investing a significant amount of money in "legacy" (fair to start calling "obsolete"?) tech is unappealing.

If you recall the User Case I described above, the man problem is intermittent "5 min" host lockups (I assume either VMware or Win 10 gets "overwhelmed" and stops for "housekeeping"), plus intermittent VM specific hangs/slowdowns.

Bottom line, it would be nice to have "future-proofing" such as nested virtualization and AVX-512/deep learning but neither is on the must have list. What we need is faster, more reliable "dev workstations". The downside with Intel is that if we look only at that requirement, Intel's offering (now and maybe in Q1 next year) seem deficient due to much lower core counts and a legacy 14nm tech. All of that argues for any of several AMD processors but my one concern is potential VMware incompatibility with these AMD chips.

1) Can you point me to a VMware compatibility page for Desktop/Workstation processors (you explained that https://www.vmware.com/resources/compatibility/search.php?deviceCategory=cpu is for ESXi /server chips).
2) You had said "the Threadripper problem thread to me is already a red flag to avoid buying it for virtualisation use". Can you elaborate on the red flag? Of course, slow boot times (with a large # of cores/VM) is bad but no one mentions that once a VM is booted that VMware has any problems. We need to run, worst case, 8 VMs where most VMs have probably two cores, no VM is "running hard" and we need to be jumping around a lot between VMs where, in each VM we spend time on part of a development lifecycle. Your thoughts about AMD in this scenario?
3) One thing I see sometimes is someone having trouble "porting" a VM originally built on an Intel processor to an AMD process where it fails to boot or run. Your thoughts?

There are issues with both Intel and AMD but I'd really value your thoughts on the risks of running VMware on one of these newer AMD processors.

As always thanks!

Reply
0 Kudos
bluefirestorm
Champion
Champion

I haven't seen a review that has confirmed PCID/INVPCID in the latest Zen 3 CPUs.

I am not aware of VMware HCL CPUs for its desktop virtualisation products. I suppose for ESXi it is more strict/precise because of the support that has to be available to large enterprise customers and its partners (like IBM, HPE, etc) and its intended market for the datacentre/cloud/enterprise use.

The red flag is the unknown cause(s) of slowness of the VM boot. Is it the lack of PCID/INVPCID? Is it because it is seen as 2 NUMA nodes but yet one node has no logical CPUs? Why hasn't AMD worked with Microsoft fix this NUMA problem (if it is a problem). It is odd for a single socket CPU to be seen as two NUMA nodes and only one has logical CPUs. Notice that node 1 has CPU mask all 0s which means that there are no logical processors in that node.

2019-11-29T08:48:48.543+01:00| vmx| I125: hostCPUID name: AMD Ryzen Threadripper 3960X 24-Core Processor
2019-11-29T08:48:48.594+01:00| vmx| I125: HostWin32: WIN32 NUMA node 0, CPU mask 0x0000ffffffffffff
2019-11-29T08:48:48.594+01:00| vmx| I125: HostWin32: WIN32 NUMA node 1, CPU mask 0x0000000000000000

I haven't gone through the entire thread so I don't know what is the common VM use cases for the people who posted on that thread. Would the slowness of boot being disk I/O intensive also mean slowness in a development activity such as build/compile process which is equally if not more intensive in disk I/O?

As for porting a VM from Intel to AMD, I have no experience on this. But it would be more than just the OS inside that is the concern. There are less issues for concern for an ordinary VM that is used for web browsing/word processing/spreadsheet. But for development purposes there is added concern on tools and library compatibility/optimisation. For tools compatibility, as an example, Android development tools have CPU specific virtualisation (e.g. Intel HAXM). This also implies nested virtualisation if the Android development environment is inside a VM.

That is what I meant that nested virtualisation may become more commonplace. Even though you may not explicitly look to set up for nested virtualisation, a development tool/framework inside a VM will likely push/force you to do nested virtualisation.

Again, if I were in position running VMs that are primarily for software development purposes, I'd stick with Intel CPU for the host.

Since I can't recommend the Skylake-X/Cascade-X despite its high core count but lacks VMCS shadowing, perhaps it is time to look at Xeon W CPU.

https://ark.intel.com/content/www/us/en/ark/products/series/125035/intel-xeon-w-processor.html#@Work....

But obviously these would be even more expensive than the Skylake-X/Cascade Lake-X; just one example

https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=198019,198016

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm,

OK, I think you've convinced me that even though Intel is still shipping 14nm (while AMD is at 7nm) the stability of VMware on the Intel platform should be primary. In that case I'm confused about your recommendation/suggestion. Earlier in the thread you pointed me to the 10 Core Core i9-10900K since it had the max # of codes and also supported VPro/VMCS shadowing. But in your last post you are suggesting a Xeon W processor.

Would you expand your thoughts on these two?

Thanks very much.

 

Reply
0 Kudos
bluefirestorm
Champion
Champion

The Xeon W Cascade Lake has AVX-512 and Deep Learning boost while the i9-10900K does not. The i9-10900K core count tops out at 10 while the Xeon W has models that are far much higher (12, 14, 18, 24, 28).

The Xeon W Cascade Lake is an alternative if the higher core count and more future forward looking features is preferred. It is much more expensive than the regular desktop CPUs and requires ECC RAM.

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm,

Sorry for the delayed response! Thanks for your alternative of a Xeon. The AVX-512 would be a nice "future use" feature but isn't critical. It doesn't make sense to encure the additional cost of ECC RAM so I guess we've settled on the i9-10900K. I say "guess" because I now need to go and configure a custom build somewhere and until I get through that I can't be sure some issue blocking use of the i9-10900K won't come up. If you would like to recommend a custom build website that you've had a good experience with that would be good info to have.

Thanks!

 

 

Reply
0 Kudos
bluefirestorm
Champion
Champion

I don't have any recommendation for a custom builder of PCs for you as I don't think we are located in the same continent, let alone the same country. Although if it were for Xeon W, the path of least resistance is to go with either Dell Precision, HP Z or Lenovo Thinkstation but these 3 vendors use proprietary connectors for their power supply for the motherboard and CPU in their workstation line.

It isn't that hard to assemble a PC. The tricky thing is researching the components/parts and ensure they place nice with each other (e.g. CPU cooler height clearance, GPU length clearance against the case, power supply can deliver enough wattage, etc), ordering/buying the parts and waiting for all parts to arrive (if purchased online). But I suppose hiring a custom builder, you just let them know what you want and let them do the legwork.

ender_
Expert
Expert

FWIW, I'm just upgrading my own workstation - currently I have Xeon E5-1620 (v1), 32GB ECC RAM, and I ordered Ryzen 9 5900X with 64 GB ECC RAM as replacement (2x 32GB, which lets me upgrade to 128 GB should the need arise; I went with Asus Pro WS X570-ACE as the motherboard).

Al_Koch
Enthusiast
Enthusiast

Hi bluefirestorm,

Thanks for getting back to me and also the suggestions about a turnkey Xeon system. I used to assemble PCs and suspect I could build whatever system we finally settle on. However, you listed some of the time demanding initial research that's required, plus there are always some glitches that require additional research and troubleshooting and I am so overloaded with other work that it makes sense to outsource this work (as must as doing so may frustrate me!).

 

Reply
0 Kudos
Al_Koch
Enthusiast
Enthusiast

Hi ender,

Thanks very much for your input!

Are you concerned at all about how well ***VMware Desktop*** will run on the Ryzen chip? Here are three things that give me pause about the (otherwise very attractive Ryzen chip):
1) https://communities.vmware.com/thread/622377?start=30&tstart=0 discusses slow boot times in some cases.
2) I'm concerned about porting VMs built on an Intel chip to the Ryzen chip.
3) bluefirestorm has been very helpful and pointed out that the Ryzen chips don't support several Intel features such as those supporting nested virtualization (VMCS shadowing) which may be useful in a software development use case.

I'd really appreciate any additional inputs you may have related to the above including your VMware use case.

Thanks again.

 

Reply
0 Kudos
ender_
Expert
Expert

I can't comment on slow boot times until I receive the CPU, but will certainly look into that.

Porting VMs shouldn't be a problem, just make sure they're powered off (not suspended). I moved VMs from Athlon64 to i7 920 years ago when I switched.

Nested virtualisation has been supported on AMD even before Intel - VMCS shadowing is something else. One thing that AMD chips really don't support are AVX-512 instructions, but given the mess Intel's made of this, the missing support likely won't be relevant for a while, if ever.

Al_Koch
Enthusiast
Enthusiast

Hi ender_

Thanks for your comments. Can you tell me when you think you'd be able to comment on your experience with VMware Desktop running on this chip? Also, if you are comfortable, please describe your use case. How many VMs running simultaneously, what are they running, # cores/VM are issues I have been trying to get info on but have had little luck.

Again, thanks for your input.

 

Reply
0 Kudos
ender_
Expert
Expert

I called the store where I ordered 5900X today, they said that I'm 6th in the line, and that they get between 3 and 10 chips per week from AMD, so hopefully I'll get mine sometime next week (I already have everything else).

As for how many VMs, I usually run 2-6, depending on what I'm testing. On my current setup they're configured with 2 cores and 4GB RAM, but I'm planning to increase that when I get the new CPU. VMs are different Windows versions, and I mainly use them to test my programs on different OSes (clean install, various upgrade paths).

Al_Koch
Enthusiast
Enthusiast

Hi ender_,

Thanks for the info. Your use case has some similarity to mine which is for software dev and test along with the various servers needed. Currently about the same # of VMs and cores and RAM/VM.

I'm sure once you get the chip and assemble your system you'll be very busy using it but if you could take a moment and let me know how well (hopefully!) the VMs ported and are running on the chip it would be very helpful and greatly appreciated.

Thanks and good luck.

 

 

Reply
0 Kudos
Al_Koch
Enthusiast
Enthusiast

Hi ender_,

Hopefully by now you got your system up and running and it has gone well. Can you offer any feedback on how the chip is working with VMware?

Thank you.

Reply
0 Kudos