ralish Tracker

Re: High CPU usage by vmnat.exe after upgrade to VMware® Workstation 17 Pro version 17.5.0 build-225

ralish — Sun, 05 Nov 2023 00:34:36 GMT

Frustrating to see this was both reported during the beta and despite the numerous replies in this thread still hasn't had a response.

For anyone from VMware who can assist with getting this resolved, this is a stack trace of the thread on which the high CPU activity occurs, in which it appears to be stuck in an infinite loop:

ntoskrnl.exe!KiSwapContext+0x76 ntoskrnl.exe!KiSwapThread+0xab5 ntoskrnl.exe!KiCommitThreadWait+0x137 ntoskrnl.exe!KeWaitForSingleObject+0x256 ntoskrnl.exe!KiSchedulerApc+0x23e ntoskrnl.exe!KiDeliverApc+0x2f6 ntoskrnl.exe!KiCheckForKernelApcDelivery+0x2b ntoskrnl.exe!ExReleaseResourceAndLeaveCriticalRegion+0xf1 win32kbase.sys!UserSessionSwitchLeaveCrit+0x137 win32kfull.sys!NtUserMsgWaitForMultipleObjectsEx+0x529 win32k.sys!NtUserMsgWaitForMultipleObjectsEx+0x20 ntoskrnl.exe!KiSystemServiceCopyEnd+0x25 wow64win.dll!NtUserMsgWaitForMultipleObjectsEx+0x14 wow64win.dll!whNtUserMsgWaitForMultipleObjectsEx+0x90 wow64.dll!Wow64SystemServiceEx+0x164 wow64cpu.dll!ServiceNoTurbo+0xb wow64cpu.dll!BTCpuSimulate+0xbb5 wow64.dll!RunCpuSimulation+0xd wow64.dll!Wow64LdrpInitialize+0x12d ntdll.dll!_LdrpInitialize+0xe7 ntdll.dll!LdrpInitializeInternal+0x6b ntdll.dll!LdrInitializeThunk+0xe win32u.dll!_NtUserMsgWaitForMultipleObjectsEx@20+0xc USER32.dll!MsgWaitForMultipleObjectsEx+0x51 USER32.dll!_MsgWaitForMultipleObjects@20+0x1f vmnat.exe+0x3348 vmnat.exe+0x15391 ntdll.dll!___RtlUserThreadStart@8+0x2b ntdll.dll!__RtlUserThreadStart@8+0x1b

Taken from VMware Workstation Pro v17.5.0 on Windows 11 23H2.

Nested hypervisor support under Windows Hypervisor Platform

ralish — Thu, 27 Jul 2023 03:59:08 GMT

Are there any plans to supported nested virtualisation under Windows Hypervisor Platform?

If Hyper-V is installed, or any of several security features are enabled on the host OS (Virtualisation Based Security), then the Windows Hypervisor Platform is used instead of VMware's native hypervisor. This breaks support for VMs which required nested virtualisation (e.g. ESXi guests, Windows guests that themselves have VBS enabled). The underlying support appears to be present in the Windows Hypervisor Platform given nested virtualisation can be used on VMs directly created using Hyper-V, but VMware Workstation does not have any awareness of it.

Previously discussed on these topics but have not received any comment from VMware:

Re: No virtual machine console preview

ralish — Sun, 29 Jan 2023 04:53:16 GMT

Good idea, but no luck I'm afraid. All testing is being performed using the default Administrator@vsphere.local user with the Administrator role assigned. I double checked in VCSA -> Administration -> Access Control -> Roles to be sure, and the Administrator role does contain those privileges.

I'm really stumped as the troubleshooting I've done suggests VCSA is calling a non-existent API, but both the ESXi host and the VCSA server are in the 8.x series (and the 7.x series before that), so there's no weird mismatch between the two systems.

Re: No virtual machine console preview

ralish — Sun, 29 Jan 2023 02:26:32 GMT

Just bumping this in the event anyone from VMware or has seen this issue themselves has any guidance? Won't be bumping it again!

Re: No virtual machine console preview

ralish — Wed, 18 Jan 2023 22:58:43 GMT

Yeah, same issue across all browsers I've tried, and the issue was present in VCSA 7.x so it's not a regression introduced by VCSA 8.x.

No virtual machine console preview

ralish — Wed, 18 Jan 2023 07:59:06 GMT

I'm trying to resolve an issue that I've had for as long as I can remember and would be glad for any advice from the community! The issue is minor: within vCenter Server the preview of the console of any selected VM doesn't load. Opening the console via either "Launch Web Console" or "Launch Remote Console" works fine, as does the console preview when accessing the ESXi system directly with the ESXi Host Client. So whatever's wrong, it seems to *only* affect the console preview within vCenter Server.

So I finally decided to take a look at it and went down a rabbit-hole. We start with the browser sending a POST request to VCSA with a URL like (content in angle brackets varies): https://<host>/ui/data/propertiesWithParameters/urn:vmomi:VirtualMachine:vm-<id>:<guid>?properties=vmScreenshot.

The request will eventually complete "successfully" with a HTTP 200 response after ~75 seconds, but it has actually timed-out server-side. In VCSA you see something like this:

So what's happening behind the scenes? Trying to trace the request path it looks to be:

Request is received by the vSphere Client (vsphere-ui) service.
The vSphere Client forwards the request or sends a separate one (not sure which) to vCenter Server (vpxd).
The vCenter Server sends a request to the vCenter Agent (vpxa) on the ESXi host where the virtual machine resides.

Here's where things ultimately seem to go wrong. From looking at the vpxa.log on the ESXi host we see something like this (unrelated operations removed for brevity):

2023-01-18T07:36:00.346Z In(166) Vpxa[2099604]: [Originator@6876 sub=vpxLro opID=SWI-2085ce76-58] [VpxLRO] -- BEGIN lro-205016 -- vpxa -- vpxapi.VpxaService.GetVmScreenshot -- 52189394-fa47-1531-ac6d-25895cdad128
2023-01-18T07:37:15.348Z Wa(164) Vpxa[2099591]: [Originator@6876 sub=IO.Connection opID=SWI-2085ce76-58] Failed to connect; <io_obj p:0x000000f550f47588, h:18, <TCP '2400:3740:200:6710::20 : 59627'>, <TCP '2400:3740:200:6710::20 : 443'>>, e: 110(Connection timed out), duration: 75001msec
2023-01-18T07:37:15.349Z In(166) Vpxa[2099591]: [Originator@6876 sub=IO.Connection opID=SWI-2085ce76-58] Failed to shutdown socket; <io_obj p:0x000000f550f47588, h:18, <TCP '2400:3740:200:6710::20 : 59627'>, <TCP '2400:3740:200:6710::20 : 443'>>, e: 104(shutdown: Connection reset by peer)
2023-01-18T07:37:15.349Z Wa(164) Vpxa[2099591]: [Originator@6876 sub=HttpConnectionPool-000000 opID=SWI-2085ce76-58] Failed to get pooled connection; <cs p:000000f5505b3950, TCP:[2400:3740:200:6710::20]:443>, (null), duration: 75002msec, N7Vmacore15SystemExceptionE(Connection timed out)
2023-01-18T07:37:15.351Z Wa(164) Vpxa[2099556]: --> [context]zKq7AVICAgAAAEkBOQEMdnB4YQAAYxwbbGlidm1hY29yZS5zbwAAsFUvAN9PNQCoUDUARVg1AH1eNQADBzAA7oEvAACeLwArGEEBO30AbGlicHRocmVhZC5zby4wAAJ90Q5saWJjLnNvLjYA[/context]
2023-01-18T07:37:15.351Z In(166) Vpxa[2099591]: [Originator@6876 sub=IO.Http opID=SWI-2085ce76-58] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection timed out)
2023-01-18T07:37:15.353Z In(166) Vpxa[2099556]: --> [context]zKq7AVICAgAAAEkBOQEMdnB4YQAAYxwbbGlidm1hY29yZS5zbwAAsFUvAN9PNQCoUDUARVg1AH1eNQADBzAA7oEvAACeLwArGEEBO30AbGlicHRocmVhZC5zby4wAAJ90Q5saWJjLnNvLjYA[/context]
2023-01-18T07:37:15.353Z Er(163) Vpxa[2099591]: [Originator@6876 sub=IO.Http opID=SWI-2085ce76-58] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection timed out)
2023-01-18T07:37:15.355Z Er(163) Vpxa[2099556]: --> [context]zKq7AVICAgAAAEkBOQEMdnB4YQAAYxwbbGlidm1hY29yZS5zbwAAsFUvAN9PNQCoUDUARVg1AH1eNQADBzAA7oEvAACeLwArGEEBO30AbGlicHRocmVhZC5zby4wAAJ90Q5saWJjLnNvLjYA[/context]
2023-01-18T07:37:15.353Z Wa(164) Vpxa[2099604]: [Originator@6876 sub=Vmomi opID=SWI-2085ce76-58] VMOMI activation LRO failed; <<52189394-fa47-1531-ac6d-25895cdad128, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 23184'>>, vpxa, vpxapi.VpxaService.GetVmScreenshot>, N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound
2023-01-18T07:37:15.358Z Wa(164) Vpxa[2099556]: --> )
2023-01-18T07:37:15.358Z Wa(164) Vpxa[2099556]: --> [context]zKq7AVICAgAAAEkBOQETdnB4YQAAYxwbbGlidm1hY29yZS5zbwABaCsYdnB4YQABxj4eAXeyIgFSPx4Cy64LbGlidnB4YXBpLXR5cGVzLnNvAAG+oCoDz+QXbGlidm1vbWkuc28AARrIGQH9DSoBQA8qAVceKgGvUykBK/UpAO6BLwAAni8AKxhBBDt9AGxpYnB0aHJlYWQuc28uMAAFfdEObGliYy5zby42AA==[/context]
2023-01-18T07:37:15.358Z In(166) Vpxa[2099604]: [Originator@6876 sub=vpxLro opID=SWI-2085ce76-58] [VpxLRO] -- FINISH lro-205016
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099604]: [Originator@6876 sub=Default opID=SWI-2085ce76-58] [VpxLRO] -- ERROR lro-205016 -- 52189394-fa47-1531-ac6d-25895cdad128 -- vpxa -- vpxapi.VpxaService.GetVmScreenshot: :vim.fault.NotFound
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> Result:
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> (vim.fault.NotFound) {
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> faultCause = (vmodl.MethodFault) null,
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> faultMessage = <unset>
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> msg = ""
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> }
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> Args:
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: -->
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> Arg vmid:
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> 9
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> Arg snapshot:
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: -->
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> Arg options:
2023-01-18T07:37:15.358Z Er(163) Vpxa[2099556]: --> "&h=120&w=160"

So it's like the vCenter Agent is unable to invoke a VMOMI method to obtain a screenshot of the console because the TCP connection to itself never completes.

The following output would seem to confirm this (interesting lines in bold):

esxcli network ip connection list
Proto Recv Q Send Q Local Address Foreign Address State World ID CC Algo World Name
----- ------ ------ --------------------------------------------- ------------------ ----------- -------- ------- ----------
tcp 0 0 [2400:3740:200:6710::20]:443 [2400:3740:200:6710::20]:59627 RCVD 0
tcp 0 0 10.10.16.32:443 10.10.16.128:35194 ESTABLISHED 2098852 newreno rhttpproxy-IO
tcp 0 0 [2400:3740:200:6710::20]:59627 [2400:3740:200:6710::20]:443 SYN_SENT 2099579 newreno vpxa-worker
tcp 0 0 10.10.16.32:443 10.10.16.128:34816 ESTABLISHED 2098852 newreno rhttpproxy-IO
tcp 0 0 10.10.16.32:443 10.50.50.40:52070 ESTABLISHED 2098852 newreno rhttpproxy-IO
tcp 0 0 10.10.16.32:443 10.50.50.40:53845 ESTABLISHED 2098851 newreno rhttpproxy-IO
tcp 0 0 10.10.16.32:443 10.10.16.128:59700 ESTABLISHED 2098852 newreno rhttpproxy-IO
tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 2098789 newreno rhttpproxy
tcp 0 0 [::]:443 [::]:0 LISTEN 2098789 newreno rhttpproxy

But why is that? I thought maybe it's an edge case the firewall doesn't handle, so tried disabling it with "esxcli network firewall unload" but it had no impact. My only thought at this point is it's something related to IPv6, but the connection is to itself (it's effectively loopback). Has anyone got any clever ideas? Is there something obvious I've overlooked?

The ESXi host is 8.0 (Build 20513097) and vCenter Server is 8.0a (Build 20920323) but this issue was around for as long as I remember on various vSphere 7.x releases before I upgraded.

VMware engagement

ralish — Thu, 17 Nov 2022 23:08:02 GMT

I'm not a VMware employee, so have no visibility to what's happening in the company or the team that develops VMware Workstation. There's doubtless things happening I'm not aware of, but as a VMware customer, it's really not a great look that a Technical Preview is launched to solicit customer feedback and there's effectively no engagement by VMware.

I've personally surfaced an issue and a feature request, both unaddressed, and the issue is the survey links for this preview are broken. That was two months ago! Presumably it doesn't matter, as if anyone cared about survey responses, they'd have noticed there aren't any and so fixed it by now. The lack of any engagement by VMware seems to be a common experience among posters.

What's the point of a Technical Preview if the feedback is just left not merely unresolved but not even acknowledged and responded to? People are taking the time to give constructive feedback and it's simply ignored. Doubtless Workstation Pro 17 will be a paid upgrade, and at least for me it's going to be a bitter feeling if I do fork out the upgrade cost knowing my and others' input has simply been ignored.

Re: Nested hypervisor support under VBS (inc. Device Guard)

ralish — Mon, 19 Sep 2022 02:46:48 GMT

Looking at the Windows Hypervisor Platform headers it does appear to be possible.

There's the NestedVirtSupport bit of the WHV_PROCESSOR_FEATURES1 structure which is passed as the input buffer to WHvSetPartitionProperty with the WHvPartitionPropertyCodeProcessorFeaturesBanks property code. There's also the WHvPartitionPropertyCodeNestedVirtualization property which appears to take a BOOL as the input buffer to the function.

I'm not clear how these two approaches differ, or how one affects the other. The WHvPartitionPropertyCodeNestedVirtualization property feels the most promising. It's noted in the Data Types documentation for the function that NestedVirtualisation is supported since Windows 10 19H2.

This is from a very quick look at the API documentation and header files, so may not be 100% accurate, but overall appears promising.

Re: Windows Server 2022 BSOD on guests with multiple vCPUs

ralish — Sun, 18 Sep 2022 23:40:11 GMT

Adding a note that somewhat curiously this doesn't affect Windows 11. It's entirely specific to Windows Server 2022 (Build 20348), and doesn't affect the latest Windows 10 release (Build 19044) or Windows 11 (Build 22000).

Windows Server 2022 BSOD on guests with multiple vCPUs

ralish — Sun, 18 Sep 2022 21:39:09 GMT

I haven't yet had the chance to test this on the latest tech preview, but Windows Server 2022 guests with recent cumulative updates will bluescreen with UNSUPPORTED_PROCESSOR if the guest is configured with multiple vCPUs. The only known workaround is to limit the guests to a single vCPU (both socket and core), with the obvious potential performance impact.

Other users have documented this here but there's been no response yet from VMware, nor is it documented in the release notes as a known issue or in the knowledge base. It's definitely still an issue in the latest stable release (v16.2.4). Hopefully this will be fixed in the next release, and ideally should be backported to supported older releases.

If anyone with the tech preview installed is able to confirm if the issue is still present that'd be very helpful, otherwise I'll try and find some time to test myself soon.

Nested hypervisor support under VBS (inc. Device Guard)

ralish — Sun, 18 Sep 2022 21:32:59 GMT

Seeing as the survey links are broken, at least for me (see here), I'm posting on the board instead.

A feature I haven't seen discussed but which would be extremely useful is nested hypervisor support under Hyper-V enabled hosts (i.e. using the Windows Hypervisor Platform). I've posted some thoughts about this before here, but to summarise, if running on a host which is Hyper-V enabled you can't run guests under VMware Workstation which expose Intel VT-x/EPT. I assume the same issue is present if exposing AMD-V/RVI but don't have such a system to test on. Virtualising the IOMMU does work.

The impact is you can't run nested virtualisation scenarios on a system with Hyper-V enabled, be it because you actually use Hyper-V alongside VMware Workstation, or it's a dependency of other features like Device Guard. Where this is particularly frustrating is it blocks running VBS enabled guests as they require VT-x/AMD-V.

This limitation doesn't appear to apply to Hyper-V itself, as such configurations work fine on Hyper-V VMs, which suggests it's technically possible.

Survey links are broken

ralish — Sun, 18 Sep 2022 21:18:10 GMT

Are the survey links broken for everyone else? I was going to respond to one of them, but both the short and detailed survey links aren't actually hyperlinks. They look like it, but once you hover over them you'll see they don't link to anything. Tested on multiple browsers.

Re: Nested hypervisor support under VBS

ralish — Fri, 26 Nov 2021 23:22:04 GMT

Bumping one time for any input from a VMware employee. Another area this causes problems is network labs using tools like GNS3.

Re: Workstation pro 16.2 Crashes

ralish — Tue, 09 Nov 2021 20:19:25 GMT

Fixed in v16.2.1.

Re: Nested hypervisor support under VBS

ralish — Tue, 09 Nov 2021 01:19:51 GMT

Some results from a very quick test using Hyper-V on a Windows 10 v21H1 x64 host w/ VBS enabled. All testing was performed in a Generation 2 VM with a fresh Windows 10 v21H1 x64 installation:

Enabling VBS (aka. Core Isolation) worked with no additional changes. All that was required was enabling Core Isolation via the Windows Security app and rebooting for the requisite Windows support to be installed and enabled. I've attached a screenshot from System Information post-reboot showing VBS enabled in the VM.
Nested virtualisation also works with a few extra steps. These are documented by Microsoft here. To summarise, you need to enable nested virtualisation for the (outer) VM, disable dynamic memory for the (outer) VM, and enable Hyper-V in the (inner) VM. I was then able to launch a Hyper-V VM inside the guest VM.

So to summarise, it clearly is possible under Hyper-V to use both VBS enabled VMs and nested virtualisation (inc. simultaneously), including on hosts which themselves have VBS enabled. It being technically possible, the next question is does Microsoft expose the necessary public APIs for 3rd-parties to leverage these configurations?

Is anyone from VMware able to comment if such support is on the development roadmap and if there are any major blockers to adding it?

Re: Nested hypervisor support under VBS

ralish — Tue, 09 Nov 2021 00:29:36 GMT

I suppose the best way to confirm the current state of affairs is to see if a Hyper-V VM can be launched with VBS enabled in the guest while VBS is enabled on the host. If the answer is no, then it's almost certainly not possible under VMware either when using Hyper-V as the virtualisation backend. If the answer is yes, the question is around if the relevant support is exposed through documented APIs.

Re: Workstation pro 16.2 Crashes

ralish — Tue, 09 Nov 2021 00:27:51 GMT

Is anyone from VMware able to acknowledge this issue and confirm a fix is forthcoming? At least in my scenario it's reproducible every time.

Re: Workstation pro 16.2 Crashes

ralish — Tue, 02 Nov 2021 01:59:17 GMT

I'm seeing the exact same issue. Have attached a stack trace if at all helpful from reproducing the crash with a debugger attached. If someone from VMware is viewing this thread feel free to reply with any extra details that would be helpful for getting out a fix.

Nested hypervisor support under VBS

ralish — Mon, 01 Nov 2021 01:16:12 GMT

Are any VMware developers monitoring this forum able to comment on future support for nested hypervisors on systems with Hyper-V (or dependent features like Device Guard)? At the time host VBS support was introduced back in Workstation v15.5.5 my understanding was this feature was missing due to limitations in the Windows Hypervisor Platform API. Is this still the case? Are there any plans to add support and what are the roadblocks to doing so?

If at all helpful as background, my use case is testing VBS configurations in VMs on a host which itself uses VBS, as well as wanting to test ESXi configurations in a VM. Both of these scenarios require Intel VT-X/EPT virtualisation (or AMD-V/RVI for AMD CPUs), which isn't supported with host VBS.

Thanks in advance!

Re: VMware Workstation upgrade issue with WDAG

ralish — Sat, 13 Jun 2020 03:43:09 GMT

Regarding your additional two Qs:

Windows 10 v2004 Enterprise x64 (Final release; i.e. Build 19041). Only the 2004 release has the support required for Hyper-V interop (excluding Insider builds).
Right now I'm not sure I'd call it a bug in either. I don't know enough about WDAG to say if keeping locks on those catalog files is expected behaviour, and the VMware Workstation Pro installer presumably just isn't expecting other processes to be maintaining references to those files. Updating the VMware Workstation Pro installer to at least check for this case would likely be the simplest path forward (and updating the release notes), but it's not necessarily a bug, just a system configuration that should be handled.