Does VMRUN support being executed by Windows task scheduler WITHOUT interactive logon?

[ams_tschoening]

I'm using VMware Workstation Pro 16.1.1 and need to run a VM after Windows booted. The important thing is that the VM needs to be executed using a specially restricted default user of Windows which WILL NOT logon itself interactively to get a desktop or shell. Instead, the current approach is to use task scheduler with a task fired after system boot executing the following command line. That task is configured to fire INDEPENDENTLY of any user login and execute as the specially created user.

"C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" -T ws start "C:\[...]\headless_test.vmx" nogui

Though, launching the VM this way always fails, while the same command line always succeeds when using it e.g. in a shell created by the user. I've read a lot of posts regarding this topic and there seem to be 3 parties in the end: People like me claiming that it doesn't work, those who claim it does work BUT enable in their task that it should fire only on interactive login and VERY few people how claim to not do the former and claim that things work. I'm somewhat sure now that VMRUN can't work WITHOUT interactive logon, because it's always resulting in the same errors like problems in finding some special pipes, getting some ports for "authd" and stuff like that. VirtualBox has pretty similar limitations and all of this even makes sense when keeping in mind that there's an explicit featured called shared VMs working differently.

The problem is: It's not documented anywhere by VMware, who should simply know how things work, if VMRUN is designed to work in this context at all. So, any chance to get a definitive answer from VMware about this setup? Thanks!

The following is the last most important logs I get:

| vmui| I005: Vix: [foundryVMPowerOps.c:984]: FoundryVMPowerStateChangeCallback: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx, vmx/execState/val = poweredOff.
| vmui| I005: VMMKS::OnMKSDeviceInfoChanged: reset pipe name now in offline.
| vmui| I005: VMMKS::OnMKSDeviceInfoChanged: reset pipe name now in offline.
| vmui| I005: SnapshotTree: Emitting refresh (C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx)
| vmui| I005: SnapshotTree: Populating (C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx)
| vmui| I005: Found vmx as C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
| vmui| I005: Starting vmx as C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
| vmui| I005: VigorExecVMXExCommon: Failed to connect the pipe: Unknown error 2 (0x2)
| vmui| I005: VigorOnline_StartAndConnectEx Failed: VMware Workstation cannot connect to the virtual machine. Make sure you have rights to run the program, access all directories the program uses, and access all directories for temporary files.
| vmui| I005+ Failed to connect pipe to virtual machine: Unknown error 2 (0x2).
| vmui| I005+ 
| vmui| I005: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx: VMHSStartVmxVigorCb: vmPath=/vm/#bc66f91c8afe1c6e/ status=error
| vmui| I005: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx: VMHSStartVmxVigorCb: error vmdb=-44 msg=VMware Workstation cannot connect to the virtual machine. Make sure you have rights to run the program, access all directories the program uses, and access all directories for temporary files.
| vmui| I005+ Failed to connect pipe to virtual machine: Das System kann die angegebene Datei nicht finden.
| vmui| I005+ 
| vmui| I005: vm power operation abort, error = VMware Workstation cannot connect to the virtual machine. Make sure you have rights to run the program, access all directories the program uses, and access all directories for temporary files.
| vmui| I005+ Failed to connect pipe to virtual machine: Das System kann die angegebene Datei nicht finden.
| vmui| I005+  .
| vmui| W003: VMware Workstation cannot connect to the virtual machine. Make sure you have rights to run the program, access all directories the program uses, and access all directories for temporary files.
| vmui| W003+ Failed to connect pipe to virtual machine: Das System kann die angegebene Datei nicht finden.
| vmui| W003+ 
| vmui| I005: DlgUI: VMware Workstation cannot connect to the virtual machine. Make sure you have rights to run the program, access all directories the program uses, and access all directories for temporary files.
| vmui| I005+ Failed to connect pipe to virtual machine: Das System kann die angegebene Datei nicht finden.
| vmui| I005: Win32U_GetFileAttributes: GetFileAttributesExW("C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmpl", ...) failed, error: 2
| vmui| I005: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx: Reloading config state.
| vmui| I005: VMHS: Transitioned vmx/execState/val to poweredOff
| vigorCnx| I005: VigorOfflineGetToolsVersion: updated tools version to 0
| vmui| I005: Vix: [foundryVMPowerOps.c:984]: FoundryVMPowerStateChangeCallback: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx, vmx/execState/val = poweredOff.
| vmui| I005: SnapshotTree: Populating (C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx)
| vmui| I005: SnapshotTree: Emitting refresh (C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx)
| vmui| I005: VMMKS::OnMKSDeviceInfoChanged: reset pipe name now in offline.
| vmui| I005: VMHSGetDataFileKey: Could not get the dataFileKey from VMDB
| vmui| I005: [headless_test] does not support shrink disks.
| vmui| I005: VMMgr::CloseVM: closing VM at /vm/#bc66f91c8afe1c6e/
| vmui| I005: cui::MKSScreenView::SetRenderTarget: hostWindow and surfaceID are none. id: 1. 
| vmui| I005: cui::MKSScreenView::OnHostWindowChanged, id: 1, unsetting destination and setting is rendering to false
| vmui| I005: Switched to tab: 
| vmui| I005: CVMUIView::UpdateView - showMks (no).
| vmui| I005: VMMgr::OnVMDestroyed: cleaning up after destroyed VM at /vm/#bc66f91c8afe1c6e/
| vmui| I005: CUIMKS: Destroy cui::MKS (5BAF290)
| vmui| I005: CUIMKS: cui::MKS::SetAttached (5BAF290): detach
| vmui| I005: CUIMKS: cui::MKS::OnSetAttachedCompleted (5BAF290)
| vmui| I005: CUIMKS: Destroying cui::MKS (5BAF290) is done.
| vmui| I005: MKSWindowTrans: The current transaction 5BFE1E0 has not been submitted.
| vmui| I005: MKSControlClient: Destroy mksControl client 5A6A418.
| vmui| I005: MKSControlClient: Disconnect is called (5A6A418) (current state 0).
| vmui| I005: MKSControlClient: Disconnect is done (5A6A418) (current state 0).
| vmui| I005: MKSControlClient: Destroying mksControl client 5A6A418 is done.
| vmui| I005: Exit requested.
| vmui| W003: wui::DlgMgr::Refresh: Removing a stale HWND
| vmui| I005: CDS: Clearing the credentials cache.
| usbCCIDEnumCards| I005: USB-CCID: Card enum thread exiting.
| vmui| I005: WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0
| vmui| I005: Exited normally.

VMRUN logs the following, especially with the last log messages rpeeated over and over again until the process stops:

| vix| I005: Host OS: 'Windows 8 Pro, 64-bit (Build 19042) 6.2.9200', product type '1', suite mask '0x0100'.
| vix-poll| I005: Win32U_GetFileAttributes: GetFileAttributesExW("C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmpl", ...) failed, error: 2
| vix-async-pipe| I005: CnxAuthdConnect: Returning false because CnxAuthdConnectPipe failed
| vix-async-pipe| I005: CnxConnectAuthd: Returning false because CnxAuthdConnect failed
| vix-async-pipe| I005: Cnx_Connect: Returning false because CnxConnectAuthd failed
| vix-async-pipe| I005: Cnx_Connect: Error message: Failed to read vmware-authd port number: Cannot connect to VMX: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx
| vix-async-pipe| I005+
| vix-async-pipe| I005: Vix: [foundryVMPowerOps.c:5918]: Error VIX_E_HOST_TCP_SOCKET_ERROR in FoundryVMOpenSocketToVMXThroughAuthd(): unable to contact authd: Failed to read vmware-authd port number: Cannot connect to VMX: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx
| vix-async-pipe| I005+
| vix| I005: Vix: [foundryHandleProperties.c:3360]: Error VIX_E_NOT_FOUND in Vix_GetPropertiesImpl(): Unable to get bool property 107.
| vix-async-pipe| I005: CnxAuthdConnect: Returning false because CnxAuthdConnectPipe failed
| vix-async-pipe| I005: CnxConnectAuthd: Returning false because CnxAuthdConnect failed
| vix-async-pipe| I005: Cnx_Connect: Returning false because CnxConnectAuthd failed
| vix-async-pipe| I005: Cnx_Connect: Error message: Failed to read vmware-authd port number: Cannot connect to VMX: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx
| vix-async-pipe| I005+
| vix-async-pipe| I005: Vix: [foundryVMPowerOps.c:5918]: Error VIX_E_HOST_TCP_SOCKET_ERROR in FoundryVMOpenSocketToVMXThroughAuthd(): unable to contact authd: Failed to read vmware-authd port number: Cannot connect to VMX: C:\Users\vmware\Documents\Virtual Machines\headless_test\headless_test.vmx

Neither of that looks like a setup problem, I don't see any problems in Process Monitor or elsewhere. As said already, when using an interactive shell things instantly work as expected. I guess there are some problems with concepts like different Windows Stations, desktops etc. of users, services and fired tasks, who is able to find which IPC mechanism under which name and stuff like that.

But this would really mean that this setup simply can't work and it would be great tot have this documented once and for all somewhere. In theory, providing the possibility to let task be fired with special user accounts makes executing headless VMs a lot safer than using shared VMs, which execute using the SYSTEM account by default.

[wila]

Hi,

Somehow I have spent a lot of time with this....

The answer is: yes if you don't want to run with a GUI then you can run the VM under another user.

Note however that a user like the SYSTEM account has a lot of limitations, like you cannot use 3D acceleration, you can't use USB, no copy & paste clipboard etc...

This is pretty much also how the old VMware Shared VM's feature used to work. Those VM's did run under the system account and that's also where the limitations for shared VM's came from.

So why did I spent a lot of time on this?

It's because I wrote a tool called vimarun

It is designed to run a VM on startup and to suspend that VM on shutdown. The default/primary usage is to run under the normal user, but you can set a VM to run in the background only.
Note that the VM (or VM's) that you configure will run without having to login.

If you _really_ need the VM to run under another user, then you can enter "SYSTEM" for username and a dummy password (it will be ignored) during setup. I have not run many tests with that, but it should work.

Note however that when you run a VM under the SYSTEM account that you cannot run the VM in the foreground.

So in short, yes it can be done.

No, I don't know why vmrun isn't working for you when you try to run it as a task under a different user.

edit: One thing that should work...
If you actually login with your restricted user and then install vimarun under that user and configure to run VM(s) for that user.
This then also gives you the opportunity to test if vmware workstation runs the VM when you try to run it from the GUI itself.

--
Wil

[ams_tschoening]

No, I don't know why vmrun isn't working for you when you try to run it as a task under a different user.

But there's a reason you are using a custom system service with service wrapper and stuff instead of VMRUN and task scheduler yourself? Did you actually ever use VMRUN+task-scheduler+restricted-user+non-interactive successfully?

Making the restricted user being a member of the ADMINISTRATORS group the VM executes successfully and I get the black screen-problem in the GUI as well. But that's not an improvement over using shared VMs and SYSTEM anyway.

It is designed to run a VM on startup and to suspend that VM on shutdown. The default/primary usage is to run under the normal user, but you can set a VM to run in the background only.
Note that the VM (or VM's) that you configure will run without having to login.

With your service in place, the VM starts automatically WITHOUT interactive login and when some users logs in, one is still able to use the normal VMware GUI to maintain the VM? Especially interact with the guest desktop if necessary?

You don't want to share what you are doing differently than VMRUN to work at all and bypass the black screen or at least what's the root cause of that when using task scheduler?

Did you ever test VIMARUN with a restricted, third user account for your service? An account NOT being the one used to interact with the desktop day-to-day, NOT being SYSTEM? And still allowing interaction with VMware GUI when its started with that special account? If that works, that would be a huge benefit over shared VMs.

[wila]

Hi,

The reason I use a service instead of using a windows task is that I have much more control from a service than from a windows task. This is not because you cannot run a VM interactively from a task, you can, but only AFTER the login. As in Yes, I did investigate using tasks, but not using other users besides the current -logged in- user and the local system account.

I have not investigated your specific user case with restricted user logins. Btw, what is your definition for a restricted user? Just a default normal windows user account that is not added to the administrator group or do you add additional restrictions to that account?

If you run a VM under another user account using vmrun then AFAIK there is no possibility to access the VM console from VMware Workstation's GUI. This includes the SYSTEM account. That it works for VMware shared VM's is because VMware controls the whole stack and could create their own communication layer for that.

I have no problem sharing knowledge, my limit is when I have to spent time... The reason I specifically state things about vimarun is because that's what I tested and it is what I support. But you can at times just read "vmrun" when I say "vimarun"..  currently vmrun is the underlying API that is used.

The black screen that you see is because if you start the VM interactively and there is no desktop active for that user at that specific moment then the VM runs in another windows session. It is -since Windows Vista- not possible to access the windows desktop from another windows session.

There is a trick.. you can suspend the VM and then resume it again. That will remove the VM from the old session that has no access to your desktop and put it in the current session.

FWIW, I'm also wondering a bit about the attack surface that you are trying to limit by running under a restricted account. Yes,  the vmx will be running within that user context, but if an attacker can break out of the VM, then we have bigger problems already. Protection in layers certainly is a good thing, just wondering if it doesn't take away here more than it adds. So not saying it isn't a good idea, just wondering.

--
Wil

[ams_tschoening]

Btw, what is your definition for a restricted user? Just a default normal windows user account that is not added to the administrator group[...]

Exactly, that's the plain basics I always try to start with and is known good practice especially for daemons/services not only on Windows.

The black screen that you see is because if you start the VM interactively and there is no desktop active for that user at that specific moment then the VM runs in another windows session. It is -since Windows Vista- not possible to access the windows desktop from another windows session.

Something like this is what I expect to be the root cause for my tests regarding VMRUN and task scheduler failing as well. Even though all processes started by task scheduler and VMRUN share the same security context like Window Station etc., I guess the started processes simply not expect to find the environment provided by task scheduler in my setup.

Too bad, but I might give VIMRUN and it's service approach another try, maybe that really changes something. Though, as you said to have only used SYSTEM and the current interactive user, I would expect to have the same problem in my setup. Task scheduler is a service in itself only and impersonates users, sc.exe doesn't do anything different in the end.

FWIW, I'm also wondering a bit about the attack surface that you are trying to limit by running under a restricted account.

Am really just trying to stick as often as possible to well-known good practices, which not only provide some sense of security against evil guys, but against own mistakes as well. Already deleted lots of Windows DLLs lots of years ago simply because being ADMIN and issuing a command in the wrong directory by accident... 🙂 One needs to start somewhere thinking about all that stuff.