Workspace ONE Assist 22.10.00.12 On-Premise - Stuck on "Sending Message"

[spg123]

Hello Community,

I am posting here as I have created a VMware support ticket which is receiving no response after several attempts. In fact, they have pretty much ghosted us. 😕

We are an on-premise customer for Workspace ONE UEM (22.9.0.7) and Workspace ONE Assist.

After VMware published a security advisory I quickly moved to patch our Workspace ONE Assist environment from 22.04.00.5 to 22.10.00.12, however we seem to be unable to connect to devices. It progresses to "Sending Message" then it fails with "The connection was closed abnormally". In the event log, it has this error message:

Dec-14-2022 07:28:48.279 UTC [INFO] : X-Frame-Options : SAMEORIGIN
Dec-14-2022 07:29:20.759 UTC [INFO] : DisplayCapture.InitializeWebSocketandConnect >> Session Collaboration enabled
Dec-14-2022 07:29:20.936 UTC [INFO] : WebSocket ON CLOSE : DISCONNECT DEVICE : The connection was closed abnormally, e.g., without sending or receiving a Close control frame
Dec-14-2022 07:29:20.936 UTC [INFO] : WebSocket ON CLOSE : Reason : 

 Has anyone come across this error message before?

I thought it could be a firewall issue, but the ports (443, 8443) are open and even on an iOS device it shows that an Administrator would like to connect, and I can begin broadcasting the screen. But still the same error message. I can connect to the AdminWebPortal also (once enabling it). Everything seems to work except connecting to a device (Windows 10, MacOS, iOS).

We have a simple set up with a single on-premise Assist server which connects to a SQL database. Rolling back to upgrade, it works perfectly (though with security vulnerabilities). I even tried deploying Assist from scratch (without any anti-virus) , and the same issue occurs.

Any help or guidance from the VMware community will be very welcome! Thank you for reading my message.

[Fabsa]

Hi,

after a quite intense troubleshooting session due to update issues, we had to install the Assist Host from Scratch.

Right now we are Stuck in exactly the same situatuion like you (running an earlierer Build of UEM). Due to our previous commuincation with the support we where asked to upload Service Logs from Assist and some DB Queries. We also should check if TLS 1.2 is enabled. Seems a bite like standart questions but maybe they have a clue.

Overall it seems like they had to release a not yet finished version of assist due to CVEs. I will ask my colleagues to update this, if they get a proper solution.

[spg123]

Hi Fabsa,

Great to see that we are not the only on-premise customer that is experiencing this issue.

I have discovered that in the .config files for all of the Assist services, the installer is not using localhost:port number but instead

 net.tcp://{0}:{1}

This is OK for a set up of Workspace ONE Assist that is spread across multiple servers (with SRV records set up in DNS), but, it seems the Workspace ONE Assist 22.10.00.12 Installer for Single-Server Set-up (which does not require the DNS records as in multi-server deployment) uses net.tcp://{0}:{1} which results in many exceptions in the Assist service logs, e.g.

Dec-15-2022 16:54:12.134 UTC [EXCEPTION] ServiceManager._ExecuteStartupSequenceAsync@1, System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:8870/svc.

I also noticed this in the .config files

<!--EndPoints that are not from DB Configs
IscEndPoint: "net.tcp://localhost:8870/svc"
IServiceManagementEndPoint: ?<TODO>
IServiceInstrumentationEndPoint: ?<TODO> -->

 I have updated my VMware Support ticket. Hopefully they fix it soon...

[spg123]

To anyone else that is currently experiencing this issue, Workspace ONE Support provided me with a windows command to run and it has managed to proceed to the next step, but still, Workspace ONE Assist 22.10 does not want to work. Workspace ONE support have tried multiple times to walk through the install process and indeed same issue each time. They currently don't know how to resolve the issue.

I genuinely think VMware have pushed out version 22.10 without proper testing to resolve the public vulnerabilities. 😐

[Helena1]

Hi,

in the course of solving the problem with Assist (i am a colleague of Fabsa), we received the following recommendation from the VMware Expert:
<<
Please check the reserved url on Assist server:

open CMD prompt and enter "netsh http show urlacl" => check any reserved url with "https://+:<HTTPS PORT used for portal access>/toolcontroller" name.
If not available, add this URL by the following command,
*netsh http add urlacl url=https://+:<Enter HTTPS PORT used for portal access>/toolcontroller user="NETWORK SERVICE"
[*for example, netsh http add urlacl url=https://+:443/toolcontroller user="NETWORK SERVICE"]
>>

The manual adding this reserved url and disabling the Windows Firewall have given us the positive results.
The Assist on version 22.10 is  finally working!

[spg123 · Accepted solution]

Hi Helena1,

Thank you very much for letting me know about this!

I have found the two things to get the Assist 22.10 installer working are indeed:

1. Open Command Prompt as an Administrator and execute
   
   

   netsh http add urlacl url=https://+:443/toolcontroller user="NETWORKSERVICE"

2. Open Windows Firewall and create an inbound rule to allow port 8443 in via Domain Network.

I really appreciate you sharing this information! You have saved us! I was and am still absolutely shocked by the Workspace ONE team and their inability to resolve this issue after 6 weeks.

Have a fantastic day!