Issues retrieving ENS token from ENS server

[somemdm]

Hi everyone.

After upgrading AirWatch to 21.05, I noticed the mechanism for supplying the ENS token has changed. I created a new child OG for BYOD devices recently and the Boxer payload is having issues retrieving the ENS token from the ENS server itself (new method).

Port 443 is definitely accessible both ways and I can reach the https://ensserver.domain/MailNotificationService/api/ens/alive URL from the console fine.

However I noticed this is the error in the AWMemApi log on the console when the retrieval fails:

(19) Error WanderingWiFi.AirWatch.BusinessImpl.ENS.EnsTenantApiClient+d__7.MoveNext Exception while retrieving token from the given ENS2 Server Address [https://ensserver.domain/MailNotificationService/api/ens]. ensHttpPostResponseContent = [] Exception = [WanderingWiFi.AirWatch.BusinessImpl.Cryptography.SigningServiceIdentityCertificateException: Identity certificate not found or does not have private key

I was wondering if anyone else has gotten this issue? I'm not certain what certificate the error is referring to. Luckily the parent OG ENS functionality still works (token when entered in on earlier versions must still be present in the DB).

Thanks

[SzymonFrankiewi]

Hey, you already have a fix for this, I have a similar problem with adding a token to the ENS.

[Noordan]

Hello,

I have seen this issue before, and that were related to a missing certificate in the database that probably were not generated from the very first UEM installation. 
If I remember correctly you can run the following DB query and you should have an output on all of them.

Select * From dbo.SystemCodeOverride
Where SystemCodeID = 5172
 
Select * From dbo.Systemcodecategory
Where name like '%identity%'
 
Select * From dbo.SystemCodeGroup
Where SystemcodeGroup = 407

Please check the output of these queries.

[SzymonFrankiewi]

After checking the SQL queries for:
1 query is empty.
2. The results on the screen

3. Msg 207, Level 16, State 1, Line 7
Invalid column name 'SystemcodeGroup'.

[Noordan · Accepted solution]

Sorry my bad, the last query should be like below.

Select * From dbo.SystemCodeGroup
Where SystemcodeGroupID = 407

And When I saw this issue, we also had an empty response in one of the queries. And the solution the support gave to us where to re the certinstaller.exe -t {token} tool manually from one of the application servers. where the {token} where replaced with a new token from my.workspaceone.com. And after that we have response on all three queries.

Make sure to take backup of database and snapshots of application servers before.

[SzymonFrankiewi]

Hi,
Thanks for the hint, but it didn't help. There are no entries regarding the certificate in the database. Identity Certificate. But there is an entry in the Certintaller log that this certificate was added.

[Noordan]

hm.. sounds like exactly the same issue that we had. But I am not sure if we had that entry in the certinstaller log or not.

So Im unsure if it is exactly the same issue. So, my suggestion is to open a SR to VMware and provide the information in this thread to them.

[somemdm]

Can confirm what Noordan said is the fix. Grab your token from https://my.workspaceone.com/mycompany/certificates/awinstall/authtoken Open a command prompt as admin, navigate to 'Current AirWatch Install Dir'\Supplemental Software\CertInstaller, Run "CertificateInstaller.exe -t TOKEN" where is the TOKEN retrieved from the URL above. Do this on both console and DS as Noordan has suggested.

[SzymonFrankiewi]

Subject to be closed. You have successfully solved the problem using the method described above. The only thing that puzzles me is why you cannot use the option, ie Offline, i.e. a file with CSR signature on the website and the phrase imported into the system.

[Pro2type]

Did u restart servers after? Did run Certinstaller on Device and console servers successfully but still "NULL" in DB. 

[SzymonFrankiewi]

I'll tell you this. This entry is missing anyway, but it works to get a key from the ENS service and other modules are working using this certificate.

So here's how: I did it as follows: you generate a token from my.workspaceone.com, Of course, remember to make a database backup and machine snaps.
Then with this command CertificateInstaller.exe -t TOKEN add the problematic cert. But remember you have to do this from all Device Services servers and the console. If you have HA it is in every node.

Then you restart the Airwatch and iis services.

Reboot with powershell:
Restart-Service "Airwatch *"
iisreset

After all, do not focus on the database but try to download the token from the ENS server.

[Pro2type]

Okay, so did CertificateInstaller.exe -t "TOKEN" (token from my.workspaceone). 

Restarted services on 2 device servers and 1 console server. Any other servers that should be included? (Not device og console). 

Tried to retrieve token from ENS server, but still: Could not retrieve token. Make sure the ENS2 server address is correct and the ENS2 server is functional, and try again. If the issue persists, review the UEM Console logs and/or ENS2 server logs and contact VMware Support.

https://<Enter the OnPrem Host here>/mailnotificationservice/api/ens

[Pro2type]

If we take some look at the network traffic when pressing "retrieve key" it doesn't give the value of the key from the ENS server.. 

[Noordan]

Hello,

Have you verified that the console server can reach the ENS url?
Have you verified the logfiles on the console server, and API log? what is the error message in the log files?

[Pro2type]

Yes, the console server can reach the ENS server on port 443. 

Seems to be missing some log files on the console server? Or where is the location for the specific log?

[Noordan]

On the console server you should have a log file that is named "WebConsole.log", you should find in the log directory in the installation directory.

And where you find the MEM API log depends on where you have you API configured. You can see that in the UEM console if you navigate to the site URLs in all settings. Then check the REST API. Many environments have the Rest API configured to their DS servers

[Pro2type]

From AW_Mem_Api log:

2022/09/09 08:42:57.878 "Servername" 938adbed-35d9-475d-85fb-c39f3fb52015 [0000000-0000000] (151) Error WanderingWiFi.AirWatch.BusinessImpl.ENS.EnsTenantApiClient.GetEnsApiTokenAsync Exception while retrieving token from the given ENS2 Server Address [https://ServerFQDN/mailnotificationservice/api/ens]. ensHttpPostResponseContent = [] Exception = [System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it IPADDRESS:443
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WanderingWiFi.AirWatch.BusinessImpl.ENS.EnsTenantApiClient.<GetEnsApiTokenAsync>d__7.MoveNext()] Method: WanderingWiFi.AirWatch.BusinessImpl.ENS.EnsTenantApiClient.GetEnsApiTokenAsync; LocationGroupID: 7; UserID: 52; UserName: Administrator;

[Noordan]

I guess you have communication issues from the API Server to ENS server

[Pro2type]

So you are telling me that since my API server is the device servers, then port 443 to the ENS server from device server also need to be open?

[Noordan]

yep I guess so.