Lost connection to devices and can't enroll devices

[SebastianRe]

Yesterday all our servers went down but we were up and running again after 1h.
Today when I log into the Airwatch portal I can see that non of the devices have checked in for 24h.
I am also unable to enroll a DEP device or manually through the HUB app.
I have made sure Airwatch SQL, Console and Device server are online.
Does someone know what might be wrong?

[jbarzFunk32]

This sounds like an issue with your Device Services server(s). I'd get onto that and make sure all AirWatch services are running and that the World Wide Web (IIS) service is running as well. It wouldn't hurt to run an iisreset command as well.

[SebastianRe]

Hello and thank you for your respons. I checked the World Wide Web Publishing service on the Device server and Console server and it was running on both of them. I also tried restarting the service and ran a iisreset /restart command but still the same issue. No devices are checking in, and when I try enroll a DEP device I get the message ' Network connection was lost' . And same message when I try enroll through the HUB app and type in my servername.
There is also a IIS Admin Service running on both servers.

[jbarzFunk32]

I had a similar issue when I upgraded to 1810 earlier this week and restarting the IIS service fixed it for me, so I'm sorry I'm not much more help. Probably wouldn't hurt to do a full reboot of the servers and make sure all the services come back up properly again. The day after I did the upgrade, the AirWatch Messaging Service stopped on both of my DS servers, stopping users to the ability to check in devices. Make sure you Apple APNs cert is not expired. Not sure if you have any Android devices in your environment. I'll keep thinking of other ideas, but it's definitely an issue with devices not being able to talk to the DS servers and vice versa.

[SebastianRe]

I tried rebooting them but the issue persist.
The APN is still active for another 10 months.
It must be something on the Device server as it was working fine 2 days ago but when the servers went down something must have happened after they went up again.
Yes please get back to me if you have any other ideas, thank you.

[UiliamFoschiera]

Hi Sebastian.... Can you enable DS debug logs into console, to identify in what step DS shows the issue?

[SebastianRe]

Hello Uiliam!
I dont really know how to enable DS debug logs but I found 2 folders under Airwatch/Logs on the DS server. One log named DeviceServicesLogs.txt and DeviceManagement.txt

When I open them they only show logs up to 2018-11-28 at around 14:00 and that's about the time our servers crashed and there are no logs after that.

If I check the logs in the Console at the time the servers went down it says:

2018-11-28 15:09:21 SYDSRVAWCS01 daeaad63-3513-4e3b-aa01-8537d0e4da51 (1) Info **************************************************************** WanderingWiFi.AirWatch.Console.Web.Global

2018-11-28 15:09:21 SYDSRVAWCS01 daeaad63-3513-4e3b-aa01-8537d0e4da51 (1) Info * Starting WanderingWiFi.AirWatch.Console.Web 9.3.0.0 WanderingWiFi.AirWatch.Console.Web.Global

2018-11-28 15:17:47 SYDSRVAWCS01 50c68b54-b5b6-4400-92f4-1a81b3003fc8 (7) Error Object reference not set to an instance of an object. WanderingWiFi.AirWatch.Console.Web.Controllers.LogoutController

2018-11-28 15:17:48 SYDSRVAWCS01 3ffea47c-6c4e-4bb3-a0c3-7a7e71820255 (8) Error Object reference not set to an instance of an object. WanderingWiFi.AirWatch.Console.Web.Controllers.LogoutController

2018-11-28 15:38:17 SYDSRVAWCS01 88fa515b-08a2-49c8-8347-7f7d12518d1e (8) Info Stopping WanderingWiFi.AirWatch.Console.Web 9.3.0.0 WanderingWiFi.AirWatch.Console.Web.Global

2018-11-29 06:58:11 SYDSRVAWCS01 88773f0f-180a-432e-998f-f375a8302d49 (1) Info ***************************************************************

[UiliamFoschiera]

Hi Sebastian,
With the administrator account, go to Groups and Settings -> All Settings -> Admin -> Diagnostics -> Logging, and select ' Enabled'  to Device Services. It will enable debug mode logging, increasing details in same logs that you found and post here (DeviceServicesLogs.txt). Maybe with it, we will can see in what point/step the issue is occuring.
Regards

[SebastianRe]

Hello, I do not have the option ' Logging'  under ' Diagnostics' . All I can see is ' System Health' .
If I go under Diagnostics > Troubleshooting I have an option called ' Web Console Log' .
If I check the WebLogFile.txt I have this line spammed
2018-12-03 10:48:49 SYDSRVAWCS01 8d0180d3-42c2-48b6-aca4-7a0e0dbd8a1c (18) Warn SystemCodeCategoryTreeNode references a nonexistent parent: Id=183,ParentId=180 0c2d8df0-8b1b-44f6-bee5-af0c7d4c14d7 WanderingWiFi.AirWatch.SystemCodeService.AwTreeBuilder

This one is still logging up to todays date which the DeviceServicesLogs.txt is not, it stopped logging 11/28 and this was the last line:
2018/11/28 14:12:21.885 SYDSRVAWDS01 5dd9055a-d879-4774-bf08-f5b276a835ff [0000000-0000000]   (34)  Error WanderingWiFi.AirWatch.DeviceServices.Handlers.SecureChannelEndPointHandler VerifySecureChannelClientSignerCertificate is false, validating client signer certificate as part of signature verification  

[UiliamFoschiera]

Hi,
About logging, it will be shown only if you log into console with THE Administrator account (not a user with administrator credentials).
About this log, maybe this link (https://support.workspaceone.com/articles/360010357814) can be applicable/helpful.
Regards

[SebastianRe]

I have enabled logging on Device Services now under Diagnostics > Logging. But the problem is still that ' DeviceServicesLogs.txt'  stopped logging 28/11 at 14:00 when the servers crashed. So even though I have enabled the DSlogging it's not updating the .txt file.
I also tried doing a restoration on the Device Server from the day before the server went down and it was working but the issue is still the same with the older backup.
Im thinking it's maybe some communication error between the devices and the mdm server. But I tried pinging the mdm server and it's external IP adress from an external network and it works.

[SebastianRe]

What ports are used between the DS, CS and SQL server and devices?

[SebastianRe]

I think I've found the problem now. From the DS server it is not possible to reach my airwatch domain through 443 https. When I go into the IIS manager and try Browse *:443 (https) I only get ' page can not be displayed' . If I try with 80 (http) it's working, I reach my airwatch domain. If I select the API or DeviceServices in IIS manager and try 443 it's the same problem.
Under site bindings I have
http 80 *
https 443 *
Someone have any suggestions how to fix 443 (https)?

[UiliamFoschiera · Accepted solution]

Hi Sebastian,

How is the certificate under https 443 binding? Is valid?

And... the 443 port shows as ' listen'  on server?

IIS logs shows some relevant info about this?

[jbarzFunk32]

Definitely could be the certificate. Otherwise, not sure how your environment is setup, but in my organization I don't manage the load-balancer or firewall configuration. If you don't either, you'll probably want to present this to your data network team to see what changes were made recently that would cause 443 to be blocked.

[SebastianRe]

It was a certificate problem with the IIS. The certificate was there and active but for some reason not working so I removed it from the server and reuploded it and it worked.

[LukeDC]

FWIW I've had windows patches remove/corrupt the IIS SSL Binding in the past. Be aware anytime you run Windows patches, things like that can happen. The binding showed in place but was not working in my case several times after Windows patching.

[DimitrijPrudkij]

Hello Sebastian,

which certificate you delete ? On DS or on CS  ? We have similar issue. We can only enroll user which are member in activesync group.

[MikeAcevedo]

We are experiencing something similar but can only enroll in DEP, not through web enrollment? I tried both an AD account and Basic account. We are receiving a blank screen and doesn't redirect to download the MDM profile.

[ElizabethB]

We just experienced the same issue with the Windows updates removing/corrupting the binding from our 443 cert.  It showed that it was there and active, but after importing it again our issue cleared up.  Thanks to those that contributed that information to this post.