1903 on-premise with more than 5,000 devices needs memcached server

[RichB2u2]

Because of this article ' CRSVC-6024: On-Premises Environments experience performance issues when Memcached is not configured'  (https://support.workspaceone.com/articles/360022526334) we looked into getting a memcached server setup. The VMWare instructions recommend a Linux server (https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/1902/Memcached_Integration/GUID-AWT-MEMC-INSTALL...). Wikipedia says memcached Windows servers are possible (https://en.wikipedia.org/wiki/Memcached) but VMWare support says Workspace ONE does not support memcached on a Windows server (ticket 1509346). Support said if we had less than 5,000 devices we would be OK. We are stuck on 1902 with over 10,000 devices until they figure out a fix for 1903+ to not require a memcached server or figure out a Windows server version since we no longer support new Linux servers in our environment.

[HugoA_EOG]

I ran into issue as well. Updated to 1903 and then experienced performance hit. Added MemCached servers. It helped but still had issues. Applied the 1903.0.10 patch which helped with performance but now we are experiencing issues with device scheduler. We are currently waiting on fix for this. Support mentioned it was a known issue and others are experiencing issues as well after the 1903.0.10 patch. I have to visually check scheduler if it's working or not frequently now until it's fixed.

[LukeDC]

Hugo, what do mean by it stopped working? The service quit? Or did you see something else? Curious so i can watch as well on my servers.

[tsmithcoc]

We're experiencing this as well after 1903.0.10.

When we navigate to Settings > Admin > Scheduler we see that the tasks have stopped running at their scheduled times. Restarting the DS service will resolve, but our DS server takes a performance hit with IIS requests. The scheduled tasks will halt every night however it seems.

[HugoA_EOG]

Like Travis mentioned, the scheduled times under Settings > Admin > Scheduler shows the Last Successful Completion times frozen. The scheduler is just stuck and will not process any of the scheduled tasks when it's stuck. Restarting the ' AirWatch Device Scheduler'  service on the Console server fixes the issue.

From the Windows Event log, I show a few errors with a source ' ServiceSIn'  and details indicating it's trying to start/stop the scheduler. The ' AirWatch Device Scheduler'  is still on running status. So the only way to tell is by the error in the event log or from Web Console Settings > Admin > Scheduler and ensure your tasks are running on the frequency that's specified.

[LukeDC]

Ah, ok. I've had that in the past but haven't seen it so far with 1903.0.10 yet. So far so good .

[MarkSchwantje]

I'm not sure the issue with the Scheduler is limited to just 1903. We're running 1810 and recently started experiencing this exact same issue. Support identified that we were impacted by a known issue (CRSVC-4417), which is supposedly fixed in 1904.

[ChristerRingblo]

We are running 18.11 and experience the same problem with needed restart of AirWatch Device Scheduler, when the Last Successful Completion times have frozen.

[chengtmskcc]

Christer, do you know if AirWatch Device Scheduler could affect VPP apps being pushed to devices upon enrollment? I had this issue since Wednesday afternoon (SaaS) and we were told a service stopped and caused the issue.

[ChristerRingblo]

The description of the service:
Device Scheduler Service reads the schedule settings from the database and writes it to the two queues APNSOutbound and C2DMOutbound.

I assume all tasks in the Scheduler relies on this service. Definitely the app-distribution.

[LukeDC]

The largest flaw with the scheduler service is that it is not clustered at all. Only one server at a time can even run the service, it will fail to start on any oher server. You'd think this would have been addressed after all this time.

[Stansfield]

I think that would go in the same basket as the ability to cancel things like wipe commands it would mean they would need to be exactly sure of what their commands are and how they act and have all the different schedulers communicate to not send duplicate commands.

[chengtmskcc]

It's the 2nd time where the Scheduler service was running within our dedicated SaaS, but the DB job for VPP push wasn't and VMware did NOT know about it. We are escalating the case as it happened to our production environment.

Does anyone know or has anyone see the jobs associated with the Scheduler service within the SQL server?

[LukeDC]

I've found my scheduler craps out if the SQL connection fails for whatever reason. The scheduler loses it's mind and won't restart after the connection to SQL is lost. Granted my SQL server never actually went down, there may have been some network errors etc that can occur.

[MarkSchwantje]

It can happen if there are any network connectivity issues with your AWCM server. Per support:

' Certain console commands run through the AWCM to the database, mostly device-related commands.  If you are seeing AWCM connection issues then some Console commands will not make it to the DB.  These failed command attempts are causing the DB Quartz errors and creating the known issue in question [scheduler service stopping and not restarting].'

[LukeDC]

My logs specifically show a SQL connection loss then a panic and a failure to start again. I find the AWCM comment a bit suspicious. I'd like more specifics on what commands they are talking about.

[MarkSchwantje]

I can tell you for sure that we ran into this situation when we had connectivity issues with our DS/AWCM servers. Connectivity from the Console server to the SQL server was never an issue. However, your exact situation was identified as a known issue in CRSVC-4417.

[RichB2u2]

So it sounds like there may be a Device Scheduler issue (CRSVC-4417) when upgrading to 1903 that is fixed in 1904, but that isn't available to on-premise, so I am staying at 1902.0.18 for now! We have not experienced any problems with Device Scheduler service on 1902 yet. I don't see a need to install the latest 1902 patch to version 1902.0.26 either so we are on hold for now...

[Stansfield]

The whole upgrade to an unavailable version to fix your issue really has made their decision on limiting on premise releases a mess, they do not seem to be back porting the fixes very well either.  I guess that is one way to force SaaS hold the system fixes hostage so if you need your console to work in the next three months you can go to SaaS because support for On Premise has been launched into the sun.  They recently told us on one of our needed fixes to upgrade to a version unavailable for On premise and then closed the ticket as resolved saying that the fact we could not get that version ever was not something they could comment on.

[LukeDC]

where is the info on CRSVC-4417? I'm not seeing it in the normal spots.

[chengtmskcc]

I'm interested in this as well. My support engineer seems to be confirming this issue also for SaaS customer which is fixed in 1904.

Only if upgrading from one version to the next is as easy as clicking ' Next' ...