1903 on-premise with more than 5,000 devices needs memcached server

[RichB2u2]

Because of this article ' CRSVC-6024: On-Premises Environments experience performance issues when Memcached is not configured'  (https://support.workspaceone.com/articles/360022526334) we looked into getting a memcached server setup. The VMWare instructions recommend a Linux server (https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/1902/Memcached_Integration/GUID-AWT-MEMC-INSTALL...). Wikipedia says memcached Windows servers are possible (https://en.wikipedia.org/wiki/Memcached) but VMWare support says Workspace ONE does not support memcached on a Windows server (ticket 1509346). Support said if we had less than 5,000 devices we would be OK. We are stuck on 1902 with over 10,000 devices until they figure out a fix for 1903+ to not require a memcached server or figure out a Windows server version since we no longer support new Linux servers in our environment.

[chengtmskcc]

This could be a problem as most VMware appliance including the one for Unified Access Gateway is a Linux based.

[LukeDC]

I imagine eventually all OnPrem components of WS1 will be linux based. I just setup 2 memcached servers yesterday and implemented them. Super simple. Airwatch has a .bin you can use to install it. The linux servers themselves are barebones and require no special network access other than to and from your app servers (console/DS). So far so good.

[chengtmskcc]

This is the piece I miss as a SaaS customer in my current role. But even in my previous role, it would take us years before we reached even 2000 devices.

[tsmithcoc]

Luke,
Is there any other benefit to standing up the 'Memcache' servers other than resolving the bug in 1903?

Thanks,

[LukeDC]

Travis, memcached will reduce workload on your SQL database in the end. That's the intention. It stores a snapshot of data (24 hours) in the memcached setup to reference instead of pounding SQL all day. It simultaneously writes to SQL and memcached. But does a lookup to memcached first to see if the data exists there. If not found, goes to SQL.

[chengtmskcc]

Luke, I like your straight shooter answer to Travis' question. I would set this up if I were an on-premises customer at least for learning purpose.

[LukeDC]

I had planned on implementing, and this gave me incentive to bump it up on my priority list! Very simple to implement.

[chengtmskcc]

Not unless you have unforeseen obstacles like mine (i.e. infosec, network, etc.)

[LukeDC]

Really it's even simple for that. It requires no outside connections, etc. Just a linux box to run the application. As long as the console and DS can reach the service it's easy enough. Can't see why anyone would object 🙂 but they probably will anyways lol

[chengtmskcc]

You will be shocked at some of the questions I've received from the other groups to explain how data at rest is encrypted after setting up a device passcode.

[LukeDC]

I imagine it's used in the SaaS and DSaaS builds for WS1. That's why it's not affecting any SaaS setups. They didn't account for ONPrem folks having issues implementing solutions like this.

[Stansfield]

There is always the argument for keeping it simple the more moving parts the more places for things to break, if you do not need the load reduction on the sql server it just adds another place to have bugs (in AirWatch or memcached itself) or to have the firewall decide to randomly block access.  It means another server to run and keep track of.

[LukeDC]

The beauty of it, if it goes down it won't impact you in any way. The console and DS just won't attempt to use it and go directly to SQL. Memcached isn't meant for smaller deployments for sure. I'm pushing 20k devices right now and it's helping me out a bit I can sense. If they go out, can't connect etc. Nothing will stop working. As long as SQL is up.

[LukeDC]

According to the release notes for 19.03, CRSVC-6024 has been resolved.

19.3.0.10 Patch Resolved Issues
CRSVC-6024: Enrollment times out on Android and iOS devices.

(https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/1903/rn/VMware-Workspace-ONE-UEM-Release-Notes-1...)

[RichB2u2]

Thanks for pointing that out Luke! I contacted support to confirm the memcached issue was also fixed since this release note only mentions enrollment time out issues as resolved. They checked with an engineer who stated the memcached requirement still exists for 1903 on-premise customers with more than 5,000 devices. They expect this to be fixed in a future unknown version but until then we are stuck on 1902 unless we fire up a Linux server. We may try this with a hosted server from Handy (https://www.handynetworks.com/).

Support updated my ticket tracking this product defect (their words) by creating a new ticket:  https://support.workspaceone.com/tickets/1516374

' 1903 release notes are showing that 19.3.0.10 Patch Resolved Issue CRSVC-6024. However, according to the tracking, it is expected to be fixed in a later version of Workspace ONE UEM.

Please confirm whether this issue has been fixed without needing to implement MemCached.'

I'll update this post when they update the ticket.

[RichB2u2]

This article ' CRSVC-6024: On-Premises Environments experience performance issues when Memcached is not configured'  (https://support.workspaceone.com/articles/360022526334) was updated last night, is listed under ' Known Issues'  and is not shown as ' [Resolved]'  yet.

[LukeDC]

I'm running 20k devices and the performance on the linux server is great. It's barely using much memory. The specs say to have 8GB RAM. I have 2GB allocated and it's staying around 300mb of usage.

As far as the fix, I wish they'd get the story straight. One place it says resolved, the other it doesn't. Super confusing.

[RichB2u2]

I got a call from support this morning and they confirmed that the latest patch to 1903 (19.3.0.10) does fix the memcached issue so it is no longer required for on-premise customers with more than 5,000 devices. They will be updating the documentation sometime since the article in the ' Known Issues'  section (https://support.workspaceone.com/articles/360022526334) still doesn't show as ' [Resolved].'

[chengtmskcc]

Nice job Rich! It must have felt like a roller-coaster.