VMware Workspace ONE Community
JacquesPerrolle
Enthusiast
Enthusiast

Memory Spikes for Java process on SEGv2s

I installed (upgraded to, actually) SEGv2 (v2.16) in September, and everything has been great since then... until this last Windows patch cycle. Maybe?

I patched earlier this week and since then, the memory usage for java process on the 2012R2 servers has been climbing to 100%. CPU cycles are very low. We're far beyond the recommended RAM settings for the VM based on SEG usage. 24GB, as opposed to the recommended 8GB. I can restart the "VMware AirWatch Secure Email Gateway" service and it goes back to normal RAM usage, but within 18 hours, it's back at 100%.

Has anyone else seen this?

Labels (1)
0 Kudos
9 Replies
Stansfield
Enthusiast
Enthusiast

How many devices do you have using email and what extra services like attachment or link re-writing do you have enabled?  Also one server or multiple?

0 Kudos
JacquesPerrolle
Enthusiast
Enthusiast

4 SEGs, which is a holdover from the SEG Classic days for only 10,000 devices getting email. No extra services at all.

That's what makes it so strange. We're over-architected and under-deviced for running out of memory.

EDIT

For the moment, I'm experimenting with setting the "Advanced" email config settings back to defaults for "Rules Refresh Interval (min)" and "Delta Rules Refresh Interval" to see if that was the issue.

0 Kudos
Stansfield
Enthusiast
Enthusiast

what clustering settings and are the ports open?

0 Kudos
JacquesPerrolle
Enthusiast
Enthusiast

They are clustered, and yes the ports are all open between hosts. All 4 nodes show as clustered and sync'd in the Admin GUI on each host.

Like I wrote earlier everything has been fine between early September and Tuesday of this week. Same number of users/devices/SEG hosts/version of UEM/version of SEG.

Only thing that's changed (that I know of) is the Windows patches.

I'd call it a memory leak in java, but that doesn't really jive. I don't think setting the configs (noted in a previous post) back to their defaults has made a difference, but I won't know until tomorrow. If worse comes to worse, I'll open a ticket with VMware as there doesn't seem to be anyone else that is seeing this or maybe hasn't caught it yet.

0 Kudos
chengtmskcc
Expert
Expert

Did you leave the 'Auto Update' setting within Java as is which will download/install the latest version automatically, or did you disable this setting altogether?

AirWatch products don't always play well with the latest version of Java. So every time you install/upgrade a component, you may be prompted to install/upgrade Java during the process to ensure compatibility.

If the scenario above fits you, one suggestion perhaps is to uninstall Java and re-install the product which again will pull down the applicable version of Java that works best for the version of your product.

Good luck!

0 Kudos
JacquesPerrolle
Enthusiast
Enthusiast

Oh don't I know it!  Java, AirWatch and I have had it out a time or two over the last 4 years. Smiley Happy

But no, that's not it. I'm going to open a ticket, this morning when I checked in there were oodles of alerts for RAM usage on 3 of the 4 SEGs.

0 Kudos
JacquesPerrolle
Enthusiast
Enthusiast

In the end, this was apparently how the java code works when installed... To paraphrase:

By default, SEG V2 dynamically configures a certain portion of system RAM as the maximum heap allocation at the time of installation. This is done by updating -Xmx JVM argument value in the configuration file.  Note: Above configuration for JVM heap allocation does not block all the allocated memory. Instead, this configuration defines the maximum cap on the java heap. The java process will start with minimum required amount of memory. On need basis, the process may consume more memory from the allocation. Also note that, JVM will periodically run garbage collection to free up the space.

You can override the default behavior by configuring a specific amount of memory for SEG java process. Follow below steps for the same:
1. In SEGDir\config\
application.properties file, add below entry

custom.heap.memory.allocation.in.mb=<value in MiB>

Then restart the host and monitor.

0 Kudos
teckler
Contributor
Contributor

Hey Jacques,
What did you end up setting for your custom.heap.memory.allocation.in.mb=<value in MiB>?

0 Kudos
JacquesPerrolle
Enthusiast
Enthusiast

Sadly, I don't think I can tell you. I went to look and interestingly that file or setting doesn't seem to exist in SEGv2 2.23 or later. 
We're also in the process of moving away from Windows Server-based SEGs to UAG-based SEGs.

However, the JVM has self assigned 6144mb.

Hope that helps somewhat?

Tags (2)
0 Kudos