I installed (upgraded to, actually) SEGv2 (v2.16) in September, and everything has been great since then... until this last Windows patch cycle. Maybe?
I patched earlier this week and since then, the memory usage for java process on the 2012R2 servers has been climbing to 100%. CPU cycles are very low. We're far beyond the recommended RAM settings for the VM based on SEG usage. 24GB, as opposed to the recommended 8GB. I can restart the "VMware AirWatch Secure Email Gateway" service and it goes back to normal RAM usage, but within 18 hours, it's back at 100%.
Has anyone else seen this?
How many devices do you have using email and what extra services like attachment or link re-writing do you have enabled? Also one server or multiple?
4 SEGs, which is a holdover from the SEG Classic days for only 10,000 devices getting email. No extra services at all.
That's what makes it so strange. We're over-architected and under-deviced for running out of memory.
EDIT
For the moment, I'm experimenting with setting the "Advanced" email config settings back to defaults for "Rules Refresh Interval (min)" and "Delta Rules Refresh Interval" to see if that was the issue.
what clustering settings and are the ports open?
They are clustered, and yes the ports are all open between hosts. All 4 nodes show as clustered and sync'd in the Admin GUI on each host.
Like I wrote earlier everything has been fine between early September and Tuesday of this week. Same number of users/devices/SEG hosts/version of UEM/version of SEG.
Only thing that's changed (that I know of) is the Windows patches.
I'd call it a memory leak in java, but that doesn't really jive. I don't think setting the configs (noted in a previous post) back to their defaults has made a difference, but I won't know until tomorrow. If worse comes to worse, I'll open a ticket with VMware as there doesn't seem to be anyone else that is seeing this or maybe hasn't caught it yet.
Did you leave the 'Auto Update' setting within Java as is which will download/install the latest version automatically, or did you disable this setting altogether?
AirWatch products don't always play well with the latest version of Java. So every time you install/upgrade a component, you may be prompted to install/upgrade Java during the process to ensure compatibility.
If the scenario above fits you, one suggestion perhaps is to uninstall Java and re-install the product which again will pull down the applicable version of Java that works best for the version of your product.
Good luck!
Oh don't I know it! Java, AirWatch and I have had it out a time or two over the last 4 years.
But no, that's not it. I'm going to open a ticket, this morning when I checked in there were oodles of alerts for RAM usage on 3 of the 4 SEGs.
In the end, this was apparently how the java code works when installed... To paraphrase:
By default, SEG V2 dynamically configures a certain portion of system RAM as the maximum heap allocation at the time of installation. This is done by updating -Xmx JVM argument value in the configuration file. Note: Above configuration for JVM heap allocation does not block all the allocated memory. Instead, this configuration defines the maximum cap on the java heap. The java process will start with minimum required amount of memory. On need basis, the process may consume more memory from the allocation. Also note that, JVM will periodically run garbage collection to free up the space.
You can override the default behavior by configuring a specific amount of memory for SEG java process. Follow below steps for the same:
1. In SEGDir\config\application.properties file, add below entry
custom.heap.memory.allocation.in.mb=<value in MiB>
Then restart the host and monitor.
Hey Jacques,
What did you end up setting for your custom.heap.memory.allocation.in.mb=<value in MiB>?
Sadly, I don't think I can tell you. I went to look and interestingly that file or setting doesn't seem to exist in SEGv2 2.23 or later.
We're also in the process of moving away from Windows Server-based SEGs to UAG-based SEGs.
However, the JVM has self assigned 6144mb.
Hope that helps somewhat?