VMware Cloud Community
St0ked56
Contributor
Contributor

Out of Memory

Hi,

I am using the vCO appliance and recieve these errors weekly, which cause the services to crash.

warn:Aug 6 15:17:53 vco kernel: [ 6521.377208] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:17:53 vco kernel: [ 6521.377211] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:17:53 vco kernel: [ 6521.419353] Out of memory: kill process 3283 (orchestrator.sh) score 662684 or a child
warn:Aug 6 15:20:54 vco kernel: [ 6703.394836] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:54 vco kernel: [ 6703.394840] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:54 vco kernel: [ 6703.437666] Out of memory: kill process 3132 (postmaster) score 82657 or a chil d
warn:Aug 6 15:20:54 vco kernel: [ 6703.441168] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:54 vco kernel: [ 6703.441171] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:54 vco kernel: [ 6703.483191] Out of memory: kill process 3132 (postmaster) score 74276 or a chil d
warn:Aug 6 15:20:54 vco kernel: [ 6703.485172] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:54 vco kernel: [ 6703.485175] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:55 vco kernel: [ 6703.527235] Out of memory: kill process 3132 (postmaster) score 61100 or a chil d
warn:Aug 6 15:20:55 vco kernel: [ 6703.596074] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:55 vco kernel: [ 6703.596084] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:55 vco kernel: [ 6703.684301] Out of memory: kill process 3132 (postmaster) score 47924 or a chil d
warn:Aug 6 15:20:55 vco kernel: [ 6703.717055] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:55 vco kernel: [ 6703.717065] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:55 vco kernel: [ 6703.806669] Out of memory: kill process 3132 (postmaster) score 34731 or a chil d
warn:Aug 6 15:20:55 vco kernel: [ 6703.816188] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:55 vco kernel: [ 6703.816198] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:56 vco kernel: [ 6703.904465] Out of memory: kill process 2906 (slapd) score 26592 or a child
warn:Aug 6 15:20:56 vco kernel: [ 6703.955858] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:56 vco kernel: [ 6703.955868] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:56 vco kernel: [ 6704.043425] Out of memory: kill process 3132 (postmaster) score 26351 or a chil d
warn:Aug 6 15:20:56 vco kernel: [ 6704.869434] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:56 vco kernel: [ 6704.869438] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:56 vco kernel: [ 6704.912952] Out of memory: kill process 3199 (va-proxy) score 16102 or a child
warn:Aug 6 15:20:56 vco kernel: [ 6704.913478] [<ffffffff810b85a0>] __out_of_memory+0x50/0xa0
warn:Aug 6 15:20:56 vco kernel: [ 6704.913607] [<ffffffff810b8788>] out_of_memory+0x198/0x210
warn:Aug 6 15:20:56 vco kernel: [ 6705.005493] Out of memory: kill process 3240 (sendmail) score 15169 or a child

The system is setup with 12GB of RAM and the java wrapper is set to 10240MB.  Not sure where to look for this memory leak and what is consuming all the resources.  Any help would be greatly appreciated. 

Reply
0 Kudos
6 Replies
St0ked56
Contributor
Contributor

Also looked at this link http://www.vmware.com/support/orchestrator/doc/vcenter-server-plugin-5011-release-notes.htmland looked into the below workaround.

For line 4 mine is set to 256m

and for line 5 mine is set to 512

These were set before looking at this file.  I didn't not lower them because i was unsure what that would do.

The vCenter Server 5.0.1.1 plug-in might run out of memory
The vCenter Server 5.0 plug-in might run out of memory with an OutOfMemory error message when you configure it to work with vCenter Orchestrator 4.1.

Workaround: Modify the jetty-service.conf file on the machine on which the Orchestrator server and the vCenter Server 5.0 plug-in are installed.

  1. Log in as an administrator to the machine on which the Orchestrator server and the vCenter Server 5.0 plug-in are installed.
  2. Locate the jetty-service.conf file.
    The default location is: C:\Program Files\VMware\Infrastructure\Orchestrator\configuration\jetty\jetty-service.conf.
  3. Open the file in a text editor.
  4. Add the following line:
    wrapper.java.additional.7=-XX:MaxPermSize=128m
  5. Change wrapper.java.maxmemory=64 to wrapper.java.maxmemory=256.
  6. Save and close the fil
Reply
0 Kudos
dvatov
VMware Employee
VMware Employee

These settings are for the configurator service. From the log you posted it looks like the orchestrator server is running out of memory. Look at its startup script what are its current memory settings. I think by default it was 2GB.

By the way - it is actually OS which is running out of memory and not the orchestrator. There are other processes killed also. Use top or whatever tool you have to find out where the memory was consumed. If this is VM with vm tools installed you may also consider the memory balooning.

Reply
0 Kudos
St0ked56
Contributor
Contributor

The system usually crashes after a Nessus scan is ran on the system.  I just watched it using top while the scan was in process.  Before the scan it was using about 2GB of RAM.  After the scan it had about 64MB of RAM free.  The process that consumed all the resources was va-proxy.  After about 3-5 mins after the scan was over all the resources were released.  But before the resources were released vcod service stopped and and kicked me out of the vCO client.  Any idea of how to prevent this from happen while being able to scan the sytem still?

Reply
0 Kudos
mmitevski
VMware Employee
VMware Employee

What is the Appliance version, that you use?

Reply
0 Kudos
St0ked56
Contributor
Contributor

I am using 4.2.1.0 Build 612841

Reply
0 Kudos
dimitrovg
VMware Employee
VMware Employee

Probably the Nessus scan uploads large data or does other similar things. To prevent killing of processes you may need to do some or all of this steps:

  • Increase the memory of the VA or to decrease the java memory (you said that 10GB are configured for the java wrapper - is this the configurator or the vco server? Where did you configured this?) If you configured 10GB for the java and 12GB total then maybe 2GB is not enough for all other processes in the OS.
  • You may try to check if there is memory ballooning in the VM (this happens when the ESXi host is short in memory). If this is the case then try to reserve memory for this VM to prevent the ballooning.
  • Create big enough swap space. By some advices under linux you may need as big swap space as the total momory available (or even 50% bigger). Note that this will help only if you have some temporary peaks in the memory usage (like during the Nessus scan) if the memory is loaded all the time this will not help.
  • Last resort (if the others did not help): If you need just to finish the nessus scan then you can kill the process va-proxy and check if the scan will finish.

Hope this helps.

BTW 10GB for the java seems reasonly high, why did you reseved so much memory for the java?

Regards,

George

Message was edited by: dimitrovg, hit submit by mistake

Reply
0 Kudos