Tag Archives: memory

New OEM 12c/13c Agent Install Won’t Keep Running – Dies After Awhile

Many agents - Copyright by Warner Bros.,The Wachowski Brothers used without permission as educational content.
Agents are often hard to kill, usually. Images Copyright by Warner Bros., The Wachowski Brothers; used without permission as educational content.

OEM agents tend to occupy memory based upon how many targets they have to keep track of in a particular host.  At another organization, we tended to spin up VM’s for each instance environment, so at maximum, a particular agent might have a few hundred targets (especially on an e-Business Suite Applications Tier.)  In those circumstances, the default Java memory settings are probably fine.

In this environment, we run our hosts to death, and on this particular proof-of-concept host, we have 43 instances running on it, with variants of 10g, 11g, and 12c databases combined.

We are doing a fresh install of OEM 12.1.0.5.0 for our POC before setting up the 13c production OMS, and after deploying the agent to this particular database host, the agent would startup fine, run for about 20 or so minutes and then abruptly die without warning.

Re-starts fine, passes the usual tests fine (before the 20 minutes or so goes by) and then dies again.

AGENT_INST=/u01/app/oracle/agent12c/agent_inst

cd $AGENT_INST/bin

./emctl status agent

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 12.1.0.5.0
OMS Version            : 12.1.0.5.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent12c/agent_inst
Agent Log Directory    : /u01/app/oracle/agent12c/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/agent12c/core/12.1.0.5.0
Agent Process ID       : 10598
Parent Process ID      : 10499
Agent URL              : https://itsrv33c.mydomain:3872/emd/main/
Local Agent URL in NAT : https://itsrv33c.mydomain:3872/emd/main/
Repository URL         : https://itsrv35g.mydomain:1159/empbs/upload
Started at             : 2016-11-09 09:57:05
Started by user        : oracle
Operating System       : HP-UX version B.11.31 (IA64W)
Last Reload            : (none)
Last successful upload                       : 2016-11-09 10:19:26
Last attempted upload                        : 2016-11-09 10:19:26
Total Megabytes of XML files uploaded so far : 0.2
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 16.92%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-11-09 10:19:53
Last successful heartbeat to OMS             : 2016-11-09 10:19:53
Next scheduled heartbeat to OMS              : 2016-11-09 10:20:53

—————————————————————
Agent is Running and Ready

./emctl pingOMS

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
EMD pingOMS completed successfully

$AGENT_INST/sysman/log/gcagent.log contains

—– Wed Nov  9 09:39:43 2016::26900::Agent Launched with PID 27336 at time Wed
Nov  9 09:39:43 2016 —–
—– Wed Nov  9 09:39:43 2016::27336::Time elapsed between Launch of Watchdog p
rocess and execing EMAgent is 34 secs —–
2016-11-09 09:39:44,287 [1:main] WARN – Missing filename for log handler ‘wsm’
2016-11-09 09:39:44,302 [1:main] WARN – Missing filename for log handler ‘opss’
2016-11-09 09:39:44,305 [1:main] WARN – Missing filename for log handler ‘opsscf
g’
Agent is going down due to an OutOfMemoryError
—– Wed Nov  9 09:40:06 2016::26900::Checking status of EMAgent : 27336 —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent exited at Wed Nov  9 09:40:06 201
6 with return value 57. —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent will be restarted because of an O
ut of Memory Exception. —–
—– Wed Nov  9 09:40:06 2016::26900::writeAbnormalExitTimestampToAgntStmp: exi
tCause=OOM : restartRequired=1 —–
—– Wed Nov  9 09:40:06 2016::26900::Restarting EMAgent. —–

That means, the agent is starting, then stopping, then restarting, then stopping (aka “thrashing”)

Take a look for the running agent daemon at the OS level:

ps -ef | grep agent12c
oracle 26900     1  0 09:39:09 pts/0     0:00 /u01/app/oracle/agent12c/core/12.1.0.5.0/perl/bin/perl /u01/app/oracle/agent12c/core/12.1.0.5.0/bin/emwd.pl agent /u01/app/oracle/…
oracle 27665 26900  0 09:40:12 pts/0     1:01 /u01/app/oracle/agent12c/core/12.1.0.5.0/jdk/bin/IA64W/java -Xmx169M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///de…

Oh – it’s set up for the default of 169MB of RAM.  Check My Oracle Support.

EM 12c: emctl start agent Fails ‘Fatal agent error: State Manager failed at Startup’ ‘restarted because of an Out of Memory Exception’ Reported in emagent.nohup /gcagent.log (Doc ID 1950490.1)

Verify this setting also in the $AGENT_INST/sysman/log/gcagent.log:

—– Wed Nov  9 09:40:06 2016::26900::Auto tuning the agent at time Wed Nov  9
09:40:06 2016 —–
inMemoryLoggingSize=6291456
_SchedulePersistTimer=30
MaxThreads=10
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M
SchedulerRandomSpreadMins=5
UploadMaxNumberXML=5000
UploadMaxMegaBytesXML=50.0
Auto tuning was successful

Well, it’s trying. Per the above Doc ID 1950490.1

Stop the agent.

$AGENT_INST/bin/emctl stop agent

Edit the $AGENT_INST/sysman/config/emd.properties   (this contains the runtime parameters for the agent):

old entry:
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M

new entry:
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M

(You may tune these values up or down according to your environment requirements)

Restart the agent:

$AGENT_INST/bin/emctl stop agent

Agent runs, and keeps running like the E-Bunny.

2014-07 Shinnyo-en Buddhism Podcast – On Loss

2014-07 Shinnyo-en Buddhism Podcast – On Loss

  • Social Media and Physical Loss of Life
  • Energy Transformations
  • The Ever-Turning Wheel of Dharma
  • You Create Permanence

Subscribe to this Podcast (RSS)oriTunes or via Flipboard

While life in the eternal world (whether you call it heaven, mecca, nirvana, spiritual or the any other term) goes on, and on, existence in this world is most definitely of limited term, and in many respects amazingly short in consideration of the billions of years of the planet itself and everything around it in the universe.

When we lose someone near to us, we feel both the pangs of remorse towards missing a person we can no longer talk with, write to, or even choose to ignore. And we feel a sense of powerlessness in not being able to do anything about the person who is gone. It is a soul gone and hopefully never forgotten, but unlike un-friending someone on Facebook, or un-following someone on Twitter, the person’s physical being has left us to become part of the soil once again. Your choice to reconnect with that person’s physical presence is obstructed as though their social life account had been deleted and removed entirely. Building tribute pages and leaving phantom accounts open provides solace to those left behind, but the bits and bytes of storage making up those frozen images are a scarce fraction of the powerful presence of a living, breathing person.

Based upon science, wherein observations relative to energy as being a thing that cannot be destroyed, but only transformed into another form of energy (such as electricity becoming heat or cold, or wind against a mountain becoming a sound, or light becoming stored as potential energy through photosynthesis or photoreaction) – we should think about all that energy that made the person we knew, who they were.  The thoughts, emotions, actions and memories were all forms of energy moving and flowing within a physical shell, processed by a cerebral cortex, and transformed by muscles into words, writing and actions. But when the human body finally stops operating, where does all that energy go?

One physics-based theory is that the memories of others, our own recollections of people gone but not forgotten, are the manifestation of the transition from a physical person, to someone that continues to exist in the synapses of all those who remember them. To think that our own cherished thoughts about someone we loved perpetuate their virtual existence is something worth pondering. Over time, the memories may fade, but each time we bring a quality of their life back into our forefront thoughts, we re-energize the little batteries that keep that light shining. And it’s not just ourselves, but every person who that person touched contribute their own microvolts of remembrance to keep the energy from dissipating as heat loss.

Or metaphysically, we consider the possibility of the cycle of dharma, which also is based on the principle that energy (or spirit) cannot be destroyed, but continues to transform from state to state. In this form, we might consider the concept of karma to be the particular formula that transforms energy from one state of permanence, to another one, whether ascending or descending in form, based upon the qualities of life during that person’s time in a particular realm.  Perhaps the concept of reincarnation is just that same transformation of ethereal energy back into another living being.

So for every candle you light in memory of a lost one, or every prayer you offer in solace, or whenever you simply think about someone you’ve lost, you become the energy source which acts like a little battery keeping that person’s spirit alive, if not in physical form because ashes once again became ashes, and dust became dust. But that person’s soul energy may perpetuate and continue to flow, never lost and never destroyed.  Shinjo and Tomoji Ito (the Shinnyo Parents) used to say, “Live and act to be a person who is missed.”  And maybe that is the definition of what permanence really means.