Tag Archives: Java

New OEM 12c/13c Agent Install Won’t Keep Running – Dies After Awhile

Many agents - Copyright by Warner Bros.,The Wachowski Brothers used without permission as educational content.
Agents are often hard to kill, usually. Images Copyright by Warner Bros., The Wachowski Brothers; used without permission as educational content.

OEM agents tend to occupy memory based upon how many targets they have to keep track of in a particular host.  At another organization, we tended to spin up VM’s for each instance environment, so at maximum, a particular agent might have a few hundred targets (especially on an e-Business Suite Applications Tier.)  In those circumstances, the default Java memory settings are probably fine.

In this environment, we run our hosts to death, and on this particular proof-of-concept host, we have 43 instances running on it, with variants of 10g, 11g, and 12c databases combined.

We are doing a fresh install of OEM 12.1.0.5.0 for our POC before setting up the 13c production OMS, and after deploying the agent to this particular database host, the agent would startup fine, run for about 20 or so minutes and then abruptly die without warning.

Re-starts fine, passes the usual tests fine (before the 20 minutes or so goes by) and then dies again.

AGENT_INST=/u01/app/oracle/agent12c/agent_inst

cd $AGENT_INST/bin

./emctl status agent

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 12.1.0.5.0
OMS Version            : 12.1.0.5.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent12c/agent_inst
Agent Log Directory    : /u01/app/oracle/agent12c/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/agent12c/core/12.1.0.5.0
Agent Process ID       : 10598
Parent Process ID      : 10499
Agent URL              : https://itsrv33c.mydomain:3872/emd/main/
Local Agent URL in NAT : https://itsrv33c.mydomain:3872/emd/main/
Repository URL         : https://itsrv35g.mydomain:1159/empbs/upload
Started at             : 2016-11-09 09:57:05
Started by user        : oracle
Operating System       : HP-UX version B.11.31 (IA64W)
Last Reload            : (none)
Last successful upload                       : 2016-11-09 10:19:26
Last attempted upload                        : 2016-11-09 10:19:26
Total Megabytes of XML files uploaded so far : 0.2
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 16.92%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-11-09 10:19:53
Last successful heartbeat to OMS             : 2016-11-09 10:19:53
Next scheduled heartbeat to OMS              : 2016-11-09 10:20:53

—————————————————————
Agent is Running and Ready

./emctl pingOMS

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
EMD pingOMS completed successfully

$AGENT_INST/sysman/log/gcagent.log contains

—– Wed Nov  9 09:39:43 2016::26900::Agent Launched with PID 27336 at time Wed
Nov  9 09:39:43 2016 —–
—– Wed Nov  9 09:39:43 2016::27336::Time elapsed between Launch of Watchdog p
rocess and execing EMAgent is 34 secs —–
2016-11-09 09:39:44,287 [1:main] WARN – Missing filename for log handler ‘wsm’
2016-11-09 09:39:44,302 [1:main] WARN – Missing filename for log handler ‘opss’
2016-11-09 09:39:44,305 [1:main] WARN – Missing filename for log handler ‘opsscf
g’
Agent is going down due to an OutOfMemoryError
—– Wed Nov  9 09:40:06 2016::26900::Checking status of EMAgent : 27336 —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent exited at Wed Nov  9 09:40:06 201
6 with return value 57. —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent will be restarted because of an O
ut of Memory Exception. —–
—– Wed Nov  9 09:40:06 2016::26900::writeAbnormalExitTimestampToAgntStmp: exi
tCause=OOM : restartRequired=1 —–
—– Wed Nov  9 09:40:06 2016::26900::Restarting EMAgent. —–

That means, the agent is starting, then stopping, then restarting, then stopping (aka “thrashing”)

Take a look for the running agent daemon at the OS level:

ps -ef | grep agent12c
oracle 26900     1  0 09:39:09 pts/0     0:00 /u01/app/oracle/agent12c/core/12.1.0.5.0/perl/bin/perl /u01/app/oracle/agent12c/core/12.1.0.5.0/bin/emwd.pl agent /u01/app/oracle/…
oracle 27665 26900  0 09:40:12 pts/0     1:01 /u01/app/oracle/agent12c/core/12.1.0.5.0/jdk/bin/IA64W/java -Xmx169M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///de…

Oh – it’s set up for the default of 169MB of RAM.  Check My Oracle Support.

EM 12c: emctl start agent Fails ‘Fatal agent error: State Manager failed at Startup’ ‘restarted because of an Out of Memory Exception’ Reported in emagent.nohup /gcagent.log (Doc ID 1950490.1)

Verify this setting also in the $AGENT_INST/sysman/log/gcagent.log:

—– Wed Nov  9 09:40:06 2016::26900::Auto tuning the agent at time Wed Nov  9
09:40:06 2016 —–
inMemoryLoggingSize=6291456
_SchedulePersistTimer=30
MaxThreads=10
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M
SchedulerRandomSpreadMins=5
UploadMaxNumberXML=5000
UploadMaxMegaBytesXML=50.0
Auto tuning was successful

Well, it’s trying. Per the above Doc ID 1950490.1

Stop the agent.

$AGENT_INST/bin/emctl stop agent

Edit the $AGENT_INST/sysman/config/emd.properties   (this contains the runtime parameters for the agent):

old entry:
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M

new entry:
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M

(You may tune these values up or down according to your environment requirements)

Restart the agent:

$AGENT_INST/bin/emctl stop agent

Agent runs, and keeps running like the E-Bunny.

Advertisements

Java 7 (1.7.0_) and Java 6 on the Same Workstation for R12 e-Business Suite EBS and other Apps

The challenge: We have a number of 3rd party Cloud applications that are still on Java 6, so we have that version (1.6.0_26 specifically) deployed to our workstations. We configured R12 to use 1.6.0_26 on the server side to minimize impact to the users. We are now implementing Informatica Information Lifecycle Management (an archive and purge technology), which has a component (the Enterprise Data Manager – EDM) which is a Java 7-based application used by certain administrative and analyst users. We needed to have those users support both Java 7 and Java 6 on the same workstation for the browsers (Internet Explorer, Firefox, Chrome).

Starting with a Windows 7 (32-bit) workstation with a working Java 1.6.0_26 installed (and 1.6.0_7 disabled).

Used the Java SE 7 SDK from the Oracle Java Archive (staged to the DBA Share /Software/Java because the 7 downloadables will be pulled from the site as of April-2015):

http://www.oracle.com/technetwork/java/javase/archive-139210.html

Chose: Java SE Development Kit 7u25 (includes the JRE runtime edition) – newer versions (post- and including update 40) contain an non-bypassable Security Warning message for using Java 6 applications.

Installed Java 7 using default parameters and completed successfully.

Open the Java Control Panel (available in the Toolbar, as the Java coffee cup icon, if a Java application is already running – e.g. EBS; or via Control Panel -> Java)

Preferred Enabled settings under the [Java] tab -> View…

06a_java_cpanel

06_java_7_and_6_versions_cpanel

De-select the Java Plug-in (next-generation) features (due to co-existence with Java 6) – we also turn off Automatic Java Updates, and default the automatic JNLP MIME/execution association setting to make it easier for users:

07_java_7_adv_tab_disable_next_gen_cpanel

Launch Informatica EDM:

04_java_launch_ilm_edm_ss

Java Update Needed – Check (Do not ask again) -> Select -> Later

01_java_upd_needed_ss

Choose default launcher for EDM’s JNLP (also can be defaulted in the Java Control Panel -> Advanced settings)

05_java_launch_ilm_edm_pick_app_ss

EDM Launches:

02_java_launch_edm_ss

03_java_launch_edm__success_ss

Launching R12 e-Business Suite under Java 6:

Launch Forms Session (Profile -> System used as an example)

Security Warning – due to Java having incomplete Certificate Authority trust to the Aramark Hosts (can be addressed also via Control Panel -> Java -> [Security] -> Manage Certificates -> Import Certificate -> add certificates provided by AUS Server Team (password required).

The Java 7 update 25 warning looks like this (click the Activate Java link; then on the next pop-up click Allow and Remember – to suppress future warning messages):

01b_java_upd_needed_pre40_ss

The post-Java 7 update 40 warning looks like this (select – I accept the risk; click [Run]):

08_java_launch_ebs_sec_warn_ss

Ref: http://java.com/en/download/faq/signed_code.xml

Why don’t I see the option to select Do not show this again for this app in the security dialog for an unsigned application?

Starting with Java 7 Update 40, the option to select Do not show this again for this app is no longer available. Unlike previous versions a user cannot suppress the security dialog for an unsigned application and will have to select the option, I accept the risk and want to run this app, each time to run the unsigned application.

Security Warning – these will come up twice in a row – once when the R12 forms servlet is instantiating and a second time when the actual Forms launch. Once Allow this application to run with the requested version (1.6.0_26) is selected, these warnings should be suppressed in future sessions.

09_java_launch_ebs_sec_warn2_run_6_ss

R12 and EDM simultaneously launched successfully.

11_java_launch_ebs_ilm_success_ss

Oracle OpenWorld 2011 Networking Opportunity Events

Where I can keep track of the special events (note: these are not “parties” as so many people are misled to believe) at OOW:

  • NAmer Presidents Reception, Saturday 5:30p-6:30p, Westin Hotel
  • OOW Opening Reception, Sunday 7p-10p, Yerba Buena Gardens
  • Oracle ACE Dinner, Sunday 7p-9p, <tbd>
  • IOUG Member Gathering, Monday 6p-7:30p, The Chieftain
  • My Oracle Support, Monday 6p-9p, <tbd>
  • interRel Cust. Appreciation, Monday 6:30p-9p, Waterfront Rest (Pier7)
  • Oracle Retail Appr, Monday 7p-10p, Banana Republic Flagship
  • OOW OTN Night, Monday 7:30P-10P, Yerba Buena
  • Cisco Exec Appr, Tuesday 5:30p-8:30p, Pier40 Yacht Cruise
  • Data Intensity, Tuesday 6p-9p, E&O
  • Oracle Press Authors, Tuesday 6p-9p, Roe Restaurant
  • Exxova, Tuesday 7p-10p, Sir Francis Drake Hotel
  • OPN/Titan Awards, Tuesday 7p-9:30p, SF City Hall
  • OAUG Appreciation Event, Weds 6p-8p, Weinstein Gallery
  • OOW Appreciation Event, Weds7:30-10:30p, Treasure Island
  • OOW It’s a Wrap, Thurs 4p-6p, Yerba Buena
Feel free to post your own additions in the comments.