Tag Archives: startup

Existing OEM 12c Agent Fails Startup and Resecure on Hostname Change

Had an issue where the hostname (on Oracle Enterprise Linux 5.9 – OEL 64-bit) happened to have an incorrect hostname and alias when I had already installed the OEM 12c ( Agent. Thus the OMS repository targets were all named incorrectly, even though the Agent was secured and registered. (This was a new database host).

In the $AGENT_HOME/sysman/log/emagent.nohup log was the following:

— EMState agent
—– Sat Mar 1 10:42:22 2014::10437::Auto tuning the agent at time Sat Mar 1
10:42:22 2014 —–
—– Sat Mar 1 10:42:23 2014::10437::Finished auto tuning the agent at time Sat Mar 1 10:42:23 2014 —–
—– Sat Mar 1 10:42:23 2014::10437::Launching the JVM with following options: -Xmx128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops —–
—– Sat Mar 1 10:42:23 2014::10491::Time elapsed between Launch of Watchdog process and execing EMAgent is 2 secs —–
—– Sat Mar 1 10:42:23 2014::10437::Agent Launched with PID 10491 at time Sat Mar 1 10:42:23 2014 —–
2014-03-01 10:42:23,962 [1:main] WARN – Missing filename for log handler ‘wsm’
2014-03-01 10:42:23,971 [1:main] WARN – Missing filename for log handler ‘opss’
2014-03-01 10:42:23,972 [1:main] WARN – Missing filename for log handler ‘opsscfg’
OMS decided to shutdown the agent because of the following reason sent from OMS: EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED
—– Sat Mar 1 10:42:37 2014::10437::Checking status of EMAgent : 10491 —–
—– Sat Mar 1 10:42:37 2014::10437::EMAgent exited at Sat Mar 1 10:42:37 2014 with return value 0. —–
—– Sat Mar 1 10:42:37 2014::10437::EMAgent was shutdown normally. —–

./emctl secure agent

./emctl start agent

Resulted in the same repeating failures.

Removed the Target Host and Agent from OEM 12c OMS

(Target -> Hosts -> (select host) -> [Remove], and then,

Setup -> Manage Cloud Control -> Agents -> (click on Agent:(port)

Agent Menu (upper-left dropdown) -> Target Setup -> Remove Target)

Re-deployed using the faster method of silent Agent deployment (Bobby Curtis has this covered on http://dbasolved.com/2013/04/10/install-oem-agents-silently-in-any-environment )

Everything ready to proceed again.


When Your Oracle DataGuarded Database is Crashed (unintentionall)

It happens:  You have a 4-node DataGuarded Oracle 11gR2 database and your System Administrators need to take the boxes down for maintenance. But they don’t know that it’s protected under DataGuard, with Fast-start failover enabled (which automatically performs (via DG Observer) the switchover from Primary to Standby, switches roles, reconfigures listeners, and tries to keep everything 99.99995% available.)  So they use good old:

sqlplus “/ as sysdba”

SQL> shutdown immediate;

Database closed.

And now you need to start it back up.  But when your admins get to their familiar old prompt it burps out:

SQL> startup

Error: ORA-16825: multiple errors or warnings, including fast-start failover-related errors or warnings, detected for the database

Now there are a number of blogs which will walk you through the tedious steps of recovering this condition manually via DGMGRL and multiple control file and standby redo log restores to all the targets. But it was 1:00A local time and I didn’t want to spend the rest of the night crawling through these trying to get this beast back on its feet.

So, I returned to the never-ending exploratory world of what is contained in Oracle Enterprise Manager 12cR3 (  And (tada sound), OEM can handle it all for you relatively automatically.

To get your Primary DB back on-line (to avoid the complaints from the users who have been disconnected)

DGMGRL> connect SYS/<pwd>@<primaryDGDBtarget>

DGMGRL> disable fast_start failover force;

SQL> startup

Database opened.

Now that the Primary is back open and accessible, return to OEM OMS and visit:

Targets -> Database -> (your Primary DB)

Availability -> Data Guard Administration

There’s a text link at the bottom which reads

Additional Administration:

Verify Configuration

That’s your ticket back to green-arrows and healthy DG in 90% of the situations.  It will perform health-checks throughout your configuration, re-created standby redo logs, re-sychronizing disconnected standby databases, shipping archive logs whereever they are needed, repairing disrupted communications, restoring fast-start failover observers, and whatever else you have.

The other alternative is to simply Remove the existing failed Standby database (select, then [Remove] the existing standby database; then visit its host and delete the existing datafiles, redo, archive and tempfiles (you do not have to touch the TNSNAMES or Listener configurations – that will be re-created for you); then use [Add Standby Database] on the same screen to restore functionality (this is good as an if-all-else-fails, so this method of recovery).

Happy Guarding Data!

Shutdown/Startup vs. Alter Statement – 12c Edition

And most of these new commands are available via a simple right-click in the current release of Oracle’s SQLDeveloper (DBA Menu).

When everyone finally moves to Oracle pluggable databases, one thing to remember is that stopping and starting a database is not going to be as simple as it use to be.  Shutting down and starting up a database is done simply with the following commands:

shutdown normal
shutdown immediate
shutdown abort
startup nomount
startup mount

If we use one of these traditional shutdown methods at the container database (CDB) layer, we will bring down the CDB and the associated PDBs under it. This is something to be aware of when looking to reboot a container databases.  Lets take a look at bringing down a CDB.

[oracle@oel ~]$ sqlplus / as sysdba
SQL*Plus: Release Production on Sun Jul 21 21:42:53 2013
Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real…

View original post 687 more words