During an online patching session with ADOP, when the prepare phase checks the data dictionary integrity, you receive the following error:
R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent:
Verifying data dictionary.
[UNEXPECTED]Data dictionary corrupted:
[UNEXPECTED]Data dictionary corruption – missing parent
1416131 ORA$BASE APPS BPA_GUIDED_PICKING SYNONYM
1416121 ORA$BASE APPS CO_CLIPPERSHIP_INTERFACE SYNONYM
1416132 ORA$BASE APPS BPA_PRINT_GUIDED_PICK_LABELS SYNONYM
1416128 ORA$BASE APPS GEM_ESIG_INQUIRY_CORE SYNONYM
[UNEXPECTED]Data dictionary corruption detected. Provide details to
[UNEXPECTED]Oracle Support and ask for a bug to be opened against the
[UNEXPECTED]Online Patching component of Oracle Application Install.
[STATEMENT] Please run adopscanlog utility, using the command
“adopscanlog -latest=yes”
Issue:
The corruption is detected with the $AD_TOP/sql/ADZDDBCC.sql scriptis executed just prior to setting up the ADOP session. This is also a mandatory check performed during pre-upgrade readiness checking for R12.2 upgrades.
Solution:
There is a rudimentary, but effective script supplied in $AD_TOP/patch/115/sql named adzddmpfix.sql (and a number of other adzd___ named scripts that address other data dictionary issues, such as the common TimeZone timestamp issues (Doc ID 2017686.1))
This script is run as SYS and requires all middle-tier services to be shutdown because it’s rebuilding all of the APPS synonyms for objects and recompiling afterwards.
sqlplus “/ as sysdba” @$AD_TOP/patch/115/sql/adzddmpfix.sql
REM $Header: adzddmpfix.sql 120.0.12020000.3 2015/09/14 10:10:32 sstomar noship$
REM dbdrv: none
REM +======================================================================+
REM | Copyright (c) 2005, 2015 Oracle and/or its affiliates. |
REM | All rights reserved. |
REM | Version 12.0.0 |
REM +======================================================================+
REM | FILENAME
REM | adzddmpfix.sql
REM |
REM | DESCRIPTION
REM | Warning: this script is an experimental workaround to the Oracle
REM | Database data Dictionary “missing parent” corruption described in
REM | Bug 20627866. This bug currently has no other known fix, but this
REM | script will repair the problem sufficiently to allow adop to continue
REM | executing.
REM |
REM | Warning: You must be connected as SYSDBA with application services
REM | shutdown to run this script
REM | Usage:
REM | sqlplus / as sysdba
REM | @adzddmpfix.sql
REM |
REM |
REM +=======================================================================+
Sample output:
SQL> @adzddmpfix.sql
“—- Fixing Data Dictionary Corruptions (missing parent) —-”
4 rows deleted.
Commit complete.
System altered.
“—- Compiling invalids —-”
Don’t forget to re-run the $AD_TOP/sql/ADZDDBCC.sql script to identify whether the data dictionary corruption is still present afterwards.
alter session set current_schema=APPS;
exec AD_ZD_PREP.ENABLE_CUSTOM_USER(‘XXCUSTOMSCHEMA’);
/* Create your SELECT grants to the editioned objects as APPS user */
grant select on ap.ap_suppliers# to XXCUSTOMSCHEMA;
/* OR if you’re running newer than 12.2.5…the editioning security will cause hundreds of invalid editioned object stubs that need re-compilation unless you instead use */
/* Create your custom view referencing the APPS synonym – which points to the editioned object */
create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM apps.ap_suppliers ;
This is the same process when creating procedures and functions with the usual caveat that unless it’s a 12c database, you cannot create materialized views on the editioned objects.
For the 11gR2 EBS users, you would stick with the existing non-editioned table object reference:
CREATE MATERIALIZED VIEW APPS.XXAP_SUPPLIER_NEW_MV
(
vendor_id,
vendor_name
)
AS
SELECT aps.vendor_id, aps.vendor_name
FROM ap.ap_suppliers aps;
/* The owning schema for the materialized view must have CREATE TABLE, VIEW AND INDEX privileges on the destination tablespace */
Oracle…sometimes blinding in complexity, and a foreshadowing of what happens when a system becomes so complex that it becomes fragile.
This is what the screen session looks like:
adnodemgrctl.sh: check the logfile /u01/oradev/fs1/inst/apps/DEV2_itsrv33m/logs/appl/admin/log/adnodemgrctl.txt for more information … [UNEXPECTED]fs_clone has failed.
[STATEMENT] Please run adopscanlog utility, using the command
“adopscanlog -latest=yes”
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 2 (Fail)
Here is what it looks like in the adop.log
[EVENT] Performing CLONE steps.
[PROCEDURE] [START 2016/11/08 08:20:19] CLONE Patch File System from Run File System
[PROCEDURE] Calling: /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
[EVENT] Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m [UNEXPECTED]Error occurred running “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”
[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”.
[EVENT] Releasing managed server ports.
[PROCEDURE] [START 2016/11/08 08:30:55] Releasing ports
[STATEMENT] Calling task: “Releasing ports on itsrv33m”; script: “/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl”; args: ” -option=close -server=itsrv33m.mwd.h2o -ports=5570,7015,7615,7415,7215,7216,7217,7815,6114,6214,10013 -log=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log”
[PROCEDURE] [START 2016/11/08 08:30:56] Releasing ports on itsrv33m
[PROCEDURE] Calling: /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl
[EVENT] Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log
[PROCEDURE] [END 2016/11/08 08:30:57] Releasing ports on itsrv33m
[PROCEDURE] [END 2016/11/08 08:30:57] Releasing ports
How cryptic! Looks complicated. This system happens to have AD/TXK.Delta.8, so the logs related to adop are all being centralized under the session # (in this case “25”.)
Creating the directory: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply
Copying the directory
———————
SOURCE : /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/log/clone
TARGET : /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply
/u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply.
*******FATAL ERROR*******
PROGRAM : (/u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Tue Nov 8 08:30:55 2016
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.
(One innocuous, but annoying file, owned by someone other than applmgr, and happened to be 600 (user read/write only):)
Done copying /u01/oradev/fs2/EBSapps/appl/wms to /u01/oradev/fs1/EBSapps/appl/wms/
Copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
WARNING:
.end std out.
cp: cannot open /u01/oradev/fs2/EBSapps/appl/mwd/12.0.0/sql/MWDPOLPO.xsl: Permission denied
.end err out.
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/mwd failed
ERROR: Copying failed, so stopping all the parallel threads
Done copying /u01/oradev/fs2/EBSapps/appl/okc to /u01/oradev/fs1/EBSapps/appl/okc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/okc failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/pqh/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh failed
Done copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/gmf failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/ies/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies failed
Done copying /u01/oradev/fs2/EBSapps/appl/msc to /u01/oradev/fs1/EBSapps/appl/msc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/msc failed
Done copying /u01/oradev/fs2/EBSapps/appl/ams to /u01/oradev/fs1/EBSapps/appl/ams/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/ams failed
Done copying /u01/oradev/fs2/EBSapps/appl/cs to /u01/oradev/fs1/EBSapps/appl/cs/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/cs failed
START: Inside exitClone….
Updating status INCOMPLETE for ApplyApplTop
START: Updating status INCOMPLETE for action ApplyApplTop
END: Updated status INCOMPLETE for action ApplyApplTop
One file. A single file. Could have been a hidden file. Could have been anywhere in the entire AppsTier filesystem. This one file stopped ADOP cold from performing. This happened to be phase=fs_clone, but could have been a patching session, or a post-clone config (adcfgclone.pl) session.
Remove the file, and:
The fs_clone phase completed successfully.
adop exiting with status = 0 (Success)
Have you met this error on db_exception_log? ~ This is how you can fix it!:
29273 : ORA-29273: HTTP request failed
ORA-06512: at “”SYS.UTL_HTTP””, line 1817
ORA-24247: network access denied by access control list (ACL)”
SYS_GRANTS.sql performs the following:
– Adds EXECUTE privileges to access DBMS_CRYPTO (UPGRADE_PASSWORDS): Provides the highest level of user password encryption.
– Adds EXECUTE privileges to access DBMS_LOCK: Provides as SLEEP operation for improved concurrency.
– Adds EXECUTE privileges to access V_$PARAMETER so that Oracle Demantra can better adapt to your database configuration.
– (10g only) Adds GRANT privileges to access the package UTL_HTTP, which enables Oracle Demantra to send notification messages to the application server and engine.
– (11g only) Adds an ACL to enable HTTP communications for Oracle Demantra to send notification messages to the application server and engine.
Find it in Demantra_Folder\Demand Planner\Database Objects\Oracle Server\admin
or directly in the root of the Demantra installation package.
Please also verify you have the correct data in
SQL> select * from sys_params where lower(pname) like ‘%url%’;
You need to run this script manually after installing or upgrading Demantra only if you did not specify a database user with full SYSDBA privileges when running the Installer.
1. !! Login AS sys, WITH sysdba privileges:
2. C:\> cd DEMANTRA_INSTALL_DIRECTORY
3. C:\DEMANTRA_INSTALL_DIRECTORY> sqlplus SYS@SERVER as sysdba@sys_grants.sql DM_SCHEMA_USER ACL_for_WebServerURL ACL_for_EngineServerURL
Example execution:
C:\DEMANTRA_INSTALL_DIRECTORY> sqlplus SYS@SERVER as sysdba@sys_grants.sql DM_SCHEMA_USER ‘/sys/acls/demantra.xml’ ‘ /sys/acls/demantra.xml’
OR by Default: if you are not sure, they will be created if they do not exist, or updated in /sys/acls/demantra.xml:
— select * from sys_params where lower(pname) like ‘%url%’;
Check: — SQL> select * from DBA_tab_privs where grantee = ‘DEMANTRA’ ; — (dem schema) — Be sure from system user run if needed entry for APPS user against demantra.xml in dba_network_acl_privileges table: BEGIN DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE ( ‘/sys/acls/demantra.xml’, ‘APPS’,TRUE,’connect’); COMMIT; END; —Also check entry for UTL_HTTP table for the query – select * from DBA_tab_privs where grantee = ‘DEMANTRA’ ;
Check
For failed login problem to Collaborator Workbench, check if sys_grants is needed to be applied again:
1. SQL> select distinct encryption_type from user_id; — This should return a single value like “SHA-1”
2. SQL> select encryption.get_highest_desc() from dual; — This should return the same value returned in step (1).
If the result of (1) is “SHA-1” but the return of (2) is something else, then you need to fix your sys_grants.
References:
Document: 454369.1 Roles and Privileges that are granted to the Demantra database during the install process Document: 730883.1 Additional Database Privilege needed for Demantra Schema when Running on Oracle 11g Database Document: 1064995.1 Additional Configurations needed when Running Demantra on Oracle 11g Databases Oracle Demantra Installation Guide for Release 7.3 Document: 825082.1
An ancient hieroglyph of a Page Not Found error – 404
One day, there will no longer be any unknown sites, addresses, or anything of the kind thanks to modern technological advances in how we look things up. That little Address bar became the gateway to all sorts of inventive ideas of how to make our lives easier, simpler, more useful. Instead of remembering arcane website addresses, or before that, actual IP addresses, we’d have browser plugin helpers for our search engines to convert any text in that address bar to searchable terms for our favorite Google, Bing, Yahoo, Ask, DuckDuckGo, whatever engine to look it up for us.
BuyThisDomainAndMakeMeRich.com
Enterprising (sort of) domain campers tried to out-think our mistakes and pre-register every variant of a misspelled or syntactically-incorrect website address out there and re-direct them to their own domain-for-sale pages to generate income.
Mozilla thought ahead and offered it’s version of simpler language “friendly” 404 pages to describe in regular words what happened.
There is nothing to see here. Move along.
Easier, and easier. Less to remember every day. Your mind is becoming a blank page, open to whatever creative thought you can imagine, unhampered by useless memorized facts and figures. Just type in whatever words you can remember describing what it is you were looking for, and presto, your browser (or Siri, or Cordera, or whatever) provides you with a nice list of places you meant to actually visit.
Thanks to some more inventive programmers, and some back-channel deals with the global service providers that actually lookup and translate the addresses to physical computers out there in the Cloud, you won’t have to be bothered with those pesky 404- errors ever again (unless you actually try typing in an invalid public IP address, in which case your browser search engine will take over and try to look that up.)
Welcome to searchguide.level3.com – no more 404’s ever. Even if you wanted them.
Now, even your worst mistakes in typing can generate income for someone else. Who’d have thought? Yea, Skynet/Genisys!
While it’s probably a matter of time before 90% of the well-known DNS service companies monetize their DNS services, leaving it up to you to either re-configure manual resolv.conf files pointing to non- monetized lookups, or at least switching to Google’s Public DNS (which tracks you everywhere you click anyway – 8.8.8.8 and 8.8.4.4) or any of the ones that still remain https://vpsboard.com/topic/2542-level3-public-dns-servers-search-engine-redirect/
Eventually, we’ll probably see the AltInternet end up creating its own subterranean DNS similar to what Tor still does.
The inventories are used by the Oracle Universal Installer (ORACLE_HOME)/oui (by reference, rapidwiz installers, too) and opatch utilities.
If they are not correct for the host you are working on, the utilities may abort with all sorts of errors.
The Oracle Universal Installer can also be used to re-create the inventory, or attach a pre-existing Oracle Home.
Formatted in XML syntax, all tags either end on a single line, or have a beginning and end tag associated with each entry.
Learning the actual structure of the file can be useful when dealing with non-Production environments when you’re upgrading and downgrading components, and need to roll-back certain changes, or are experimenting with modifying the filesystem layout. This way, if the inventory has minor errors (such as when the IDX numbers get out-of-sequence due to removed components, or you have to roll-back an OEM Agent upgrade to a plugin), you will have an idea where to correct the file to make your OUI happy again.
All () references refer to your unique environment’s actual physical filesystem locations – e.g. (ORACLE_BASE) might be /u01/app/oracle on your host.
## Master location files indicating which inventory is being used (can be manually modified to re-point to older legacy homes that are not multi-home aware)
## an 11g and older Middle-tier home locator
$> cat /etc/oraInst.loc
#Oracle Installer Location File Location
#Wed Dec 01 07:15:54 EST 2010
inst_group=oinstall
inventory_loc=(INST_TOP)/admin/oraInventory
## an 11g and older Database-tier home locator
$> cat /etc/oraInst.loc
inventory_loc=(DB_ORACLE_HOME)admin/oui/(ENV_ID)/oraInventory
## a 12c home locator – will be maintained in-parallel with the 11i version for now – multiple entries delimited by colons
$> cat /etc/oragchomelist
## A main R12 Middle-tier inventory, with a 12.1.0.4.0 OEM Agent
## located in (inventory_loc)/ContentsXML/inventory.xml
#Notes:
# REMOVED=”T” – means ignore this during reading of the inventory
# IDX=(n) – a unique number assigned (preferably sequentially, but not required to be in numerical sequence in the file)
# HOME NAME= – a unique name (without spaces or special characters) – there are specific naming formats are required by RapidClone for R12 components and should not be changed, the non-R12 labels are less strict.
# Note that the 12c entries begin to include version information for each individual sub-component (REF_HOME_LOC)
HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).
mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/10.1.3.1/Apache/Apache/logs/ssl_mutex.10549 (System error follows)
System: No such file or directory (errno: 2)
And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).
applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status
You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: ENV_ID.myhostname
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
OEM will show the HTTP service as being down, even though the pages are probably still responding:
R12 EBS OEM AMS monitor HTTP Service down screenshot
Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.
These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration. These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:
# ###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
)
## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop
# Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *
As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.
Errors in file /opt/oracle/diag/rdbms/vstbpro/vstbpro1/trace/vstbpro1_j000_7796.trc:
ORA-12012: error on auto execute of job 11762
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073
3. Details of the trace file
Trace file /opt/oracle/diag/rdbms/vstbpro/vstbpro1/trace/vstbpro1_j000_7796.trc
Oracle Database 11g Release 11.1.0.7.0 – 64bit Production
With the Real Application Clusters option
ORACLE_HOME = /opt/oracle/product/11g
System name: SunOS
Node name: wvpdb09
Release: 5.10
Version: Generic_138888-01
Machine: sun4u
Instance name: vstbpro1
Redo thread mounted by this instance: 1
Oracle process number: 73
Unix process pid: 7796, image: oracle@wvpdb09 (J000)
ORA-12012: error on auto execute of job 11762
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073
ORA-06512: at line 1
Had an issue where the hostname (on Oracle Enterprise Linux 5.9 – OEL 64-bit) happened to have an incorrect hostname and alias when I had already installed the OEM 12c (12.1.0.3) Agent. Thus the OMS repository targets were all named incorrectly, even though the Agent was secured and registered. (This was a new database host).
In the $AGENT_HOME/sysman/log/emagent.nohup log was the following:
— EMState agent
—– Sat Mar 1 10:42:22 2014::10437::Auto tuning the agent at time Sat Mar 1
10:42:22 2014 —–
—– Sat Mar 1 10:42:23 2014::10437::Finished auto tuning the agent at time Sat Mar 1 10:42:23 2014 —–
—– Sat Mar 1 10:42:23 2014::10437::Launching the JVM with following options: -Xmx128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops —–
—– Sat Mar 1 10:42:23 2014::10491::Time elapsed between Launch of Watchdog process and execing EMAgent is 2 secs —–
—– Sat Mar 1 10:42:23 2014::10437::Agent Launched with PID 10491 at time Sat Mar 1 10:42:23 2014 —–
2014-03-01 10:42:23,962 [1:main] WARN – Missing filename for log handler ‘wsm’
2014-03-01 10:42:23,971 [1:main] WARN – Missing filename for log handler ‘opss’
2014-03-01 10:42:23,972 [1:main] WARN – Missing filename for log handler ‘opsscfg’
OMS decided to shutdown the agent because of the following reason sent from OMS: EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED
—– Sat Mar 1 10:42:37 2014::10437::Checking status of EMAgent : 10491 —–
—– Sat Mar 1 10:42:37 2014::10437::EMAgent exited at Sat Mar 1 10:42:37 2014 with return value 0. —–
—– Sat Mar 1 10:42:37 2014::10437::EMAgent was shutdown normally. —–
./emctl secure agent
./emctl start agent
Resulted in the same repeating failures.
Removed the Target Host and Agent from OEM 12c OMS
(Target -> Hosts -> (select host) -> [Remove], and then,
Setup -> Manage Cloud Control -> Agents -> (click on Agent:(port)
Agent Menu (upper-left dropdown) -> Target Setup -> Remove Target)
It happens: You have a 4-node DataGuarded Oracle 11gR2 database and your System Administrators need to take the boxes down for maintenance. But they don’t know that it’s protected under DataGuard, with Fast-start failover enabled (which automatically performs (via DG Observer) the switchover from Primary to Standby, switches roles, reconfigures listeners, and tries to keep everything 99.99995% available.) So they use good old:
sqlplus “/ as sysdba”
SQL> shutdown immediate;
Database closed.
And now you need to start it back up. But when your admins get to their familiar old prompt it burps out:
SQL> startup
Error: ORA-16825: multiple errors or warnings, including fast-start failover-related errors or warnings, detected for the database
Now there are a number of blogs which will walk you through the tedious steps of recovering this condition manually via DGMGRL and multiple control file and standby redo log restores to all the targets. But it was 1:00A local time and I didn’t want to spend the rest of the night crawling through these trying to get this beast back on its feet.
So, I returned to the never-ending exploratory world of what is contained in Oracle Enterprise Manager 12cR3 (12.1.0.3.0) And (tada sound), OEM can handle it all for you relatively automatically.
To get your Primary DB back on-line (to avoid the complaints from the users who have been disconnected)
DGMGRL> connect SYS/<pwd>@<primaryDGDBtarget>
DGMGRL> disable fast_start failover force;
SQL> startup
Database opened.
Now that the Primary is back open and accessible, return to OEM OMS and visit:
Targets -> Database -> (your Primary DB)
Availability -> Data Guard Administration
There’s a text link at the bottom which reads
Additional Administration:
Verify Configuration
That’s your ticket back to green-arrows and healthy DG in 90% of the situations. It will perform health-checks throughout your configuration, re-created standby redo logs, re-sychronizing disconnected standby databases, shipping archive logs whereever they are needed, repairing disrupted communications, restoring fast-start failover observers, and whatever else you have.
The other alternative is to simply Remove the existing failed Standby database (select, then [Remove] the existing standby database; then visit its host and delete the existing datafiles, redo, archive and tempfiles (you do not have to touch the TNSNAMES or Listener configurations – that will be re-created for you); then use [Add Standby Database] on the same screen to restore functionality (this is good as an if-all-else-fails, so this method of recovery).
Bought 3 of these Western Digital 3TB My Books from BandH.com because I wanted the core drives out of them (the $130 price per unit was less than buying the bare drives, so go figure…) They happened to be the Mac versions (p/n WDBAAG0030HCH-02 to be exact, Model 111a), which differ only by what software is pre-loaded on the maintenance partition (from here – http://support.wdc.com/product/download.asp?groupid=125&lang=en ).
Guts of a My Book 111a USB 3.0 External Enclosure
Decided to re-use the flimsy, but working plastic enclosures for a left-over Seagate 750GB drive. Connected everything and using the working sequence of power-plug first, power on using the little button, then plug in the USB connector.
Bink! The drive appears as a CD-ROM. A genuine read-only ROM drive. Cool (not.) Can’t format it, partition it, wipe it, nothing.
Google’d all over the place, and everyone was unplug-this, it’s your OS, it’s a bad USB. But I noticed in the Device Manager, I had a missing driver for a “TUSB3410 Boot Device”. So took care of that by finding Texas Instrument’s original driver at: http://www.ti.com/product/tusb3410 – that cleared the yellow exclamation point.
Then tried the suggested “WD Quick Formatter” utility from the WD website but kept receiving the same “This Media is Write Protected” error and the utility ends in error and exits. Hm…
On a hunch, I tried next the “WD SES Driver” – that installed fine, but didn’t change anything (and doesn’t hurt anything – it adds a specialised SCSI extension to allow access to some of the firmware features on the circuit board (LED controls, remote On/Off, etc.)
Re-connect the USB cable and let the PC recognize the drive,
Press [Retry]
And for whatever reason, it nicely reformatted the drive as NTFS and re-wrote the firmware back on the drive. At that point, I was able to use standard Drive Management to re-format it back to FAT32 (what I needed), and everything is perfectly normal.
==== Addendum 11/13/2012 –
I’ve reused one of the My Book enclosures with an older smaller SATA drive (happens to be a Seagate) and was running into a problem that MANY buyers of the cheaper My Book external drives have been facing:
You plug it in, and try to start it, and the front LED just blinks. You hold the reset button, and it still blinks. You leave the power off for an hour, plug it back in, and it still just blinks. Leave it overnight, and it powers on normally. Turn it back off and try to restart it, and it just blinks again (no drive spin-up.) [Link to Western Digital’s Community Forum discussion on this topic: http://community.wdc.com/t5/External-Drives-for-Mac/WD-Elements-2TB-External-hard-drive-BLINKING-LIGHT-Problem/m-p/503802#M6647]
What did I just discover?
11/13/2012 – SOLVED (for the blinking light on power-on). Okay this was a really dumb idea. But all I did when I turned it off and it wouldn’t turn back on (unplugged everything and kept the power off for at least 30 minutes and still only got the blinking light. I stuck the My Book in the freezer for 5 minutes. (Yes…that’s what I just did.)
Just took it out, with a cold plastic case, plugged it in, and it powered on perfectly.
I have never had to refrigerate a USB hard drive controller to get it to reset, but this is a totally new one for me.
If you Care a Little More, Things Happen. Bees can be dangerous. Always wear protective clothing when approaching or dealing with bees. Do not approach or handle bees without proper instruction and training.
During an online patching session with ADOP, when the prepare phase checks the data dictionary integrity, you receive the following error:
Bought 3 of these 
Thoughts from James H. Lui 
You must be logged in to post a comment.