Tag Archives: adop

19c DB Bug in ADZDDBCC.sql – Online Patching Readiness Checker – Poor Performance

As of EBS ICM Application DBA Online Patching ADOP Analyzer version 200.39

ADOP Analyzer was not updated for 19c databases:

Per 19c Database with Oracle E-Business Suite R12 Known Issues & Solutions ( Doc ID 2662860.1 )

Running ADZDDBCC.sql in 19c database is taking time for Section 26.

Before running report set parameter _px_cdb_view_enabled to false .
alter session set “_px_cdb_view_enabled”=false ;

This is run both as part of 12.2 upgrade steps, and whenever you log an SR for ATG/ICM.

Work-around – add the “ALTER SESSION” command towards the top of the ADZDDBCC.sql – otherwise, on a 19c database, you’ll discover that when the dependency check portion runs, which looks like:
SELECT col.owner,
FROM dba_tables tab, dba_tab_columns col
WHERE tab.owner IN (SELECT oracle_username
FROM fnd_oracle_userid
WHERE read_only_flag IN (‘A’,
AND col.owner = tab.owner
AND col.table_name = tab.table_name
AND col.data_type_owner NOT IN (‘APPS_NE’,
FROM SYSTEM.fnd_oracle_userid fou,
fnd_product_installations fpi,
ad_obsolete_objects aoo
WHERE fpi.application_id = aoo.application_id
AND fou.oracle_id = fpi.oracle_id
AND fou.oracle_username = tab.owner
AND aoo.object_name = tab.table_name
AND aoo.object_type = ‘TABLE’)
FROM fnd_oracle_userid au, dba_users du
WHERE au.oracle_username = du.username
AND au.read_only_flag = ‘U’
AND du.editions_enabled = ‘Y’)

on 19c databases, you start seeing PX queue waits for every row fetch.

Adding the alter session avoids the queue/dequeue process occurring.

James Lui
2022 OATUG President
OATUG Board of Directors


R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent

Adop-PrepareDuring an online patching session with ADOP, when the prepare phase checks the data dictionary integrity, you receive the following error:

R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent:
Verifying data dictionary.
[UNEXPECTED]Data dictionary corrupted:
[UNEXPECTED]Data dictionary corruption – missing parent
[UNEXPECTED]Data dictionary corruption detected. Provide details to
[UNEXPECTED]Oracle Support and ask for a bug to be opened against the
[UNEXPECTED]Online Patching component of Oracle Application Install.

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”


The corruption is detected with the $AD_TOP/sql/ADZDDBCC.sql scriptis executed just prior to setting up the ADOP session.  This is also a mandatory check performed during pre-upgrade readiness checking for R12.2 upgrades.


There is a rudimentary, but effective script supplied in $AD_TOP/patch/115/sql named adzddmpfix.sql (and a number of other adzd___ named scripts that address other data dictionary issues, such as the common TimeZone timestamp issues (Doc ID 2017686.1))

This script is run as SYS and requires all middle-tier services to be shutdown because it’s rebuilding all of the APPS synonyms for objects and recompiling afterwards.

sqlplus “/ as sysdba” @$AD_TOP/patch/115/sql/adzddmpfix.sql

REM $Header: adzddmpfix.sql 120.0.12020000.3 2015/09/14 10:10:32 sstomar noship$
REM dbdrv: none
REM +======================================================================+
REM | Copyright (c) 2005, 2015 Oracle and/or its affiliates. |
REM | All rights reserved. |
REM | Version 12.0.0 |
REM +======================================================================+
REM | adzddmpfix.sql
REM | Warning: this script is an experimental workaround to the Oracle
REM | Database data Dictionary “missing parent” corruption described in
REM | Bug 20627866. This bug currently has no other known fix, but this
REM | script will repair the problem sufficiently to allow adop to continue
REM | executing.
REM | Warning: You must be connected as SYSDBA with application services
REM | shutdown to run this script
REM | Usage:
REM | sqlplus / as sysdba
REM | @adzddmpfix.sql
REM +=======================================================================+

Sample output:

SQL> @adzddmpfix.sql
“—- Fixing Data Dictionary Corruptions (missing parent) —-”

4 rows deleted.

Commit complete.

System altered.

“—- Compiling invalids —-”

Don’t forget to re-run the $AD_TOP/sql/ADZDDBCC.sql script to identify whether the data dictionary corruption is still present afterwards.


R12.2 adop phase fs_clone fails because it can’t copy a file.

Complexity leading to fragility
Complex Systems Integration (Courtesy of: https://www.mulesoft.com/resources/esb/business-process-integration)

Oracle…sometimes blinding in complexity, and a foreshadowing of what happens when a system becomes so complex that it becomes fragile.

This is what the screen session looks like:

adnodemgrctl.sh: check the logfile /u01/oradev/fs1/inst/apps/DEV2_itsrv33m/logs/appl/admin/log/adnodemgrctl.txt for more information …
[UNEXPECTED]fs_clone has failed.
[STATEMENT] Please run adopscanlog utility, using the command
“adopscanlog -latest=yes”
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 2 (Fail)

Here is what it looks like in the adop.log

[EVENT]     Performing CLONE steps.
[PROCEDURE] [START 2016/11/08 08:20:19] CLONE Patch File System from Run File System
[PROCEDURE] Calling: /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
[EVENT]     Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m
[UNEXPECTED]Error occurred running “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”
[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”.
[EVENT]     Releasing managed server ports.
[PROCEDURE] [START 2016/11/08 08:30:55] Releasing ports
[STATEMENT] Calling task: “Releasing ports on itsrv33m”; script: “/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl”; args: ” -option=close -server=itsrv33m.mwd.h2o -ports=5570,7015,7615,7415,7215,7216,7217,7815,6114,6214,10013 -log=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log”
[PROCEDURE] [START 2016/11/08 08:30:56] Releasing ports on itsrv33m
[PROCEDURE] Calling: /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl
[EVENT]     Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log
[PROCEDURE] [END   2016/11/08 08:30:57] Releasing ports on itsrv33m
[PROCEDURE] [END   2016/11/08 08:30:57] Releasing ports

How cryptic!  Looks complicated.  This system happens to have AD/TXK.Delta.8, so the logs related to adop are all being centralized under the session # (in this case “25”.)

Diving deep into the log subdirectories:


Inside copyComnUtil()…

Directory /u01/oradev/fs1/EBSapps/comn already exists.
Removing the directory: /u01/oradev/fs1/EBSapps/comn/util

Copying the directory
SOURCE : /u01/oradev/fs2/EBSapps/comn/util
TARGET : /u01/oradev/fs1/EBSapps/comn/util

Execute SYSTEM command : cp -rH  /u01/oradev/fs2/EBSapps/comn/util /u01/oradev/fs1/EBSapps/comn/util
Copied /u01/oradev/fs2/EBSapps/comn/util into /u01/oradev/fs1/EBSapps/comn/utilLOG FILE: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/cp_rH.out

Inside copyCloneLogsToFSNE()…

Creating the directory: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply

Copying the directory
SOURCE : /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/log/clone
TARGET : /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply

/u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply.
*******FATAL ERROR*******
PROGRAM : (/u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Tue Nov  8 08:30:55 2016
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.


(One innocuous, but annoying file, owned by someone other than applmgr, and happened to be 600 (user read/write only):)

Done copying /u01/oradev/fs2/EBSapps/appl/wms to /u01/oradev/fs1/EBSapps/appl/wms/
Copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
.end std out.
cp: cannot open /u01/oradev/fs2/EBSapps/appl/mwd/12.0.0/sql/MWDPOLPO.xsl: Permission denied

.end err out.
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/mwd failed
ERROR: Copying failed, so stopping all the parallel threads
Done copying /u01/oradev/fs2/EBSapps/appl/okc to /u01/oradev/fs1/EBSapps/appl/okc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/okc failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/pqh/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh failed
Done copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/gmf failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/ies/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies failed
Done copying /u01/oradev/fs2/EBSapps/appl/msc to /u01/oradev/fs1/EBSapps/appl/msc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/msc failed
Done copying /u01/oradev/fs2/EBSapps/appl/ams to /u01/oradev/fs1/EBSapps/appl/ams/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/ams failed
Done copying /u01/oradev/fs2/EBSapps/appl/cs to /u01/oradev/fs1/EBSapps/appl/cs/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/cs failed
START: Inside exitClone….
Updating status INCOMPLETE for ApplyApplTop
START: Updating status INCOMPLETE for action ApplyApplTop
END: Updated status INCOMPLETE for action ApplyApplTop

One file.  A single file.  Could have been a hidden file. Could have been anywhere in the entire AppsTier filesystem.  This one file stopped ADOP cold from performing.  This happened to be phase=fs_clone, but could have been a patching session, or a post-clone config (adcfgclone.pl) session.

Remove the file, and:

The fs_clone phase completed successfully.
adop exiting with status = 0 (Success)

Complexity leads to unintentional fragility.