Category Archives: Oracle

Because I work with it. A lot.

R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent

Adop-PrepareDuring an online patching session with ADOP, when the prepare phase checks the data dictionary integrity, you receive the following error:

R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent:
Verifying data dictionary.
[UNEXPECTED]Data dictionary corrupted:
[UNEXPECTED]Data dictionary corruption – missing parent
1416131 ORA$BASE APPS BPA_GUIDED_PICKING SYNONYM
1416121 ORA$BASE APPS CO_CLIPPERSHIP_INTERFACE SYNONYM
1416132 ORA$BASE APPS BPA_PRINT_GUIDED_PICK_LABELS SYNONYM
1416128 ORA$BASE APPS GEM_ESIG_INQUIRY_CORE SYNONYM
[UNEXPECTED]Data dictionary corruption detected. Provide details to
[UNEXPECTED]Oracle Support and ask for a bug to be opened against the
[UNEXPECTED]Online Patching component of Oracle Application Install.

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

Issue:

The corruption is detected with the $AD_TOP/sql/ADZDDBCC.sql scriptis executed just prior to setting up the ADOP session.  This is also a mandatory check performed during pre-upgrade readiness checking for R12.2 upgrades.

Solution:

There is a rudimentary, but effective script supplied in $AD_TOP/patch/115/sql named adzddmpfix.sql (and a number of other adzd___ named scripts that address other data dictionary issues, such as the common TimeZone timestamp issues (Doc ID 2017686.1))

This script is run as SYS and requires all middle-tier services to be shutdown because it’s rebuilding all of the APPS synonyms for objects and recompiling afterwards.

sqlplus “/ as sysdba” @$AD_TOP/patch/115/sql/adzddmpfix.sql

REM $Header: adzddmpfix.sql 120.0.12020000.3 2015/09/14 10:10:32 sstomar noship$
REM dbdrv: none
REM +======================================================================+
REM | Copyright (c) 2005, 2015 Oracle and/or its affiliates. |
REM | All rights reserved. |
REM | Version 12.0.0 |
REM +======================================================================+
REM | FILENAME
REM | adzddmpfix.sql
REM |
REM | DESCRIPTION
REM | Warning: this script is an experimental workaround to the Oracle
REM | Database data Dictionary “missing parent” corruption described in
REM | Bug 20627866. This bug currently has no other known fix, but this
REM | script will repair the problem sufficiently to allow adop to continue
REM | executing.
REM |
REM | Warning: You must be connected as SYSDBA with application services
REM | shutdown to run this script
REM | Usage:
REM | sqlplus / as sysdba
REM | @adzddmpfix.sql
REM |
REM |
REM +=======================================================================+

Sample output:

SQL> @adzddmpfix.sql
“—- Fixing Data Dictionary Corruptions (missing parent) —-”

4 rows deleted.

Commit complete.

System altered.

“—- Compiling invalids —-”

Don’t forget to re-run the $AD_TOP/sql/ADZDDBCC.sql script to identify whether the data dictionary corruption is still present afterwards.

 

Advertisements

#C18LV Collaborate 2018 Networking Opportunity Events

Where I can keep track of the special events (note: these are not “parties” as so many people are misled to believe) at Collaborate (April 20-26, 2018 – Las Vegas, Nevada). To attend one of these events:

  1. You’re registered as an Collaborate Attendee.
  2. You’re either a prospect, customer, or goodwill contact for the host.
  3. You visit the host’s booth at Collaborate in order to pick up whatever is required for entry.
  4. Do not just show up at the event and attempt to “crash” it – just spend your time at a regular #C18LV reception the same evening and you’ll still get plenty of party time.

My sessions for this year:

R12.2 Running AutoConfig Manually

The follwing script is located in $AD_TOP/bin:

adconfig.sh

This file which is normally executed under the adautocfg.sh wrapper script in $ADMIN_SCRIPTS_HOME, can also be executed manually to bypass certain limitations found when configuring the PATCH filesystem, in particular.

The format of the command is:

adconfig.sh run=INSTE8 contextfile=$CONTEXT_FILE

Sample output:

Enter the APPS user password: Script execution phase specified: INSTE8

The log file for this session is located at: /u01/oracle/fs1/inst/apps/$ENV_ID/admin/log/02120957/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
 Using CONFIG_HOME location : /u01/oracle/fs1/inst/apps/$ENV_ID
 Classpath : /u01/oracle/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/oracle/fs1/EBSapps/comn/java/classes

Using Context file : /u01/oracle/fs1/inst/apps/$ENV_ID/appl/admin/$ENV_ID.xml

Context Value Management will now update the Context file

Updating Context file...COMPLETED

Execution phase specified at run time: INSTE8
 No uploading of Context File and its templates to database.

Configuring templates from all of the product tops...
 Configuring AD_TOP........COMPLETED
 Configuring FND_TOP.......COMPLETED
 Configuring ICX_TOP.......COMPLETED
 Configuring MSC_TOP.......COMPLETED
 Configuring IEO_TOP.......COMPLETED
 Configuring BIS_TOP.......COMPLETED
 Configuring CZ_TOP........COMPLETED
 Configuring SHT_TOP.......COMPLETED
 Configuring AMS_TOP.......COMPLETED
 Configuring CCT_TOP.......COMPLETED
 Configuring WSH_TOP.......COMPLETED
 Configuring CLN_TOP.......COMPLETED
 Configuring OKE_TOP.......COMPLETED
 Configuring OKL_TOP.......COMPLETED
 Configuring OKS_TOP.......COMPLETED
 Configuring CSF_TOP.......COMPLETED
 Configuring IBY_TOP.......COMPLETED
 Configuring JTF_TOP.......COMPLETED
 Configuring MWA_TOP.......COMPLETED
 Configuring CN_TOP........COMPLETED
 Configuring CSI_TOP.......COMPLETED
 Configuring WIP_TOP.......COMPLETED
 Configuring CSE_TOP.......COMPLETED
 Configuring EAM_TOP.......COMPLETED
 Configuring GMF_TOP.......COMPLETED
 Configuring PON_TOP.......COMPLETED
 Configuring FTE_TOP.......COMPLETED
 Configuring ONT_TOP.......COMPLETED
 Configuring AR_TOP........COMPLETED
 Configuring AHL_TOP.......COMPLETED
 Configuring IES_TOP.......COMPLETED
 Configuring OZF_TOP.......COMPLETED
 Configuring CSD_TOP.......COMPLETED
 Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.

Oracle eBusiness Suite R12.2.7 – New Features/Just the Facts

r1227_new_feat.banner
Oracle e-Business Suite R12.2.7 Released

Oracle eBusiness Suite R12.2.7 – New Features/Just the Facts

Full PDF Presentation here – Oracle eBusiness Suite R1227_Highlights_jlui

Summary Based Upon Presentation by Cliff Godwin, Oracle E-Business Suite Release 12.2.7 Highlights, September 2017 – Watch the video here.

Roadmap [image]

iProcurement 12.2.7

  • Auto-Suggest Catalog Items
  • New Supplier Creation Flow for Non-Catalog & Contractors
  • New Supplier Creation Flow for Spot Buys
  • Show Procurement Activities to Requesters
  • New Requester Notifications for P.O. Status Updates
  • Simplified Shopping Cart UI

Auto-Suggested Catalog Items [image]

Supplier On-boarding Flow [image]

Supplier Creation Flow for Spot Buys [image]

PO Information to Requesters [image]

Purchasing / Contracts 12.2.7

  • Purchasing
    • Automated Grouping of Multiple Requests to PO’s
    • Validate Agreement Line Amounts During Order/Release Submission
  • Procurement Contracts
    • Content Search
  • Procurement Command Center
    • Partial Searches / Snippet Preview

PO Requisition Auto-Grouping [image]

PO Line Amount Validation [image]

Contracts Content Search [image]

iSupplier / Supplier Lifecycle Mgt

  • Improved Automatic Recurring Assessments
  • Contact and Address Purpose and Associated Notifications
  • Visibility of Dynamic Supplier Lists

Supplier LCM – Recurring Assessments [image]

Supplier LCM / iSupplier Portal [image]

Project Procurement 12.2.7

  • Support for Technical Specifications
  • Support for Buyer Notes
  • Support for Descriptive Flexfields
  • Planning by Need-By Date
  • Change History Page UX Enhancements

Projects 12.2.7

  • Project Costing
    • Calculate Overtime Labor for Work-Based Timecard Layouts
  • Project Planning & Control
    • Work Plan Lifecycle
  • Advanced Project Planning & Controls
    • Support for Unit Rate Contract Schedule of Values

PP&C Workplan Lifecycle [image]

AP&C Unit Rate Contracts [image]

Order Management 12.2.7

  • Order Management
    • Milestone Billing
    • Manual Item Substitution After Booking
    • Schedule Order Enhancements
  • Quoting
    • Quote Status Flow by OU; Defaults, Updates
  • iStore
    • Order Type per Specialty Site; Email Notification Style
  • Customer Hub
    • Enhanced Party Merge

OM: Milestone Billing [image]

Quote Status Flow by OU [image]

Logistics / Inventory 12.2.7

  • Inventory / WMS
    • Material Workbench UI
    • Enhanced Receiving & Shipping UI
    • Enhanced Shipping / Oracle Transportation Mgt Integration
  • Material Supply Chain Administration (MSCA)
    • Label Reprints by Any User; Default Org on Login

Material Workbench UI

Manufacturing 12.2.7

  • Discrete MES
    • Enhanced Supervisor Review of Operator Times
    • Express Complete for Related Job Operations
  • Process Manufacturing
    • Automated Time-Based Sampling
  • Cost Management
    • Accrual Reconciliation for Inventories

Express Complete for Related Job Operations

Automated Time-Based Sampling

Value Chain Planning 12.2.7

  • Functional Security for ADF UI’s
  • Org & Function Security in Supply & Service Parts Planning ADF Work Areas
  • Production Scheduling of Phantoms

Org & Function Security

Asset Lifecycle Management 12.2.7

  • Enterprise Asset Management
    • Work Order Mass Updates
    • Generate Preventative Maintenance Work Order Report
    • Operation Completion as Mandated
    • Add Supplier Info for Work Order Directives
  • Installed Base
    • WebADI for Extended Attributes

WebADI for Installed Base Extended Attributes

Service 12.2.7

  • Advanced Scheduling
    • Add Resource Cost Factoring
  • Spares Mgt
    • Error Packing and Shipping Instructions for Parts Requirements; Copy Requirements to Internal Orders
  • Field Service
    • Self-Assign Labor Debrief Checklist Consolidated Debrief
  • TeleService
    • Enhanced UI’s
  • Depot Repair / Depot Info Discovery
    • Improved Graphs, Drilldowns

Depot Repair Information Discovery

Channel Revenue Management 12.2.7

  • Multi-Currency Approval Rule for Claims
  • Standalone Installation Support

Channel Revenue Mgt Standalone Support [image]

Human Capital Management 12.2.7

  • Time & Labor
    • Timecard Entry for Terminated Employees
    • Timekeeper Grouping by Supervisor
    • OT Calc for Project Work-Based Layout
  • Payroll
    • Dynamic Assignment Sets UI/WebADI
  • Learning Management
    • Test Overview for Online Assessments
    • Certification-Only Classes

Payroll Dynamic Assignment Sets [image]

Learning Mgt: Online Assessments [image]

ATG OAF User Experience 12.2.7

  • Administrator Personalizations
    • Admin Personalization Workbench
    • Themes
  • End-User Interactions
    • HGrid Column Freeze
    • Add to Favorites via Icons
    • Enhanced Look-Ahead LOV’s
    • Enhanced Attachments

OAF Roadmap 12.2.x [image]

Admin Personalization Workbench [image]

OAF Themes [image]

Oracle OpenWorld 2017 Networking Opportunity Events #OOW17

Oracle Openworld and JavaONE 2017 Banner Image
Register for Oracle Openworld and JavaONE 2017 https://www.oracle.com/openworld/index.html

Where I can keep track of the special events (note: these are not “parties” as so many people are misled to believe – you should expect to be contacted by dozens of sales and account professionals at each event) at OOW.  To attend one of these events:

  1. You’re registered as an OpenWorld Attendee.
  2. You’re either a prospect, customer, or goodwill contact for the host. That means you actually contact the host and ask to attend before OOW.
  3. You visit the host’s booth at OpenWorld in order to pick up whatever is required for entry and get to know what the host actually does.
  4. Do not just show up at the event and attempt to “crash” it – fair warning. You will be sold Oracle products and consulting, and like it, and start telling all your friends about it and be assimilated.

From 2016 – still being confirmed:

Visit all those vendors and make them feel welcomed this year!

http://www.7×7.com/eat-drink/ultimate-foodie-tour-ferry-building

Oracle OpenWorld Fun Community Events
There are 3 community events at OOW this year,

  1. A guided bike tour hosted by Oracle’s PL/SQL and cycling guru, Bryn Llewelyn on Saturday, September 30th at 10 a.m.
  2. A run across the Golden Gate Bridge hosted by the SQL Dev team on Sunday, October 1st at 7:30 a.m.
  3. 6th annual swim in the San Francisco Bay hosted by Oracle ACE Director Chet Justice on Monday, October 2th at 7:30 a.m.

The 2016 List of OOW Events

The 2015 List of OOW Events

The 2014 List of OOW Events

The 2013 List of OOW Events

The 2012 List of OOW Events

Oaktable 2017 will also be at OOW Again!

Getting Started with OEM 13cR2 (13.2.2.0.0) Enterprise Manager

oem13c_mainpage_ss
OEM13c Release 2 Enterprise Manager Summary Page

Relatively easy installation, a new learning curve to master for post-installation maintenance and patching.
The principal changes in Release 2 of 13c are the segregation of the Cloud Management Services plugins (and related functionality) to specific licensable plugin components – namely:

  • Cloud Services Management – used for provisioning, allocation, and administration (think “infrastructure”) of Oracle Public Cloud-based resources (databases, development tools, middleware, etc.)
  • Oracle Cloud Application – a portal punch-out to access Oracle Cloud-based applications such as hosted e-Business Suite, Cloud Content Management, Cloud SOA Suite, Cloud Identity Management, etc. (think “apps”.)
  • Cloud Framework – a repackaging of the EM OMS framework components, and the only portion of the “cloud” components included in the base OEM licensing.
    Installation guide: http://www.oracle.com/technetwork/oem/grid-control/documentation/oem-091904.html
Oracle Cloud Framework
Cloud Management related plugins for OEM13cR2

This proof-of-concept testbed system was based upon the latest available installer set – Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (13.2.0.0) – we happen to be testing backwards-compatibility with some 10g databases before committing to the upgrade.  Direct (out-of-place) upgrades are supported for version of EM 12.1.0.4 and newer.

Download page for OEM 13c R2
http://www.oracle.com/technetwork/oem/enterprise-manager/downloads

Fusion Middleware in this release is Weblogic 12.1.3
The host system is built upon Windows Server 2012 R2, single-tier, 16GB RAM, 2 cores, also hosting the OMS repository database in a 12.2.0.1.0 EE installation.  Since the DB Templates provided seemed to be specific for a 12.1.x database (and errors out in certain steps during the build), I simply created a generic database from the standard template with the pre-requisite parameter settings (e.g. _allow_insert_with_update_check=TRUE; sessions > 300; shared_pool_size (10% of SGA); etc.) , and pre-created the required tablespaces:

MGMT_AD4J_TS
MGMT_ECM_DEPOT_TS
MGMT_TABLESPACE

  During the installation, the (Oracle Universal Installer) OUI will create the following new users assigned to these tablespaces:

Open:
SYSMAN
SYSMAN_TYPES
SYSMAN_BIPLATFORM
SYSMAN_STB
SYSMAN_OPSS
SYSMAN_MDS
MGMT_VIEW

Expired & Locked:
CLOUD_ENGINE_USER
CLOUD_SWLIB_USER
EUS_ENGINE_USER
SYSMAN_RO

The standard download set includes a single executable file (.exe or .bin) and several additional ZIP archives.  These are to be staged in a single directory, and the ZIP files do NOT need to be pre-extracted.

Overall, the installation was straight-forward and as-documented (despite this being a Windows installation – which usually has its own quirks.) The portion that I feel has the steepest learning curve is post-installation maintenance, and particularly patching. The new tool in 13c, the OMSpatcher (Patchset 19999993 – think of it as a Java wrapper for opatch) was really designed for use specficially in a cloud-based environment and hasn’t been completely polished for use on-premise in smaller installations.

For example, it is designed to rely upon an active Weblogic AdminServer to instantiate it’s requests for inventory versioning information and component availability for patching. But if any of those components are actually in-use by the AdminServer itself, it will tend to fail to apply, and instead provide a lengthy step-by-step instruction set of how to accomplish all of the steps manually. Depending on the complexity of your OEM setup (e.g. the number of registered plugins and target types) this could take over an hour to get the patching utility to fail and then provide the manual steps.

The OEM-specific version of OPatch (Patchset 6880880) is quite different than its predecessors.  While the OMSPatcher uses the prior “unzip into the existing Oracle Home” installation technique, the OPatch utility itself now is a Java installer that relies on the OUI to allow updating of component versioning and inventory adjustments. Most importantly, you will need to “install” the new version of OPatch before being allowed to continue with your planned patching:

$JAVA_PATH\java -jar $PATCH_STAGE\6880880\opatch_generic.jar -silent oracle_home=$ORACLE_HOME (being updated for OPatch)

This OPatch update needs to be applied (before patching) to every Oracle Home using 13c technology (the OMS home, the Agent Home, other FMW homes at version 12.1.3 or newer, etc.)

See MOS EM 13c: How to Apply a Patch to the Enterprise Manager 13c Cloud Control OMS Oracle Home (Doc ID 2091619.1) for details about making an OMSPatcher property file, and why you want to create one.

Final steps executed:
Build general purpose 12.2.0.1.0 database with new tablespaces.
Install Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (13.2.0.0)
Patch 19999993: EM OMSPatcher latest version 13.8.0.0.2 (unzip to ORACLE_HOME)
Patch 6880880: EM OPatch latest version 13.9.0.0.0 (both OMS and Agent)

cd $PATCH_STAGE\<patch#>
$FMW_HOME\OMSPatcher\omspatcher apply -analyze -property_file 
$FMW_HOME\OMSPatcher\omspatcher apply -property_file  [OMS_DISABLE_HOST_CHECK=true  -- a useful option added to deal with virtual host names]

Patch 25163555: Tracking bug for Back-porting 24588124 oms side fix
Patch 25604219: MERGE REQUEST ON TOP OF 13.2.0.0.0 FOR BUGS 25497622 25497731 25506784
Patch 25387277: APR-2017 PSU OMS 13.2.0.0.170418
Patch 25162444: EM-BEACON Bundle Patch 13.2.0.0.161231 (Agent)
Patch 25580746: EM-AGENT Bundle Patch 13.2.0.0.170331 (Agent)

Other useful references:

Enterprise Manager 13.2 Master Bundle Patch List (Doc ID 2219797.1)
13.2.0.0.170418 Enterprise Manager Base Platform Patch Set Update (PSU) Readme for Oracle Management Server (OMS) (Doc ID 2246778.1)

How To Get Those Customizations to Work Correctly with Oracle EBS R12.2 Editioning (EBR)

Oracle e-Business Suite R12.2 Overview of Edition-Based Redefinition (EBR) in 11gR2 databases.
Oracle e-Business Suite R12.2 Overview of Edition-Based Redefinition (EBR) in 11gR2 databases.

Scenario:
Creating a new view based upon the AP.AP_SUPPLIERS table to reside in a new XXCUSTOMSCHEMA.

If you attempt the simple:

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM ap.ap_suppliers ;

It works, but your data and table definition may change whenever patching editions are in-play.

If you attempt to just reference the editioned object instead:

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM ap.ap_suppliers# /* or the preferred apps.ap_suppliers synonym */

You will receive the ominous ORA-38818: illegal reference to editioned object error message.

What’s missing? The XXCUSTOMSCHEMA isn’t recognized as an edition-friendly schema by Oracle’s e-Business Suite (nor the database), yet.

Create your XXCUSTOMSCHEMA in the usual way:

create user XXCUSTOMSCHEMA
identified by (password)
default tablespace USERS
temporary tablepace TEMP
profile DEFAULT
account UNLOCK;

/* as APPS user */

exec FND_ORACLE_USER_PKG.LOAD_ROW(‘XXCUSTOMSCHEMA’, ‘CUSTOM’, ‘INVALID’, NULL, ‘N’, ‘B’);

/* as SYSTEM user */

alter session set current_schema=APPS;
exec AD_ZD_PREP.ENABLE_CUSTOM_USER(‘XXCUSTOMSCHEMA’);

/* Create your SELECT grants to the editioned objects as APPS user */

grant select on ap.ap_suppliers#  to XXCUSTOMSCHEMA;

/* Create your custom view referencing the APPS synonym – which points to the editioned object */

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM apps.ap_suppliers ;

This is the same process when creating procedures and functions with the usual caveat that unless it’s a 12c database, you cannot create materialized views on the editioned objects.

For the 11gR2 EBS users, you would stick with the existing non-editioned table object reference:

CREATE MATERIALIZED VIEW APPS.XXAP_SUPPLIER_NEW_MV
(
vendor_id,
vendor_name
)
AS
SELECT aps.vendor_id, aps.vendor_name
FROM ap.ap_suppliers aps;

/* The owning schema for the materialized view must have CREATE TABLE, VIEW AND INDEX privileges on the destination tablespace */

Setup and Install the Oracle EBS Integrated SOA Gateway for R12.2.x

https://blogs.oracle.com/ebusinesssuiteintegration
1317697.1Functional Diagram of the Integrated SOA Gateway in Oracle e-Business Suite

Registering the Integrated SOA Gateway included in the Oracle E-Business Suite R12.2.x instances is actually reasonably simple, but the documents make it sound much worse than it really is, simply by being several hundred pages of documentation.

The reason to do this is to expose the RESTful services interface for EBS so that other web applications can do useful things, such as check usernames for validity, lookup business process (workflow) statuses, and even query for completion of processes, such as concurrent programs.  The built-in library of accessable Services is quite vast and enables lots of interconnectivity with other applications.

This is also useful if you happen to be extending the application using the Mobile Application Framework, which allows you to build custom applications for mobile devices that access all the various EBS functionalities.

References:
NOTE:1311068.1 – Installing Oracle E-Business Suite Integrated SOA Gateway, Release 12.2
NOTE:1317697.1 – Oracle E-Business Suite Integrated SOA Gateway Troubleshooting Guide, Release 12.2

Before commencing, have the following prepared:
Unlock the ASADMIN (FND_USER) account and set the password so that it can login without a change password challenge (no expiry. required.
APPS password
Weblogic password
IP Address of ApplTier Host
Ports for ApplTier Host (both base port and WLS console – e.g. 8000 7001)
Your base port is the one in your usual EBS URL and the WLS console URL can be determined via Oracle Application Manager in the Site Map link for Weblogic Administration.
Services will be interrupted by bounce twice (2X) during installation.
Installation takes about 45 minutes to complete.

====================================  Start of SOA Integration Steps
#Register new Internal SOA Gateway Configuration

itsrv33m [/export/home/dbausr/oradev] ant -f $JAVA_TOP/oracle/apps/fnd/txk/util/txkSoaConfigUtility.xml
Buildfile: $COMMON_TOP/java/classes/oracle/apps/fnd/txk/util/txkSoaConfigUtility.xml
checkenvset:
getServerDetails:
[input] Enter Oracle WebLogic Server Administration Console URL (host:port) :
[FQWLSHostname:port]
[input] Enter Oracle WebLogic Server Admin User Name :  [weblogic]
getWLSAdminPasswordUnix:
echoON:
[input] Enter the password for user weblogic :
echoOFF:
getWLSAdminPasswordWindows:
getPasswordUnix:
echoON:
[input] Enter the password for user apps :
echoOFF:
getPasswordWindows:
getASADMINUser:
[input] Enter the ASADMIN user name :  [ASADMIN]
getASADMINPasswordUnix:
echoON:
[input] Enter the password for user ASADMIN :
echoOFF:
getASADMINPasswordWindows:
getSOAServerDetails:
[input] Enter SOA Server  Hostname : (hostname.domain)
[FQEBShostname]
itsrv36m.mwd.h2o
[input] Enter External URL of SOA Suite SOA Managed Server (protocol://managed_server_hostname:managed_server_port) :
http://%5BFQEBShostname%5D:%5Bport%5D

[input] Enter Internal URL of SOA Suite SOA Managed Server (protocol://managed_server_hostname:managed_server_port) :
http://%5BFQEBShostname%5D:%5Bport%5D

[input] Enter External URL of SOA Suite Admin Server (protocol://admin_server_hostname:admin_server_port) :
http://%5BFQWLSHostname:port%5D

commonArgs:
checkRequiredJars:
getDbcFile:
ValidateASADMINConnection:
getJDBCURL:
addASADMINToFNDVault:
[echo] ********************************************************************
[echo]                  Adding/Updating ASADMIN user in fnd_vault
[echo] ********************************************************************
[echo] Connecting to the EBS database using below properties
[echo] User: apps
[echo] JDBC Url: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=[FQhostname])(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=[$ORACLE_SID])))
[sql] Executing commands
[sql] 0 rows affected
[sql] 1 of 1 SQL statements executed successfully
[echo] Sucessfully Added/Updated ASADMIN in fnd_vault
createUserCredUpdationScript:
executeUserCredUpdationUnix:
[exec]
[exec] CLASSPATH=$CLASSPATH
[exec]
[exec] Initializing WebLogic Scripting Tool (WLST) …
[exec]
[exec] Welcome to WebLogic Server Administration Scripting Shell
[exec]
[exec] Type help() for help on available commands
[exec]
[exec]
[exec] Connecting to server using username:weblogic url:[SOAhost:port]
[exec]
[exec] Connecting to t3://[SOAhost:port] with userid weblogic …
[exec] Successfully connected to Admin Server ‘AdminServer’ that belongs to domain ‘EBS_domain_[ORACLE_SID]’.
[exec]
[exec] Warning: An insecure protocol was used to connect to the
[exec] server. To ensure on-the-wire security, the SSL port or
[exec] Admin port should be used instead.
[exec]
[exec] Creating user ASADMIN
[exec] Sucessfully created user ASADMIN
[delete] Deleting: $INST_TOP/logs/appl/rgf/TXK/updateuser_02_November_2016_08_17_38.py
executeUserCredUpdationWindows:
updateWLSuser:
updateSoaContextVariables:
[echo]  Updating Context Variable s_soa_external_url
[echo]  Updating Context Variable s_soa_internal_url
[echo]  Updating Context Variable s_soa_admin_url
runAutoconfigUnix:
[echo]  Running Autoconfig
[exec] Enter the APPS user password:
[exec]
[exec] The log file for this session is located at: $INST_TOP/admin/log/[session#]/adconfig.log
[exec]
[exec] AutoConfig is configuring the Applications environment…
[exec]
[exec] AutoConfig will consider the custom templates if present.
[exec]     Using CONFIG_HOME location     : $INST_TOP
[exec]     Classpath                   :
[exec]
[exec]     Using Context file          : $CONTEXT_FILE
[exec]
[exec] Context Value Management will now update the Context file
[exec]
[exec]     Updating Context file…COMPLETED
[exec]
[exec]     Attempting upload of Context file and templates to database…COMPLETED
[exec]
[exec] Configuring templates from all of the product tops…
[exec]     Configuring AD_TOP……..COMPLETED
[exec]     Configuring FND_TOP…….COMPLETED
[exec]     Configuring ICX_TOP…….COMPLETED
[exec]     Configuring MSC_TOP…….COMPLETED
[exec]     Configuring IEO_TOP…….COMPLETED
[exec]     Configuring BIS_TOP…….COMPLETED
[exec]     Configuring CZ_TOP……..COMPLETED
[exec]     Configuring AMS_TOP…….COMPLETED
[exec]     Configuring CCT_TOP…….COMPLETED
[exec]     Configuring WSH_TOP…….COMPLETED
[exec]     Configuring CLN_TOP…….COMPLETED
[exec]     Configuring OKE_TOP…….COMPLETED
[exec]     Configuring OKL_TOP…….COMPLETED
[exec]     Configuring OKS_TOP…….COMPLETED
[exec]     Configuring CSF_TOP…….COMPLETED
[exec]     Configuring IBY_TOP…….COMPLETED
[exec]     Configuring JTF_TOP…….COMPLETED
[exec]     Configuring MWA_TOP…….COMPLETED
[exec]     Configuring CN_TOP……..COMPLETED
[exec]     Configuring CSI_TOP…….COMPLETED
[exec]     Configuring WIP_TOP…….COMPLETED
[exec]     Configuring CSE_TOP…….COMPLETED
[exec]     Configuring EAM_TOP…….COMPLETED
[exec]     Configuring GMF_TOP…….COMPLETED
[exec]     Configuring PON_TOP…….COMPLETED
[exec]     Configuring FTE_TOP…….COMPLETED
[exec]     Configuring ONT_TOP…….COMPLETED
[exec]     Configuring AR_TOP……..COMPLETED
[exec]     Configuring AHL_TOP…….COMPLETED
[exec]     Configuring IES_TOP…….COMPLETED
[exec]     Configuring OZF_TOP…….COMPLETED
[exec]     Configuring CSD_TOP…….COMPLETED
[exec]     Configuring IGC_TOP…….COMPLETED
[exec]
[exec] AutoConfig completed successfully.
runAutoconfigWindows:
setEBSProfiles:
[echo]  Updating Profile  FND_SERVER_DESKTOP_USER
[java] Nov 02, 2016 8:21:24 AM oracle.apps.fnd.txk.util.TXKFMWGenericConfigUtility readPasswordFromStdin
[java] INFO: Enter APPS Password:
[echo]  Updating Profile FND_SERVER_SEC
[java] Nov 02, 2016 8:21:26 AM oracle.apps.fnd.txk.util.TXKFMWGenericConfigUtility readPasswordFromStdin
[java] INFO: Enter APPS Password:
[echo]  Updating Profile FND_SERVER_IP_SEC
[java] Nov 02, 2016 8:21:28 AM oracle.apps.fnd.txk.util.TXKFMWGenericConfigUtility readPasswordFromStdin
[java] INFO: Enter APPS Password:
[echo]  Updating Profile FND_SQLNET_ACCESS
[java] Nov 02, 2016 8:21:29 AM oracle.apps.fnd.txk.util.TXKFMWGenericConfigUtility readPasswordFromStdin
[java] INFO: Enter APPS Password:

ebsConfigForSOA:

BUILD SUCCESSFUL
Total time: 6 minutes 36 seconds
# Verify new Profile Option Settings
SQL Statement which produced this data:
/* Formatted on 11/17/2016 7:07:54 AM (QP5 v5.300) */
SELECT n.user_profile_option_name              NAME,
TO_CHAR (v.last_update_date, ‘DD-MON-RR’)”Last Updated”,
DECODE (v.level_id,
10001, ‘Site’,
10002, ‘Application’,
10003, ‘Responsibility’,
10004, ‘User’,
10005, ‘Server’,
10007, ‘SERVRESP’,
v.level_id)
LEVEL_SET,
DECODE (TO_CHAR (v.level_id),
‘10001’, ”,
‘10002’, app.application_short_name,
‘10003’, rsp.responsibility_key,
‘10005’, svr.node_name,
‘10006’, org.name,
‘10004’, usr.user_name,
‘10007’, v.LEVEL_VALUE2,
‘UnDef’)
“CONTEXT”,
v.profile_option_value                  VALUE
FROM fnd_profile_options      p,
fnd_profile_option_values v,
fnd_profile_options_tl   n,
fnd_user                 usr,
fnd_application          app,
fnd_responsibility       rsp,
fnd_nodes                svr,
hr_operating_units       org
WHERE     p.profile_option_id = v.profile_option_id(+)
AND p.profile_option_name = n.profile_option_name
AND UPPER (p.profile_option_name) IN (‘FND_SERVER_DESKTOP_USER’,
‘FND_SERVER_SEC’,
‘FND_SERVER_IP_SEC’,
‘FND_SQLNET_ACCESS’)
AND n.language = ‘US’
/* Use for Friendly PONs e.g. MO: Security Option
AND UPPER (n.user_profile_option_name) =
UPPER (”)
*/
AND usr.user_id(+) = v.level_value
AND rsp.application_id(+) = v.level_value_application_id
AND rsp.responsibility_id(+) = v.level_value
AND app.application_id(+) = v.level_value
AND svr.node_id(+) = v.level_value
AND org.organization_id(+) = v.level_value
ORDER BY v.last_update_date DESC;

NAME|Last Updated|LEVEL_SET|CONTEXT|VALUE
FND: Desktop Nodes allowed|02-NOV-16|User|ASADMIN|[FQhostname]

NAME|Last Updated|LEVEL_SET|CONTEXT|VALUE
SQLNet Access|02-NOV-16|Site||ALLOW_RESTRICTED

NAME|Last Updated|LEVEL_SET|CONTEXT|VALUE
FND: Validate IP address|02-NOV-16|Site||D

NAME|Last Updated|LEVEL_SET|CONTEXT|VALUE
FND: Validate User Type|02-NOV-16|Site||D

# Bounce certain services (minimizes downtime – can also bounce everything)
cd $ADMIN_SCRIPTS_HOME

admanagedsrvctl.sh stop oafm_server1
admanagedsrvctl.sh start oafm_server1
admanagedsrvctl.sh stop oacore_server1
admanagedsrvctl.sh start oacore_server1
[etc for any other oacore servers]

#Verify registered payload Class ID (ensures payload can be accessed)
SQL Statement which produced this data:
SELECT class_id
FROM fnd_irep_classes
WHERE     UPPER (irep_name) LIKE ‘%EBIZHOMEPAGESERVICE%’
AND class_type = ‘SERVICEBEAN’;

CLASS_ID
[a Number]

#Try downloading the payload for a specific function (Class ID discovered above)
XSD.zip DEV2 URL:  http://%5ByourEBSURL:port%5D/webservices/isgagent/?generatexsd=%5Bthe above number]
XSD.zip

# Configure the ISG BPEL Payload for use by SOA Suite
ant -f $JAVA_TOP/oracle/apps/fnd/bpel/util/txk_integration_packager.xml \ -DIntegration=isg  -DFMWNodeName=itsrv33m.mwd.h2o -DFMWIpAddress=[MT or Virtual host IP]
[echo] Bpel Payload is archived at $INST_TOP/logs/appl/rgf/TXK/EBS_BPEL_payload.zip

# Verify the “new” SOA Node is registred (MWD is just re-using the built in SOA)
SQL Statement which produced this data:
SELECT node_name, server_address, server_id
FROM fnd_nodes
WHERE UPPER(node_name) in(‘[hostname]’);

NODE_NAME|SERVER_ADDRESS|SERVER_ID
[hostname]|[IP address]|3E3B399AEB80184EE0548A507414599D22425228221243006566902023757213

# A Fix for HPUX, if needed, add set +u to the line right after #!/bin/sh
# vi $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/bin/setDomainEnv.sh

. $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/bin/setDomainEnv.sh

# Warning: the following command will immediately shutdown and restart the WLS Managed Services
ant -f $JAVA_TOP/oracle/apps/fnd/txk/util/txkISGConfigurator.xml ebsSetup

# Change default NAME=APPS connection to NAME=ASADMIN in WLS Admin Console.
# (you want to do this to support APPS password changes in the future – the ASADMIN
# password does not have to be disclosed to Developers once the setup is completed – just
# grant the Integrated SOA Gateway Role and Responsibility to users that need to deploy
# various services.

Data Sources -> OAEADatasource -> Configuration -> Connection Pool
[Lock & Edit]
Modify Properties user=APPS -> user=ASADMIN
Enter ASADMIN password in both password fields
[Save] -> [Activate Changes]

If the ASADMIN password is changed in the target EBS instance, then all of the following steps need to be re-executed in order to register the change with SOA:

# Clean out the prior OAEADatasource connections and definitions
Login to WLS Administration Console.
Services -> Data Sources -> [Lock & Edit] -> Select OAEADatasource -> [Delete]
Security Realms -> myrealm -> Providers -> Select IsgAuthenticator -> [Delete]
[Activate Changes]

$] cd $DOMAIN_HOME/config/jdbc
$] rm OAEADatasource-*-jdbc.xml

Repeat the SOA Integration steps again.

Otherwise, attempting REST deployments will receive an error:
Error
java.lang.SecurityException: User: ASADMIN~~[ORACLE_SID], failed to be authenticated.

New OEM 12c/13c Agent Install Won’t Keep Running – Dies After Awhile

Many agents - Copyright by Warner Bros.,The Wachowski Brothers used without permission as educational content.
Agents are often hard to kill, usually. Images Copyright by Warner Bros., The Wachowski Brothers; used without permission as educational content.

OEM agents tend to occupy memory based upon how many targets they have to keep track of in a particular host.  At another organization, we tended to spin up VM’s for each instance environment, so at maximum, a particular agent might have a few hundred targets (especially on an e-Business Suite Applications Tier.)  In those circumstances, the default Java memory settings are probably fine.

In this environment, we run our hosts to death, and on this particular proof-of-concept host, we have 43 instances running on it, with variants of 10g, 11g, and 12c databases combined.

We are doing a fresh install of OEM 12.1.0.5.0 for our POC before setting up the 13c production OMS, and after deploying the agent to this particular database host, the agent would startup fine, run for about 20 or so minutes and then abruptly die without warning.

Re-starts fine, passes the usual tests fine (before the 20 minutes or so goes by) and then dies again.

AGENT_INST=/u01/app/oracle/agent12c/agent_inst

cd $AGENT_INST/bin

./emctl status agent

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 12.1.0.5.0
OMS Version            : 12.1.0.5.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent12c/agent_inst
Agent Log Directory    : /u01/app/oracle/agent12c/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/agent12c/core/12.1.0.5.0
Agent Process ID       : 10598
Parent Process ID      : 10499
Agent URL              : https://itsrv33c.mydomain:3872/emd/main/
Local Agent URL in NAT : https://itsrv33c.mydomain:3872/emd/main/
Repository URL         : https://itsrv35g.mydomain:1159/empbs/upload
Started at             : 2016-11-09 09:57:05
Started by user        : oracle
Operating System       : HP-UX version B.11.31 (IA64W)
Last Reload            : (none)
Last successful upload                       : 2016-11-09 10:19:26
Last attempted upload                        : 2016-11-09 10:19:26
Total Megabytes of XML files uploaded so far : 0.2
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 16.92%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-11-09 10:19:53
Last successful heartbeat to OMS             : 2016-11-09 10:19:53
Next scheduled heartbeat to OMS              : 2016-11-09 10:20:53

—————————————————————
Agent is Running and Ready

./emctl pingOMS

Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
—————————————————————
EMD pingOMS completed successfully

$AGENT_INST/sysman/log/gcagent.log contains

—– Wed Nov  9 09:39:43 2016::26900::Agent Launched with PID 27336 at time Wed
Nov  9 09:39:43 2016 —–
—– Wed Nov  9 09:39:43 2016::27336::Time elapsed between Launch of Watchdog p
rocess and execing EMAgent is 34 secs —–
2016-11-09 09:39:44,287 [1:main] WARN – Missing filename for log handler ‘wsm’
2016-11-09 09:39:44,302 [1:main] WARN – Missing filename for log handler ‘opss’
2016-11-09 09:39:44,305 [1:main] WARN – Missing filename for log handler ‘opsscf
g’
Agent is going down due to an OutOfMemoryError
—– Wed Nov  9 09:40:06 2016::26900::Checking status of EMAgent : 27336 —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent exited at Wed Nov  9 09:40:06 201
6 with return value 57. —–
—– Wed Nov  9 09:40:06 2016::26900::EMAgent will be restarted because of an O
ut of Memory Exception. —–
—– Wed Nov  9 09:40:06 2016::26900::writeAbnormalExitTimestampToAgntStmp: exi
tCause=OOM : restartRequired=1 —–
—– Wed Nov  9 09:40:06 2016::26900::Restarting EMAgent. —–

That means, the agent is starting, then stopping, then restarting, then stopping (aka “thrashing”)

Take a look for the running agent daemon at the OS level:

ps -ef | grep agent12c
oracle 26900     1  0 09:39:09 pts/0     0:00 /u01/app/oracle/agent12c/core/12.1.0.5.0/perl/bin/perl /u01/app/oracle/agent12c/core/12.1.0.5.0/bin/emwd.pl agent /u01/app/oracle/…
oracle 27665 26900  0 09:40:12 pts/0     1:01 /u01/app/oracle/agent12c/core/12.1.0.5.0/jdk/bin/IA64W/java -Xmx169M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///de…

Oh – it’s set up for the default of 169MB of RAM.  Check My Oracle Support.

EM 12c: emctl start agent Fails ‘Fatal agent error: State Manager failed at Startup’ ‘restarted because of an Out of Memory Exception’ Reported in emagent.nohup /gcagent.log (Doc ID 1950490.1)

Verify this setting also in the $AGENT_INST/sysman/log/gcagent.log:

—– Wed Nov  9 09:40:06 2016::26900::Auto tuning the agent at time Wed Nov  9
09:40:06 2016 —–
inMemoryLoggingSize=6291456
_SchedulePersistTimer=30
MaxThreads=10
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M
SchedulerRandomSpreadMins=5
UploadMaxNumberXML=5000
UploadMaxMegaBytesXML=50.0
Auto tuning was successful

Well, it’s trying. Per the above Doc ID 1950490.1

Stop the agent.

$AGENT_INST/bin/emctl stop agent

Edit the $AGENT_INST/sysman/config/emd.properties   (this contains the runtime parameters for the agent):

old entry:
agentJavaDefines=-Xmx169M -XX:MaxPermSize=96M

new entry:
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M

(You may tune these values up or down according to your environment requirements)

Restart the agent:

$AGENT_INST/bin/emctl stop agent

Agent runs, and keeps running like the E-Bunny.

R12.2 adop phase fs_clone fails because it can’t copy a file.

Complexity leading to fragility
Complex Systems Integration (Courtesy of: https://www.mulesoft.com/resources/esb/business-process-integration)

Oracle…sometimes blinding in complexity, and a foreshadowing of what happens when a system becomes so complex that it becomes fragile.

This is what the screen session looks like:

adnodemgrctl.sh: check the logfile /u01/oradev/fs1/inst/apps/DEV2_itsrv33m/logs/appl/admin/log/adnodemgrctl.txt for more information …
[UNEXPECTED]fs_clone has failed.
[STATEMENT] Please run adopscanlog utility, using the command
“adopscanlog -latest=yes”
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 2 (Fail)

Here is what it looks like in the adop.log

[EVENT]     Performing CLONE steps.
[PROCEDURE] [START 2016/11/08 08:20:19] CLONE Patch File System from Run File System
[PROCEDURE] Calling: /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl
[EVENT]     Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m
[UNEXPECTED]Error occurred running “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”
[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: “perl /u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -patchcontextfile=/u01/oradev/fs1/inst/apps/DEV2_itsrv33m/appl/admin/DEV2_itsrv33m.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=25 -timestamp=20161108_081645 -outdir=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m”.
[EVENT]     Releasing managed server ports.
[PROCEDURE] [START 2016/11/08 08:30:55] Releasing ports
[STATEMENT] Calling task: “Releasing ports on itsrv33m”; script: “/u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl”; args: ” -option=close -server=itsrv33m.mwd.h2o -ports=5570,7015,7615,7415,7215,7216,7217,7815,6114,6214,10013 -log=/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log”
[PROCEDURE] [START 2016/11/08 08:30:56] Releasing ports on itsrv33m
[PROCEDURE] Calling: /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/install/txkCloneAcquirePort.pl
[EVENT]     Log: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/txkCloneAcquirePort.log
[PROCEDURE] [END   2016/11/08 08:30:57] Releasing ports on itsrv33m
[PROCEDURE] [END   2016/11/08 08:30:57] Releasing ports

How cryptic!  Looks complicated.  This system happens to have AD/TXK.Delta.8, so the logs related to adop are all being centralized under the session # (in this case “25”.)

Diving deep into the log subdirectories:

/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log

================================
Inside copyComnUtil()…
================================

Directory /u01/oradev/fs1/EBSapps/comn already exists.
Removing the directory: /u01/oradev/fs1/EBSapps/comn/util

Copying the directory
———————
SOURCE : /u01/oradev/fs2/EBSapps/comn/util
TARGET : /u01/oradev/fs1/EBSapps/comn/util

Execute SYSTEM command : cp -rH  /u01/oradev/fs2/EBSapps/comn/util /u01/oradev/fs1/EBSapps/comn/util
EXIT STATUS: 1
Copied /u01/oradev/fs2/EBSapps/comn/util into /u01/oradev/fs1/EBSapps/comn/utilLOG FILE: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/cp_rH.out

<snip>
======================================
Inside copyCloneLogsToFSNE()…
======================================

Creating the directory: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply

Copying the directory
———————
SOURCE : /u01/oradev/fs2/inst/apps/DEV2_itsrv33m/admin/log/clone
TARGET : /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply

/u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply.
*******FATAL ERROR*******
PROGRAM : (/u01/oradev/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Tue Nov  8 08:30:55 2016
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /u01/oradev/fs2/EBSapps/comn/adopclone_itsrv33m/bin/adclone.pl did not go through successfully.

/u01/oradev/fs_ne/EBSapps/log/adop/25/20161108_081645/fs_clone/itsrv33m/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_11080821.log

(One innocuous, but annoying file, owned by someone other than applmgr, and happened to be 600 (user read/write only):)

Done copying /u01/oradev/fs2/EBSapps/appl/wms to /u01/oradev/fs1/EBSapps/appl/wms/
Copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
WARNING:
.end std out.
cp: cannot open /u01/oradev/fs2/EBSapps/appl/mwd/12.0.0/sql/MWDPOLPO.xsl: Permission denied

.end err out.
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/mwd failed
ERROR: Copying failed, so stopping all the parallel threads
Done copying /u01/oradev/fs2/EBSapps/appl/okc to /u01/oradev/fs1/EBSapps/appl/okc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/okc failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/pqh/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/pqh failed
Done copying /u01/oradev/fs2/EBSapps/appl/gmf to /u01/oradev/fs1/EBSapps/appl/gmf/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/gmf failed
Done copying /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies to /u01/oradev/fs1/EBSapps/comn/java/classes/oracle/apps/ies/
WARNING: Copying of /u01/oradev/fs2/EBSapps/comn/java/classes/oracle/apps/ies failed
Done copying /u01/oradev/fs2/EBSapps/appl/msc to /u01/oradev/fs1/EBSapps/appl/msc/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/msc failed
Done copying /u01/oradev/fs2/EBSapps/appl/ams to /u01/oradev/fs1/EBSapps/appl/ams/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/ams failed
Done copying /u01/oradev/fs2/EBSapps/appl/cs to /u01/oradev/fs1/EBSapps/appl/cs/
WARNING: Copying of /u01/oradev/fs2/EBSapps/appl/cs failed
START: Inside exitClone….
Updating status INCOMPLETE for ApplyApplTop
START: Updating status INCOMPLETE for action ApplyApplTop
END: Updated status INCOMPLETE for action ApplyApplTop

One file.  A single file.  Could have been a hidden file. Could have been anywhere in the entire AppsTier filesystem.  This one file stopped ADOP cold from performing.  This happened to be phase=fs_clone, but could have been a patching session, or a post-clone config (adcfgclone.pl) session.

Remove the file, and:

The fs_clone phase completed successfully.
adop exiting with status = 0 (Success)

Complexity leads to unintentional fragility.

Monitoring Oracle e-Business Suite (R12.2)

People ask what do I look for when monitoring an EBS instance for availability? Oracle Enterprise Manager (OEM) with the Applications Management Pack (AMS) gives a good starting point, but here is a condensed list of my general health check items for my production instance. It’s not everything possible, but it’s what I generally would check during a shake-down of a disaster-recovery exercise.

Warning Critical
Oracle E-Business Suite
Oracle Applications Infrastructure Service
Database Instance select SYSDATE from dual; Up/Down
Tablespace Freespace Available >80% >90%
Temporary Tablespace Used >80% >90%
Undo Tablespace Used >80% >90%
Fast Recovery Area Utilization >80% >90%
Redo Log Switch Rate > 10/hr. >20/hr
Blocking Sessions >1 >5
Database Listener tnsping <ORACLE_SID> Up/Down
TNSPing Response Time > 25ms > 100ms
Applications Listener tnsping <FNDFS_ORACLE_SID> Up/Down
Host Monitoring
CPU Utilization > 80% > 95%
RAM Utilization > 90% > 99%
Mountpoint Free Space Available <10% <5%
Concurrent Processing Services
Internal Manager Up/Down
Standard Manager < 10
Output Post Processor < 3
Receiving Transactions Manager < 1
Document Approval Manager < 1
Forms Applications Service
Forms Listener Service Up/Down
Forms Runtime Processes <1
Forms Response Time >5 secs >30 secs
Active Forms Sessions > 100 > 200
Self-Service Applications Service
HTTP Listener Service Up/Down
OPMN Processor Up/Down
OC4J Service Up/Down
WebLogic Deployed Services Up/Down
WebLogic AdminServer Up/Down
OA Framework Page Response Time > 5 secs > 10 secs
Oracle Applications JVM Usage >80% >90%
Workflow Serivce OAM Dashboard Up/Down
Workflow Agent Listener Up/Down
Workflow Background Engine Up/Down
Workflow Notification Mailer Services Up/Down
Oracle E-Business Suite Custom Objects
Invalid Object Count > 482 > 500
Oracle E-Business Suite Patch Information
Patch Inventory
ADOP Session Status >1 Pending >1 Failed

Oracle OpenWorld 2016 in 5 Pages

 

All of Oracle Cloud on a Page
Oracle OpenWorld 2016 Cloud Services and Products Summary

4 Days (almost 7 for me) of Product Directors and Managers spinning as many plates in the air as humanly possible, all being tossed into the Cloud with a fervor only matched by the lack of compensation being provided for conventional technology stack sales, left me with some specific value-adds for my time spent (not including the Safe Harbor information, we can’t say anything about….sshhhh.)

Note: (I’m EBS-centric and we have very little budget for hardware, so while I was amazed at the new SPARC S7 processors with on-board encryption and J/SQL optimization, alas they didn’t make it into my summary pages this year.)

AD/TXK Delta.8
Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.8 and R12.TXK.C.Delta.8 (Doc ID 2159750.1)

Configuring Oracle E-Business Suite Release 12.x Using Logical Host Names (Doc ID 1968231.1)

  • e-Mail Notifications at Parent and Child Levels for Request Sets
  • Save and Re-use Concurrent Program Parameters (CP Templates)
  • Regex Support in Profile Option Values and FND_MESSAGES
  • Multiple OA Framework Windows and Tabs in Multiple Browser Windows
  • EBS Healthcheck Diagnostics
  • 64-bit JDK’s on CM Tier
  • adop Improvements
  • Delta Sync Mode (syncmode=patch/delta) – changed rows only
  • Rsync support in phase=prepare
  • Custom module sync auto-supported (no custom drivers req’d)
  • Standardized logging SessionID/ExecutionID/phase/node

EBS 12.2.6.AD Oracle E-Business Suite 12.2.6 Readme (Doc ID 2114016.1)

  • Security Configuration Console
  • JSP Whitelisting
  • Redirection Whitelisting
  • AD/TKX.Delta.8 incorporated
  • More HTML User Interfaces: HTML alternatives to several existing Forms; interfaces offer simpler layouts and more modern user interactions.

–Examples include the Update Assets UI in Enterprise Asset Management, and AOL setup UIs for Users, Responsibilities, and Flexfields.

SecureFiles does not allow shrinking of the tablespace after deletions (AP Imaging)

Available Interconnect Cloud Service adapters:

  • –PL/SQL
  • –Java Beans
  • –Apps Modules

Roadmap ICS adapters:

  • –REST
  • –Concurrent Programs
  • –XML Gateway
  • –BPEL/Business Services

CAAS (Cloud Adaptive Assistant Service) – The widget Larry was text-chatting with.

Mobile Foundation and Mobile Application Framework

Foundation v6 released August 2016

–Incorporates MAF 2.3.2

–Enhancements for Approvals (HR, Projects, Service Centers, Quoting, Custom Approvals)

Foundation v7 scheduled for April 2017

MAF Roadmap items:

–Airwatch and MobileIron support

–Mobile REST API’s

–EBS Integrated Service Gateway support (SOA)

PDF Version for download:       oracle_oow16_summary_jlui