Tag Archives: EBS

R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent

Adop-PrepareDuring an online patching session with ADOP, when the prepare phase checks the data dictionary integrity, you receive the following error:

R12.2 eBusiness Suite EBS – During ADOP phase = prepare data dictionary corruption missing parent:
Verifying data dictionary.
[UNEXPECTED]Data dictionary corrupted:
[UNEXPECTED]Data dictionary corruption – missing parent
1416131 ORA$BASE APPS BPA_GUIDED_PICKING SYNONYM
1416121 ORA$BASE APPS CO_CLIPPERSHIP_INTERFACE SYNONYM
1416132 ORA$BASE APPS BPA_PRINT_GUIDED_PICK_LABELS SYNONYM
1416128 ORA$BASE APPS GEM_ESIG_INQUIRY_CORE SYNONYM
[UNEXPECTED]Data dictionary corruption detected. Provide details to
[UNEXPECTED]Oracle Support and ask for a bug to be opened against the
[UNEXPECTED]Online Patching component of Oracle Application Install.

[STATEMENT] Please run adopscanlog utility, using the command

“adopscanlog -latest=yes”

Issue:

The corruption is detected with the $AD_TOP/sql/ADZDDBCC.sql scriptis executed just prior to setting up the ADOP session.  This is also a mandatory check performed during pre-upgrade readiness checking for R12.2 upgrades.

Solution:

There is a rudimentary, but effective script supplied in $AD_TOP/patch/115/sql named adzddmpfix.sql (and a number of other adzd___ named scripts that address other data dictionary issues, such as the common TimeZone timestamp issues (Doc ID 2017686.1))

This script is run as SYS and requires all middle-tier services to be shutdown because it’s rebuilding all of the APPS synonyms for objects and recompiling afterwards.

sqlplus “/ as sysdba” @$AD_TOP/patch/115/sql/adzddmpfix.sql

REM $Header: adzddmpfix.sql 120.0.12020000.3 2015/09/14 10:10:32 sstomar noship$
REM dbdrv: none
REM +======================================================================+
REM | Copyright (c) 2005, 2015 Oracle and/or its affiliates. |
REM | All rights reserved. |
REM | Version 12.0.0 |
REM +======================================================================+
REM | FILENAME
REM | adzddmpfix.sql
REM |
REM | DESCRIPTION
REM | Warning: this script is an experimental workaround to the Oracle
REM | Database data Dictionary “missing parent” corruption described in
REM | Bug 20627866. This bug currently has no other known fix, but this
REM | script will repair the problem sufficiently to allow adop to continue
REM | executing.
REM |
REM | Warning: You must be connected as SYSDBA with application services
REM | shutdown to run this script
REM | Usage:
REM | sqlplus / as sysdba
REM | @adzddmpfix.sql
REM |
REM |
REM +=======================================================================+

Sample output:

SQL> @adzddmpfix.sql
“—- Fixing Data Dictionary Corruptions (missing parent) —-”

4 rows deleted.

Commit complete.

System altered.

“—- Compiling invalids —-”

Don’t forget to re-run the $AD_TOP/sql/ADZDDBCC.sql script to identify whether the data dictionary corruption is still present afterwards.

 

Advertisements

R12.2 Running AutoConfig Manually

The follwing script is located in $AD_TOP/bin:

adconfig.sh

This file which is normally executed under the adautocfg.sh wrapper script in $ADMIN_SCRIPTS_HOME, can also be executed manually to bypass certain limitations found when configuring the PATCH filesystem, in particular.

The format of the command is:

adconfig.sh run=INSTE8 contextfile=$CONTEXT_FILE

Sample output:

Enter the APPS user password: Script execution phase specified: INSTE8

The log file for this session is located at: /u01/oracle/fs1/inst/apps/$ENV_ID/admin/log/02120957/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
 Using CONFIG_HOME location : /u01/oracle/fs1/inst/apps/$ENV_ID
 Classpath : /u01/oracle/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/oracle/fs1/EBSapps/comn/java/classes

Using Context file : /u01/oracle/fs1/inst/apps/$ENV_ID/appl/admin/$ENV_ID.xml

Context Value Management will now update the Context file

Updating Context file...COMPLETED

Execution phase specified at run time: INSTE8
 No uploading of Context File and its templates to database.

Configuring templates from all of the product tops...
 Configuring AD_TOP........COMPLETED
 Configuring FND_TOP.......COMPLETED
 Configuring ICX_TOP.......COMPLETED
 Configuring MSC_TOP.......COMPLETED
 Configuring IEO_TOP.......COMPLETED
 Configuring BIS_TOP.......COMPLETED
 Configuring CZ_TOP........COMPLETED
 Configuring SHT_TOP.......COMPLETED
 Configuring AMS_TOP.......COMPLETED
 Configuring CCT_TOP.......COMPLETED
 Configuring WSH_TOP.......COMPLETED
 Configuring CLN_TOP.......COMPLETED
 Configuring OKE_TOP.......COMPLETED
 Configuring OKL_TOP.......COMPLETED
 Configuring OKS_TOP.......COMPLETED
 Configuring CSF_TOP.......COMPLETED
 Configuring IBY_TOP.......COMPLETED
 Configuring JTF_TOP.......COMPLETED
 Configuring MWA_TOP.......COMPLETED
 Configuring CN_TOP........COMPLETED
 Configuring CSI_TOP.......COMPLETED
 Configuring WIP_TOP.......COMPLETED
 Configuring CSE_TOP.......COMPLETED
 Configuring EAM_TOP.......COMPLETED
 Configuring GMF_TOP.......COMPLETED
 Configuring PON_TOP.......COMPLETED
 Configuring FTE_TOP.......COMPLETED
 Configuring ONT_TOP.......COMPLETED
 Configuring AR_TOP........COMPLETED
 Configuring AHL_TOP.......COMPLETED
 Configuring IES_TOP.......COMPLETED
 Configuring OZF_TOP.......COMPLETED
 Configuring CSD_TOP.......COMPLETED
 Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.

How To Get Those Customizations to Work Correctly with Oracle EBS R12.2 Editioning (EBR)

Oracle e-Business Suite R12.2 Overview of Edition-Based Redefinition (EBR) in 11gR2 databases.
Oracle e-Business Suite R12.2 Overview of Edition-Based Redefinition (EBR) in 11gR2 databases.

Scenario:
Creating a new view based upon the AP.AP_SUPPLIERS table to reside in a new XXCUSTOMSCHEMA.

If you attempt the simple:

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM ap.ap_suppliers ;

It works, but your data and table definition may change whenever patching editions are in-play.

If you attempt to just reference the editioned object instead:

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM ap.ap_suppliers# /* or the preferred apps.ap_suppliers synonym */

You will receive the ominous ORA-38818: illegal reference to editioned object error message.

What’s missing? The XXCUSTOMSCHEMA isn’t recognized as an edition-friendly schema by Oracle’s e-Business Suite (nor the database), yet.

Create your XXCUSTOMSCHEMA in the usual way:

create user XXCUSTOMSCHEMA
identified by (password)
default tablespace USERS
temporary tablepace TEMP
profile DEFAULT
account UNLOCK;

/* as APPS user */

exec FND_ORACLE_USER_PKG.LOAD_ROW(‘XXCUSTOMSCHEMA’, ‘CUSTOM’, ‘INVALID’, NULL, ‘N’, ‘B’);

/* as SYSTEM user */

alter session set current_schema=APPS;
exec AD_ZD_PREP.ENABLE_CUSTOM_USER(‘XXCUSTOMSCHEMA’);

/* Create your SELECT grants to the editioned objects as APPS user */

grant select on ap.ap_suppliers#  to XXCUSTOMSCHEMA;

/* Create your custom view referencing the APPS synonym – which points to the editioned object */

create or replace force view XXCUSTOMSCHEMA.AP_SUPPLIER_NEW_V
as
SELECT *
FROM apps.ap_suppliers ;

This is the same process when creating procedures and functions with the usual caveat that unless it’s a 12c database, you cannot create materialized views on the editioned objects.

For the 11gR2 EBS users, you would stick with the existing non-editioned table object reference:

CREATE MATERIALIZED VIEW APPS.XXAP_SUPPLIER_NEW_MV
(
vendor_id,
vendor_name
)
AS
SELECT aps.vendor_id, aps.vendor_name
FROM ap.ap_suppliers aps;

/* The owning schema for the materialized view must have CREATE TABLE, VIEW AND INDEX privileges on the destination tablespace */

Monitoring Oracle e-Business Suite (R12.2)

People ask what do I look for when monitoring an EBS instance for availability? Oracle Enterprise Manager (OEM) with the Applications Management Pack (AMS) gives a good starting point, but here is a condensed list of my general health check items for my production instance. It’s not everything possible, but it’s what I generally would check during a shake-down of a disaster-recovery exercise.

Warning Critical
Oracle E-Business Suite
Oracle Applications Infrastructure Service
Database Instance select SYSDATE from dual; Up/Down
Tablespace Freespace Available >80% >90%
Temporary Tablespace Used >80% >90%
Undo Tablespace Used >80% >90%
Fast Recovery Area Utilization >80% >90%
Redo Log Switch Rate > 10/hr. >20/hr
Blocking Sessions >1 >5
Database Listener tnsping <ORACLE_SID> Up/Down
TNSPing Response Time > 25ms > 100ms
Applications Listener tnsping <FNDFS_ORACLE_SID> Up/Down
Host Monitoring
CPU Utilization > 80% > 95%
RAM Utilization > 90% > 99%
Mountpoint Free Space Available <10% <5%
Concurrent Processing Services
Internal Manager Up/Down
Standard Manager < 10
Output Post Processor < 3
Receiving Transactions Manager < 1
Document Approval Manager < 1
Forms Applications Service
Forms Listener Service Up/Down
Forms Runtime Processes <1
Forms Response Time >5 secs >30 secs
Active Forms Sessions > 100 > 200
Self-Service Applications Service
HTTP Listener Service Up/Down
OPMN Processor Up/Down
OC4J Service Up/Down
WebLogic Deployed Services Up/Down
WebLogic AdminServer Up/Down
OA Framework Page Response Time > 5 secs > 10 secs
Oracle Applications JVM Usage >80% >90%
Workflow Serivce OAM Dashboard Up/Down
Workflow Agent Listener Up/Down
Workflow Background Engine Up/Down
Workflow Notification Mailer Services Up/Down
Oracle E-Business Suite Custom Objects
Invalid Object Count > 482 > 500
Oracle E-Business Suite Patch Information
Patch Inventory
ADOP Session Status >1 Pending >1 Failed

Oracle OpenWorld 2016 in 5 Pages

 

All of Oracle Cloud on a Page
Oracle OpenWorld 2016 Cloud Services and Products Summary

4 Days (almost 7 for me) of Product Directors and Managers spinning as many plates in the air as humanly possible, all being tossed into the Cloud with a fervor only matched by the lack of compensation being provided for conventional technology stack sales, left me with some specific value-adds for my time spent (not including the Safe Harbor information, we can’t say anything about….sshhhh.)

Note: (I’m EBS-centric and we have very little budget for hardware, so while I was amazed at the new SPARC S7 processors with on-board encryption and J/SQL optimization, alas they didn’t make it into my summary pages this year.)

AD/TXK Delta.8
Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.8 and R12.TXK.C.Delta.8 (Doc ID 2159750.1)

Configuring Oracle E-Business Suite Release 12.x Using Logical Host Names (Doc ID 1968231.1)

  • e-Mail Notifications at Parent and Child Levels for Request Sets
  • Save and Re-use Concurrent Program Parameters (CP Templates)
  • Regex Support in Profile Option Values and FND_MESSAGES
  • Multiple OA Framework Windows and Tabs in Multiple Browser Windows
  • EBS Healthcheck Diagnostics
  • 64-bit JDK’s on CM Tier
  • adop Improvements
  • Delta Sync Mode (syncmode=patch/delta) – changed rows only
  • Rsync support in phase=prepare
  • Custom module sync auto-supported (no custom drivers req’d)
  • Standardized logging SessionID/ExecutionID/phase/node

EBS 12.2.6.AD Oracle E-Business Suite 12.2.6 Readme (Doc ID 2114016.1)

  • Security Configuration Console
  • JSP Whitelisting
  • Redirection Whitelisting
  • AD/TKX.Delta.8 incorporated
  • More HTML User Interfaces: HTML alternatives to several existing Forms; interfaces offer simpler layouts and more modern user interactions.

–Examples include the Update Assets UI in Enterprise Asset Management, and AOL setup UIs for Users, Responsibilities, and Flexfields.

SecureFiles does not allow shrinking of the tablespace after deletions (AP Imaging)

Available Interconnect Cloud Service adapters:

  • –PL/SQL
  • –Java Beans
  • –Apps Modules

Roadmap ICS adapters:

  • –REST
  • –Concurrent Programs
  • –XML Gateway
  • –BPEL/Business Services

CAAS (Cloud Adaptive Assistant Service) – The widget Larry was text-chatting with.

Mobile Foundation and Mobile Application Framework

Foundation v6 released August 2016

–Incorporates MAF 2.3.2

–Enhancements for Approvals (HR, Projects, Service Centers, Quoting, Custom Approvals)

Foundation v7 scheduled for April 2017

MAF Roadmap items:

–Airwatch and MobileIron support

–Mobile REST API’s

–EBS Integrated Service Gateway support (SOA)

PDF Version for download:       oracle_oow16_summary_jlui

Oracle Configuration Manager Quick Tip – EBS R12.2 Disconnected Collections

Collecting Disconnected Mode OCM data for e-Business Suite R12.2
per MOS Document ID 1447334.1 - Oracle E-Business Suite Release 12.2: Release Notes for Oracle Configuration Manager
My Oracle Support - Download Collector page
MOS Oracle Configuration Manager Collector Download screen.
This method is used whenever your EBS instance is behind a firewall and no proxy server is available to reach the OCM collector end-point (ccr.oracle.com)

The reason you want to use OCM is not only because it provides a complete catalog of your Oracle software versions, modules and plug-ins, so that they can easily be associated with a Service Request (SR) instead of the analyst asking for every RDA script being run for every component in your application and database. 

It also enables the full functionality of that Systems Tab in My Oracle Support (buried just above your Settings menu) that allows you to do light versions of lifecycle management comparisons (such as what the instance looked like 2 weeks ago versus now) without having to own the OEM Lifecycle Management license. It comes compliments of your support agreement.

This also enables the full functionality of the patch and security recommendations section vital towards identifying high priority one-off patches specific to your particular configuration and platform.

Basically, instead of having a cron job periodically automatically upload the configurations to ccr.oracle.com, you will be creating the same staged configuration file, but uploading it to your Customer Support Identifier (CSI) via a Draft Service Request (SR) or a currently Open SR via the File Attachment... feature of the SRs.

In the background of MOS is a managed file transfer handler that looks for the specifically named "ocmconfig.jar" files and when detected, route them over to the Configuration Manager engine for automatic uploading to your CSI account.

While not as convenient as the automatic cron method, for those of use with no direct internet connectivity to ccr.oracle.com, our OEM and MOS functionalities become quite limited if the configurations aren't present under our CSI's.e.g. ORACLE_BASE=/u01/app/oracle
# Oracle DB Home cd $ORACLE_HOME unset ORACLE_CONFIG_HOME export JAVA_HOME=$ORACLE_HOME/jdk cd $ORACLE_HOME/ccr # Clear prior CCR installation, if required. rm -rf $ORACLE_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip cd $ORACLE_HOME/ccr/bin # Run setupCCR in Disconnected mode ./setupCCR -s -d # Create ORACLE_OCM user and related grants . $ORACLE_HOME/ccr/admin/scripts/installCCRSQL.sh collectconfig -s <SID> -r SYS # Additional grants for APPS EBS Collector . $ORACLE_HOME/ccr/admin/scripts/installCCRSQL.sh ebs_collectconfig -u <APPSUSERNAME> # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $ORACLE_HOME/ccr/hosts/$HOSTNAME/state/upload # Oracle 10.1.2 iAS Home . $ORACLE_BASE/EBSapps.env run export ORACLE_HOME=$ORACLE_HOME export ORACLE_CONFIG_HOME=$INST_TOP/ora/10.1.2 export JAVA_HOME=$ORACLE_HOME/jdk # Remove prior OCM config, if needed rm -rf $ORACLE_HOME/ccr/* rm -rf $ORACLE_CONFIG_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip # Bugfix specific to HP-UX Itanium - verify file permissions to execute chmod 754 $ORACLE_HOME/ccr/bin/deployPackages chmod 754 $ORACLE_HOME/ccr/bin/emSnapshotEnv cd $ORACLE_HOME/ccr/bin ./setupCCR -s -d # Disconnected mode # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $INST_TOP/ora/10.1.2/ccr/state/upload #Repeat for patch filesystem; . $ORACLE_BASE/EBSapps.env patch # Oracle FMW WebLogic Home # source the Domain environment . $ORACLE_BASE/EBSapps.env run . $EBS_DOMAIN_HOME/bin/setDomainEnv.sh export ORACLE_HOME=$FMW_HOME/utils export ORACLE_CONFIG_HOME=$INST_TOP/ora/FMW # Remove prior OCM config, if needed rm -rf $ORACLE_HOME/ccr/* rm -rf $ORACLE_CONFIG_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip # Bugfix specific to HP-UX Itanium - verify file permissions to execute chmod 754 $ORACLE_HOME/ccr/bin/deployPackages chmod 754 $ORACLE_HOME/ccr/bin/emSnapshotEnv cd $ORACLE_HOME/ccr/bin ./setupCCR -s -d # Disconnected mode # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $INST_TOP/FMW/ccr/state/upload #Repeat for patch filesystem; . $ORACLE_BASE/EBSapps.env patch # Upload all of your ocmconfig.jar files (can be named uniquely - e.g. mydbocmconfig.jar) per MOS Document ID 763142.1 - How to upload the collection file ocmconfig.jar to My Oracle Support for Oracle Configuration Manager (OCM) running in Disconnected Mode.
# Also can ZIP up all of them at once and upload them to your Draft SR (or current open SR as long as you're happy with the CSI (Customer Support Identifier) that will receive the new configurations.
# Trick is have the string "ocmconfig" and .jar as the extension of each file
Have fun with your new Systems tab in My Oracle Support!

@Oracle R12 EBS HTTP Server Error Child Could Not Open Sslmutex Lockfile

HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).

mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/10.1.3.1/Apache/Apache/logs/ssl_mutex.10549 (System error follows)
System: No such file or directory (errno: 2)

And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).

applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: ENV_ID.myhostname
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status 
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   16788 | Alive  
OC4JGroup:default_group          | OC4J:forms         |   16709 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |   21689 | Alive  
HTTP_Server                      | HTTP_Server        |   16494 | Stop

OEM will show the HTTP service as being down, even though the pages are probably still responding:

R12 EBS OEM AMS monitor HTTP Service down screenshot
R12 EBS OEM AMS monitor HTTP Service down screenshot

Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.

These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration.  These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:

# ###############################################################
#
# This file is automatically generated by AutoConfig.  It will be read and
# overwritten.  If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
)

## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop

#  Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *

ssl.conf:SSLMutex  file:/ptcharmk/inst/apps/${ENV_ID}/logs/ora/10.1.3/Apache/ssl_mutex

# Append a line to the existing custom template
echo -e “\nAcceptMutex pthread\n” >> $FND_TOP/admin/template/custom_conf_1013.tmp

# Modify the existing SSL template (doesn’t seem to have a custom include on this one)
vi $FND_TOP/admin/template/ssl_conf_1013.tmp
=============

Comment:  # SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
Insert:     SSLMutex sem

# Execute AutoConfig

$SCRIPT_TOP/adautocfg.sh -appspass=$APPS_PW

# Re-Start Apache
$SCRIPT_TOP/adapcctl.sh start

# After Change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep -i utex $INST_TOP/ora/10.1.3/Apache/Apache/conf/*

custom.conf:AcceptMutex pthread
ssl.conf:# SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
ssl.conf:SSLMutex  sem

# Confirm semaphore mode is activated

applmgr@myhostname $> grep -i mutex $INST_TOP/logs/ora/10.1.3/Apache/*
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 11:58:14 2015] [notice] Accept mutex: pthread (Default: fcntl)
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 12:12:12 2015] [notice] Accept mutex: pthread (Default: fcntl)

As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.

R12 e-Business Suite and OEM Monitoring – Oracle Spins Freezes

Every so often, system load on an e-Business Suite instance ramps up and response time to users starts climbing, often resulting in user observed errors such as:

  • FRM-92100 Your connect to the server has been interrupted
  • FRM-92102 A network error has occurred

    FRM-92102 Forms Error R12 EBS
    That dreaded FRM-91201 / FRM-91200 error causing you to restart your session.

Or sometimes, the screen just freezes (aka spins, stops, is broken, stuck, motionless, looks like a screen saver,can’t do anything, won’t work, froze-up, etc.) and the person has to close their browser, or even shut-down their workstation and restart.

It's simply not doing anything - Nothing to see here, just move your cursor around. And wait... and wait.
It’s simply not doing anything – Nothing to see here, just move your cursor around. And wait… and wait.

Old technology often barks with unrelated error messages to the actual cause.  If there’s a lot going on with concurrent requests, or interfaces, or analytic extracts running, the front-end response-time slows down, sometimes sufficiently to trigger these kinds of Form errors, even though technically there was no interruption to the network connectivity, either between the hosts, nor the workstation and the middle-tier application server.

However, on the database, the user-experience can be seen, although not necessarily in the place you might expect.  OEM  had introduced it’s Adaptive Metric Thresholds technology back in OEM 11g (in a slightly different place than in 12c (in Oracle Management Server/OMS 12.1.0.4.0).  In OEM 11g, they were a link under the AWR Baseline Reports page.

OEM 11g AWR Baselines Page
See the Baseline Metric Thresholds link at the bottom.

In OEM 12c, you’ll find them under

Targets -> Database -> Peformance -> Adaptive Thresholds -> Baseline Metric Thresholds -> Edit Thresholds:

OEM 12c Baseline Metric Thresholds
Where those adaptive metric thresholds moved in 12c.

 

 

On this page and in the list of Baseline Metrics, when you click into them, you can access the trending statistics being gathered for each metric.  Many times this will provide direct insight into what a user experiences as the “the system is frozen” translates into “the back-end database response time is incredibly bad.”

OEM 12c Baseline Metric Response Time per Transaction vs. Baseline
See the spikes around 7AM and 11:30AM? Those are being associated with “System Froze” reports.

 

In the example here, the database experienced a dramatic slow-down in response almost 5 to 10 times slower than usual, which only lasted a few seconds. But that can be enough to show up in many users’ sessions who might have just kicked off a query, or were trying to save something.  Based upon the information gathered, we set the Warning and Critical thresholds to 1500ms and 2000ms respectively to start sending e-mail alert notifications upon breach of the levels. If the settings are left at “None”, no incident would be raised, and thus, no notification would be sent.

If you’re experiencing odd transient outages or sluggish behavior that defies the normal AWR and ADDM snapshot analysis, go take a look at what OEM has been gathering in the background over time and see if the statistics correlate to any of your issues.  There’s value in that data. Just mine it.

Using Demo Data for Oracle Data Mining Tools

With the forthcoming (but already available) SQLDeveloper 4.1 edition, an improved version of the Oracle Data Miner tools is incorporated into the SQLDeveloper console.  However, I found that there were a number of steps needed to actually use this new data modeling product other than just responding ‘Yes’ to the “Do you wish to enable the Data Miner Repository on this database?” prompt.

Here’s what I ended up doing to get things up and running (so that I could play with data modeling and visualization using Excel and the new SQLDeveloper DM extensions.)

#In this case, I’m adding back the demonstration data (i.e. EMP, DEPTNO type tables; the SH, OE, HR, et.al. schemas) into an existing R12 e-Business Suite (12.1.3) instance.

# Installing the Oracle Demo data in an R12 instance.

# Use the runInstaller from the R12 $ORACLE_HOME
cd $ORACLE_HOME/oui/bin
export DISPLAY=<workstation IP>:0.0
./runInstaller

# Choose the source products.xml from the staging area – Download and stage the DB Examples CD from OTN
/mnt/nfs/database/11203/examples/stage/products.xml

# Complete the OUI installation through [Finish]

cd $ORACLE_HOME/demo/schema
mkdir -p $ORACLE_HOME/demo/schema/log
echo $ORACLE_HOME/demo/schema/log/     ## used to respond to the Log Directory prompt during mksample.sql

sqlplus “/ as sysdba”
— will need passwords for: SYS/SYSTEM and APPS (used for all of the demo schemas, some of which pre-exist such as, HR, OE (PM, IX, SH and BI were okay for 12.1.3).

— ## Be sure to comment out any DROP USER <HR, OE, etc.) commands in this script (or you will be restoring your EBS instance from a backup because it just dropped your Module schema tables…) ##
— They look like this:
/*
mksample.sql:– DROP USER hr CASCADE;
mksample.sql:– DROP USER oe CASCADE;
mksample.sql:DROP USER pm CASCADE;
mksample.sql:DROP USER ix CASCADE;
mksample.sql:DROP USER sh CASCADE;
mksample.sql:DROP USER bi CASCADE;
*/

–Similarly – if/when you decide you no longer need the data – do NOT just use the $ORACLE_HOME/demo/schema/drop_sch.sql script
–or you just dropped your HR/OE/BI EBS schemas; don’t do that.
/*
drop_sch.sql:PROMPT Dropping Sample Schemas
drop_sch.sql:– DROP USER hr CASCADE;
drop_sch.sql:– DROP USER oe CASCADE;
drop_sch.sql:DROP USER pm CASCADE;
drop_sch.sql:DROP USER ix CASCADE;
drop_sch.sql:DROP USER sh CASCADE;
drop_sch.sql:DROP USER bi CASCADE;

order_entry/oe_main.sql:– Dropping the user with all its objects
order_entry/oe_main.sql:– DROP USER oe CASCADE;
order_entry/oe_main.sql:– ALTER USER oe DEFAULT TABLESPACE &tbs QUOTA UNLIMITED ON &tbs;

*/
— in this instance the $APPS_PW is synchronized to all application module schemas (i.e. AR, HR, GL, etc.)
— log directory would be the actual path from echo $ORACLE_HOME/demo/schema/log/ (including the trailing slash)
SQL> @mksample.sql

# to additionally create the Data Mining user (DM in this case)

create user &&dmuser identified by &&dmuserpwd
default tablespace &&usertblspc
temporary tablespace &&temptblspc
quota unlimited on &&usertblspc;

GRANT CREATE JOB TO &&dmuser;
GRANT CREATE MINING MODEL TO &&dmuser;       — required for creating models
GRANT CREATE PROCEDURE TO &&dmuser;
GRANT CREATE SEQUENCE TO &&dmuser;
GRANT CREATE SESSION TO &&dmuser;
GRANT CREATE SYNONYM TO &&dmuser;
GRANT CREATE TABLE TO &&dmuser;
GRANT CREATE TYPE TO &&dmuser;
GRANT CREATE VIEW TO &&dmuser;
GRANT EXECUTE ON ctxsys.ctx_ddl TO &&dmuser;
GRANT CREATE ANY DIRECTORY TO &&dmuser;
— Grant the SH Demo table and package objects to the DM user
@?/rdbms/demo/dmshgrants.sql &&dmuser

connect &&dmuser/&&dmuserpwd
— Create the Data Mining Views against the  SH Demo table and package objects
@?/rdbms/demo/dmsh.sql

@?/rdbms/demo/dmabdemo.sql — Builds the Adaptive Baynes Model demo
@?/rdbms/demo/dmaidemo.sql — Builds the Attribute Importance demo
@?/rdbms/demo/dmardemo.sql — Builds the Association Rules demo
@?/rdbms/demo/dmdtdemo.sql — Builds the Decision Tree demo
@?/rdbms/demo/dmdtxvlddemo.sql — Builds the Cross Validation demo
@?/rdbms/demo/dmglcdem.sql — Builds the Generalized Linear model demo
@?/rdbms/demo/dmglrdem.sql — Builds the General Linear Regression model demo
@?/rdbms/demo/dmhpdemo.sql — not a Data Mining program – Hierarchical Profiler
@?/rdbms/demo/dmkmdemo.sql — Builds the K-Means Clustering model demo
@?/rdbms/demo/dmnbdemo.sql — Builds the Naive Baynes Model data
@?/rdbms/demo/dmnmdemo.sql — Builds the Non-negative Matrix Factorization model
@?/rdbms/demo/dmocdemo.sql — Builds the O-Cluster model Demo
@?/rdbms/demo/dmsvcdem.sql — Builds the Support Vector Machine model demo
@?/rdbms/demo/dmsvodem.sql — Builds the One-Class Support Vector Machine model demo
@?/rdbms/demo/dmsvrdem.sql — Builds the Support Vector Regression model demo
@?/rdbms/demo/dmtxtfe.sql — Builds the Oracle Text Term Feature Extractor demo
@?/rdbms/demo/dmtxtnmf.sql — Builds the Text Mining Non-Negative Matrix Factorization model demo
@?/rdbms/demo/dmtxtsvm.sql — Builds the Text Mining Support Vector Machine model demo

## End of Data Mining Demo user (DM) setup and configuration for use of Oracle Demo Data

Java 7 (1.7.0_) and Java 6 on the Same Workstation for R12 e-Business Suite EBS and other Apps

The challenge: We have a number of 3rd party Cloud applications that are still on Java 6, so we have that version (1.6.0_26 specifically) deployed to our workstations. We configured R12 to use 1.6.0_26 on the server side to minimize impact to the users. We are now implementing Informatica Information Lifecycle Management (an archive and purge technology), which has a component (the Enterprise Data Manager – EDM) which is a Java 7-based application used by certain administrative and analyst users. We needed to have those users support both Java 7 and Java 6 on the same workstation for the browsers (Internet Explorer, Firefox, Chrome).

Starting with a Windows 7 (32-bit) workstation with a working Java 1.6.0_26 installed (and 1.6.0_7 disabled).

Used the Java SE 7 SDK from the Oracle Java Archive (staged to the DBA Share /Software/Java because the 7 downloadables will be pulled from the site as of April-2015):

http://www.oracle.com/technetwork/java/javase/archive-139210.html

Chose: Java SE Development Kit 7u25 (includes the JRE runtime edition) – newer versions (post- and including update 40) contain an non-bypassable Security Warning message for using Java 6 applications.

Installed Java 7 using default parameters and completed successfully.

Open the Java Control Panel (available in the Toolbar, as the Java coffee cup icon, if a Java application is already running – e.g. EBS; or via Control Panel -> Java)

Preferred Enabled settings under the [Java] tab -> View…

06a_java_cpanel

06_java_7_and_6_versions_cpanel

De-select the Java Plug-in (next-generation) features (due to co-existence with Java 6) – we also turn off Automatic Java Updates, and default the automatic JNLP MIME/execution association setting to make it easier for users:

07_java_7_adv_tab_disable_next_gen_cpanel

Launch Informatica EDM:

04_java_launch_ilm_edm_ss

Java Update Needed – Check (Do not ask again) -> Select -> Later

01_java_upd_needed_ss

Choose default launcher for EDM’s JNLP (also can be defaulted in the Java Control Panel -> Advanced settings)

05_java_launch_ilm_edm_pick_app_ss

EDM Launches:

02_java_launch_edm_ss

03_java_launch_edm__success_ss

Launching R12 e-Business Suite under Java 6:

Launch Forms Session (Profile -> System used as an example)

Security Warning – due to Java having incomplete Certificate Authority trust to the Aramark Hosts (can be addressed also via Control Panel -> Java -> [Security] -> Manage Certificates -> Import Certificate -> add certificates provided by AUS Server Team (password required).

The Java 7 update 25 warning looks like this (click the Activate Java link; then on the next pop-up click Allow and Remember – to suppress future warning messages):

01b_java_upd_needed_pre40_ss

The post-Java 7 update 40 warning looks like this (select – I accept the risk; click [Run]):

08_java_launch_ebs_sec_warn_ss

Ref: http://java.com/en/download/faq/signed_code.xml

Why don’t I see the option to select Do not show this again for this app in the security dialog for an unsigned application?

Starting with Java 7 Update 40, the option to select Do not show this again for this app is no longer available. Unlike previous versions a user cannot suppress the security dialog for an unsigned application and will have to select the option, I accept the risk and want to run this app, each time to run the unsigned application.

Security Warning – these will come up twice in a row – once when the R12 forms servlet is instantiating and a second time when the actual Forms launch. Once Allow this application to run with the requested version (1.6.0_26) is selected, these warnings should be suppressed in future sessions.

09_java_launch_ebs_sec_warn2_run_6_ss

R12 and EDM simultaneously launched successfully.

11_java_launch_ebs_ilm_success_ss

R12.2.3 / R12.2.x Oracle e-Business Suite – Resetting the WebLogic Password

…because occasionally you either forget it (or in my case, a technology patch reset it to something I wasn’t sure about.)wls_denied_ss

There are 2 basic routes:


The “short” process – re-run the post-clone config (adcfgclone.pl):

# Have to remove the existing FMW home because it will re-unzip a new one from the clone staging archive
applptch@aburpaofinm01 $> cd /u01/fs2
applptch@aburpaofinm01 $> ls
EBSapps  FMW_Home  inst
applptch@aburpaofinm01 $> ls -latr
total 24
drwxrwxr-x  8 applptch oinstall 4096 May  8 21:49 ..
drwxr-xr-x  5 applptch oinstall 4096 May  9 12:33 EBSapps
drwxr-xr-x  3 applptch oinstall 4096 Jul 16 15:30 inst
drwxr-xr-x  6 applptch oinstall 4096 Jul 16 15:41 .
drwxr-x— 13 applptch oinstall 4096 Jul 16 16:16 FMW_Home
applptch@aburpaofinm01 $> rm -rf FMW_HOME_BAK
# Check that the Oracle Homes are not in the OraInventory
# Or you will receive the error:
# Below Oracle Homes are already registered in the global inventory:
#/u01/fs2/FMW_Home/Oracle_EBS-app1
#/u01/fs2/FMW_Home/webtier
#/u01/fs2/FMW_Home/oracle_common
#
# Exiting Cloning…
#
# Ensure that the above Oracle Homes are unregistered from the global inventory “/u01/oraInventory/ContentsXML/inventory.xml” and re-run adcfgclone.pl script
vi /u01/oraInventory/ContentsXML/inventory.xml
# and remove the <HOME> through </HOME> entries for the 3 listed entries
# Begin the clone – needs the APPS password
applptch@aburpaofinm01 $> cd $COMMON_TOP/clone/bin
applptch@aburpaofinm01 $> perl adcfgclone.pl appsTier


The “long” process – by manually manipulating the admin account in the /u01/<fs#>/FMW_Home:

# Verify your Weblogic Environment home
applptch@aburpaofinm01 $> set | grep EBS

EBS_DOMAIN_HOME=/u01/fs2/FMW_Home/user_projects/domains/EBS_domain_PTCHARMK

# Stop Weblogic
$ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PW
Weblogic Password: [anything because you don’t know it]

# Set your environment
cd $EBS_DOMAIN_HOME/bin
. setDomainEnv.sh

# Remove the existing LDIFT’s, boot.properties and LDAP directory
rm -rf $EBS_DOMAIN_HOME/security/DefaultAuthenticatorInit.ldift ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/ldap

# Start AdminServer from command line manually
cd $EBS_DOMAIN_HOME ;\
startWebLogic.sh

# Enter new credentials as above
<Sep 16, 2014 4:58:09 PM PDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:

Change NodeManager password

Log in to the WebLogic Administration console.
Click the Lock & Edit button.
In the left panel, click on the EBS Domain link.
Select the Security tab.
Click on the ‘Advanced’ link.
Edit the ‘Node Manager password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Edit the ‘Confirm NodeManager Password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Save and activate the changes.

Shut down AdminServer from the console. For the first time, AdminServer has to be stopped from the Admin console. Follow these steps:

Log in to the WebLogic Administration console.
Shut down AdminServer.

Shut down NodeManager using the normal AD script.

$ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

Copy the boot.properties file for all managed servers.

cp $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties \
$EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties

The EBS WebLogic Server domain password has now been changed, and all servers can now be started using the normal AD scripts.

To start AdminServer:

$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

# Start it all back up again
$ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PW
Weblogic Password: [weblogic pwd]

Now your password change is complete (or at least you can get back into your Admin Console).

# Clone the working result from the run filesystem to the patch filesystem
$ adop phase=fs_clone

** Based upon a regular WebLogic post by:

http://shivabizint.wordpress.com/2012/10/15/how-to-reset-the-lost-weblogic-admin-user-password/

**And an R12.2 version by:

http://nagendrachillale.wordpress.com/2014/08/22/if-the-admin-password-of-an-ebs-weblogic-domain-is-lost-or-forgotten/