Relatively easy installation, a new learning curve to master for post-installation maintenance and patching.
The principal changes in Release 2 of 13c are the segregation of the Cloud Management Services plugins (and related functionality) to specific licensable plugin components – namely:
Cloud Services Management – used for provisioning, allocation, and administration (think “infrastructure”) of Oracle Public Cloud-based resources (databases, development tools, middleware, etc.)
Oracle Cloud Application – a portal punch-out to access Oracle Cloud-based applications such as hosted e-Business Suite, Cloud Content Management, Cloud SOA Suite, Cloud Identity Management, etc. (think “apps”.)
This proof-of-concept testbed system was based upon the latest available installer set – Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (126.96.36.199) – we happen to be testing backwards-compatibility with some 10g databases before committing to the upgrade. Direct (out-of-place) upgrades are supported for version of EM 188.8.131.52 and newer.
Fusion Middleware in this release is Weblogic 12.1.3
The host system is built upon Windows Server 2012 R2, single-tier, 16GB RAM, 2 cores, also hosting the OMS repository database in a 184.108.40.206.0 EE installation. Since the DB Templates provided seemed to be specific for a 12.1.x database (and errors out in certain steps during the build), I simply created a generic database from the standard template with the pre-requisite parameter settings (e.g. _allow_insert_with_update_check=TRUE; sessions > 300; shared_pool_size (10% of SGA); etc.) , and pre-created the required tablespaces:
During the installation, the (Oracle Universal Installer) OUI will create the following new users assigned to these tablespaces:
The standard download set includes a single executable file (.exe or .bin) and several additional ZIP archives. These are to be staged in a single directory, and the ZIP files do NOT need to be pre-extracted.
Overall, the installation was straight-forward and as-documented (despite this being a Windows installation – which usually has its own quirks.) The portion that I feel has the steepest learning curve is post-installation maintenance, and particularly patching. The new tool in 13c, the OMSpatcher (Patchset 19999993 – think of it as a Java wrapper for opatch) was really designed for use specficially in a cloud-based environment and hasn’t been completely polished for use on-premise in smaller installations.
For example, it is designed to rely upon an active Weblogic AdminServer to instantiate it’s requests for inventory versioning information and component availability for patching. But if any of those components are actually in-use by the AdminServer itself, it will tend to fail to apply, and instead provide a lengthy step-by-step instruction set of how to accomplish all of the steps manually. Depending on the complexity of your OEM setup (e.g. the number of registered plugins and target types) this could take over an hour to get the patching utility to fail and then provide the manual steps.
The OEM-specific version of OPatch (Patchset 6880880) is quite different than its predecessors. While the OMSPatcher uses the prior “unzip into the existing Oracle Home” installation technique, the OPatch utility itself now is a Java installer that relies on the OUI to allow updating of component versioning and inventory adjustments. Most importantly, you will need to “install” the new version of OPatch before being allowed to continue with your planned patching:
$JAVA_PATH\java -jar $PATCH_STAGE\6880880\opatch_generic.jar -silent oracle_home=$ORACLE_HOME (being updated for OPatch)
This OPatch update needs to be applied (before patching) to every Oracle Home using 13c technology (the OMS home, the Agent Home, other FMW homes at version 12.1.3 or newer, etc.)
See MOS EM 13c: How to Apply a Patch to the Enterprise Manager 13c Cloud Control OMS Oracle Home (Doc ID 2091619.1) for details about making an OMSPatcher property file, and why you want to create one.
Final steps executed:
Build general purpose 220.127.116.11.0 database with new tablespaces.
Install Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (18.104.22.168)
Patch 19999993: EM OMSPatcher latest version 22.214.171.124.2 (unzip to ORACLE_HOME)
Patch 6880880: EM OPatch latest version 126.96.36.199.0 (both OMS and Agent)
$FMW_HOME\OMSPatcher\omspatcher apply -analyze -property_file
$FMW_HOME\OMSPatcher\omspatcher apply -property_file [OMS_DISABLE_HOST_CHECK=true -- a useful option added to deal with virtual host names]
Patch 25163555: Tracking bug for Back-porting 24588124 oms side fix
Patch 25604219: MERGE REQUEST ON TOP OF 188.8.131.52.0 FOR BUGS 25497622 25497731 25506784
Patch 25387277: APR-2017 PSU OMS 184.108.40.206.170418
Patch 25162444: EM-BEACON Bundle Patch 220.127.116.11.161231 (Agent)
Patch 25580746: EM-AGENT Bundle Patch 18.104.22.168.170331 (Agent)
Other useful references:
Enterprise Manager 13.2 Master Bundle Patch List (Doc ID 2219797.1)
22.214.171.124.170418 Enterprise Manager Base Platform Patch Set Update (PSU) Readme for Oracle Management Server (OMS) (Doc ID 2246778.1)
Registering the Integrated SOA Gateway included in the Oracle E-Business Suite R12.2.x instances is actually reasonably simple, but the documents make it sound much worse than it really is, simply by being several hundred pages of documentation.
The reason to do this is to expose the RESTful services interface for EBS so that other web applications can do useful things, such as check usernames for validity, lookup business process (workflow) statuses, and even query for completion of processes, such as concurrent programs. The built-in library of accessable Services is quite vast and enables lots of interconnectivity with other applications.
This is also useful if you happen to be extending the application using the Mobile Application Framework, which allows you to build custom applications for mobile devices that access all the various EBS functionalities.
NOTE:1311068.1 – Installing Oracle E-Business Suite Integrated SOA Gateway, Release 12.2
NOTE:1317697.1 – Oracle E-Business Suite Integrated SOA Gateway Troubleshooting Guide, Release 12.2
Before commencing, have the following prepared:
Unlock the ASADMIN (FND_USER) account and set the password so that it can login without a change password challenge (no expiry. required.
IP Address of ApplTier Host
Ports for ApplTier Host (both base port and WLS console – e.g. 8000 7001)
Your base port is the one in your usual EBS URL and the WLS console URL can be determined via Oracle Application Manager in the Site Map link for Weblogic Administration.
Services will be interrupted by bounce twice (2X) during installation.
Installation takes about 45 minutes to complete.
==================================== Start of SOA Integration Steps
#Register new Internal SOA Gateway Configuration
itsrv33m [/export/home/dbausr/oradev] ant -f $JAVA_TOP/oracle/apps/fnd/txk/util/txkSoaConfigUtility.xml
[input] Enter Oracle WebLogic Server Administration Console URL (host:port) :
[input] Enter Oracle WebLogic Server Admin User Name : [weblogic]
[input] Enter the password for user weblogic :
[input] Enter the password for user apps :
[input] Enter the ASADMIN user name : [ASADMIN]
[input] Enter the password for user ASADMIN :
[input] Enter SOA Server Hostname : (hostname.domain)
[input] Enter External URL of SOA Suite SOA Managed Server (protocol://managed_server_hostname:managed_server_port) : http://%5BFQEBShostname%5D:%5Bport%5D
FND: Validate IP address|02-NOV-16|Site||D
FND: Validate User Type|02-NOV-16|Site||D
# Bounce certain services (minimizes downtime – can also bounce everything)
admanagedsrvctl.sh stop oafm_server1
admanagedsrvctl.sh start oafm_server1
admanagedsrvctl.sh stop oacore_server1
admanagedsrvctl.sh start oacore_server1
[etc for any other oacore servers]
#Verify registered payload Class ID (ensures payload can be accessed)
SQL Statement which produced this data:
WHERE UPPER (irep_name) LIKE ‘%EBIZHOMEPAGESERVICE%’
AND class_type = ‘SERVICEBEAN’;
# Configure the ISG BPEL Payload for use by SOA Suite
ant -f $JAVA_TOP/oracle/apps/fnd/bpel/util/txk_integration_packager.xml \ -DIntegration=isg -DFMWNodeName=itsrv33m.mwd.h2o -DFMWIpAddress=[MT or Virtual host IP]
[echo] Bpel Payload is archived at $INST_TOP/logs/appl/rgf/TXK/EBS_BPEL_payload.zip
# Verify the “new” SOA Node is registred (MWD is just re-using the built in SOA)
SQL Statement which produced this data:
SELECT node_name, server_address, server_id
WHERE UPPER(node_name) in(‘[hostname]’);
# Warning: the following command will immediately shutdown and restart the WLS Managed Services
ant -f $JAVA_TOP/oracle/apps/fnd/txk/util/txkISGConfigurator.xml ebsSetup
# Change default NAME=APPS connection to NAME=ASADMIN in WLS Admin Console.
# (you want to do this to support APPS password changes in the future – the ASADMIN
# password does not have to be disclosed to Developers once the setup is completed – just
# grant the Integrated SOA Gateway Role and Responsibility to users that need to deploy
# various services.
Data Sources -> OAEADatasource -> Configuration -> Connection Pool
[Lock & Edit]
Modify Properties user=APPS -> user=ASADMIN
Enter ASADMIN password in both password fields
[Save] -> [Activate Changes]
If the ASADMIN password is changed in the target EBS instance, then all of the following steps need to be re-executed in order to register the change with SOA:
# Clean out the prior OAEADatasource connections and definitions
Login to WLS Administration Console.
Services -> Data Sources -> [Lock & Edit] -> Select OAEADatasource -> [Delete]
Security Realms -> myrealm -> Providers -> Select IsgAuthenticator -> [Delete]
$] cd $DOMAIN_HOME/config/jdbc
$] rm OAEADatasource-*-jdbc.xml
Repeat the SOA Integration steps again.
Otherwise, attempting REST deployments will receive an error:
java.lang.SecurityException: User: ASADMIN~~[ORACLE_SID], failed to be authenticated.
Oracle Enterprise Manager 13c includes several enhancements including:
A new Oracle Fusion Middleware technology stack
The “Alta” user interface to enhance user experience
Edition Based Redefinition (EBR) support
Hybrid Cloud Managment
Oracle Application Management Suite now delivers several capabilities to monitor and manage Oracle E-Business Suite environments on Oracle Cloud and on-premises. These capabilities are delivered through the Enterprise Manager command line interface (EMCLI) for greater flexibility.
Discovering Orchestration Virtual Machine
Provisioning a New Oracle E-Business Suite 12.2 Instance
Backup and Restore an Oracle E-Business Suite Instance
Lift and Shift E-Business Suite instances from On-premise to Oracle Cloud
Cloning E-Business Suite instances within Oracle Cloud
Ability to run FS_Clone only option
Apply E-Business Suite patches
Start and Stop E-Business Suite instances on Cloud
To create custom packages
System Managment for Oracle E-Business Suite
Discovery and Monitoring
Real User Experience Insight (RUEI) regions integrated within the Oracle E-Business Suite Summary page
In this release, we have integrated five different RUEI regions within the Oracle E-Business Suite Summary page. Customers can personalize and activate one or all RUEI regions. These RUEI regions are specific to a given Oracle E-Business Suite instance and customers can view the data and drill down for detailed information.
Auto Host Aliasing
Host aliasing is now automated and will attempt to match the host name in the Oracle E-Business Suite context file and the discovered host name in Oracle Enterprise Manager. If this attempt fails, you will be prompted to perform the mapping manually. Customization Manager and Patch Manager features now support host aliasing.
Change Management for Oracle E-Business Suite
Patch Manager – Oracle E-Business Suite Technology Stack Pack Deployments
During the prior release of Application Management Suite, you had the ability to view patch recommendations for E-Business Suite applications and the technology stack. In this release, you now have the ability to deploy Database and WebLogic server patches using Enterprise Manager patch plans.
The Applications Management Suite plug-in for Oracle Enterprise Manager simplifies discovery of the myriad of subtargets that make up an Oracle E-Business Suite instance. Correspondingly, the number of alerts sent out can rise dramatically because of the inter-relationships between the components.
For example, the outage of a single Apache process triggers all of the following associated targets also to flag as a service Down status:
INSTANCE-Oracle E-Business Suite
INSTANCE-Infrastructure INSTANCE_host-APPL_TOP Context
The number of downed targets increases if a subcomponent of a primary component (such as a single JVM thread under the OACore process) experiences an outage.
This is a simple walk-through of navigating one of the e-mail alerts to start figuring out what happened.
The e-mail alert looks like this:
From: OEM12 Burbank Sent: Monday, December 07, 2015 8:30 AM To: DBAs Subject: EM Event: Fatal:INSTANCE-Oracle E-Business Suite – Target is down; 1 member is down: INSTANCE_EBS Availability System
Target type=Oracle E-Business Suite
Target name=INSTANCE-Oracle E-Business Suite
Message=Target is down; 1 member is down: INSTANCE_EBS Availability System
Event reported time=Dec 7, 2015 8:29:14 AM PST
Target Lifecycle Status=Production
Associated Incident Id=390885
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Target Availability
Root Cause Analysis Status=Symptom
Rule Name=EBS Notifications,Rule_EBS_Notifications (to get notified, you set up Rule Sets that tell OEM when and what to notify you about)
Target is down; 1 member is down: INSTANCE_EBS Availability System
Incident created by rule (Name = Incident management rule set for all targets, Incident creation rule for a Target Down availability status [System generated rule]).
To investigate an event alert, click on the Associated Incident ID (e.g. the 390885 which on your system will be a URL taking you into OEM) which will take you to the associated Incident Summary page.
Click on Related Events to investigate what raised the event alert (there may be more than one cause):
From the screen, it shows the red mark on PRODARMK-Infrastructure PRODARMK_ascopofinm01-APPL_TOP Context (Oracle E-Business Suite Node).
Click on that link in the list of Targets.
Navigate to Monitoring -> Status History:
Change the Availability History view to All History (the related underlying event caused is displayed.)
If you click on the related Message (e.g. Target is down; 1 member is down: INSTANCE_hostname.auca.corp_oacore_JVM_…); you will then be shown the related Event page for that target:
Click on the Related Events tab for this target, to confirm the service alert recorded:
If this is a recurring issue, by sliding the timeline back and forth (and adjusting the period view to a larger sample) you can see if there are any associated time-related occurrences that can be used to identify root cause.
For the specific issue, login to the associated host, and view the output and error logs for the process itself to determine what triggered the alert (in this case, the JVM automatically restarted the OACore process that had run out of memory.)
We had a unique encounter with the issue of Forms “freezing” on launch (also when seeing FRM-40734: Internal Error: PL/SQL Error occurred; FRM-92102: A network error has occurred)
One of the key debugging actions taken is the steps related to My Oracle Support (support.oracle.com) article:
R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications ( Doc ID 438652.1 )
From this screenshot, you can see that the Forms Server is actually successfully launching (that’s why you can see the Navigator and Menu Bar), but something is not populating the actual session-related information (the menu items, the functions available,etc.)
Since we are on Single-Sign On (SSO) you might be freezing instead just after the Forms Login screen.
Turned out to be incorrect settings for the UTL_FILE_DIR parameter for the database which had been set using an online ALTER SYSTEM SET command, but would not show up until the database was re-started. (nb UTL_FILE_DIR is notoriously picky about quotes and commas for multiple directories being specified.) The ALTER command should appear as (for 11g):
SQL> alter system set UTL_FILE_DIR=’dir1′,’dir2′,’dir3′
But often someone uses the following (which used to work in 8i, 9i, but is now invalid):
SQL> alter system set UTL_FILE_DIR=’dir1,dir2,dir3′
Once the DB was restarted, it ended up with an incorrect UTL_FILE_DIR parameter and was unable to write the session flag files to the expected temporary directory (usually APPLPTMP).
One of the DB Trace files produced during the attempt to login included:
/* Section below shows AXF framework extensions are in-use */
Note: This script is based upon an OEL 5.9 environment with a completed first-stage R12.2 e-Business Suite upgrade from 12.1.3. Using a pre-existing registered module (ZFA – Oracle Financial Analyzer in our case), this clones the settings required to fulfill the R12.2 upgrade pre-requisite check for additonal applications, which had been added since the 12.1.3 release.) It is referenced in the primary 12.2.3 Upgrade Post and fully listed herein.
If you Care a Little More, Things Happen. Bees can be dangerous. Always wear protective clothing when approaching or dealing with bees. Do not approach or handle bees without proper instruction and training.