What’s New Features in OEM Applications Management Pack 13.4(.2) Released 15-JUN-2020

Per Getting Started with Oracle Application Management Pack (AMP) for Oracle E-Business Suite, Release 13.4.1.0.0 (MOS Note 2676355.1)

New Features

Certification with Oracle Enterprise Manager 13.4 Release Update 2

Certification with Oracle Database 19c (for Discovery, Monitoring, Approvals, Patch Manager, and Customization Manager, only)

Support for Oracle Real User Experience Insight 13.3.1.0.2

References

Angelo Rosado’s Oracle Blog on OEM AMS/AMP Announcements:

https://blogs.oracle.com/ebstech/oracle-application-management-pack-release-134100-is-now-available

New EBS AMS Plugin Homepage Screenshot 13c 13.1.1.1.0

Key Features (very similar to 13.1.1.1.0):

System Management

Oracle Application Management Pack provides monitoring alerts and notifications for Oracle Application Framework, Concurrent Processing, Forms services, concurrent managers and requests, Oracle Workflow services, Oracle Database, and middle tier/web servers.

Standardize monitoring settings across your Oracle E-Business Suite environments by utilizing templates to initially specify the monitoring settings and apply those to your monitored targets. You can save, edit, and apply these templates across one or more targets. A monitoring template is specified for a particular target type and can only be applied to targets of the same type.

Use the Concurrent Processing dashboard for a complete picture of concurrent processing on your Oracle E-Business Suite system of both current activities as well as usage statistics. Add concurrent programs and managers as user-defined targets and the system will collect metrics on activity and usage for them.

Configuration Management

Oracle Application Management Pack collects and stores the technology configurations of Oracle E-Business Suite to help centralize monitoring and tracking of changes to Oracle E-Business Suite technology stack configurations including host configuration, database configuration, middleware configuration, patches applied, key profile option changes, versions of technology components, and custom object changes.

Compare configurations between two or more Oracle E-Business Suite systems directly or by comparing configuration snapshots taken at different time intervals.

Configuration templates help to standardize the configuration standards across the company and reduce the configuration drift between various Oracle E-Business Suite instances.

Administrators can track changes by setting up notifications when any unauthorized changes occur to technology stack configurations.

Compliance Management

Compliance framework integration allows you to ensure your Oracle E-Business Suite is compliant with your IT audit requirements, as well as industry and regulatory requirements. Compliance Management dashboards show trends and compliance violations. Proactive, real-time compliance monitoring can significantly reduce the operational risk to business.

Change Management

Change Approval Framework

All changes made with Oracle-delivered patches or customizations in Change Management are processed through a change approval mechanism. A multi-level, hierarchical list of approvers can be set up for each Oracle E-Business Suite target. Auditors can review the change requests as welll as the approvals.

Patch Management

Patch recommendations for Oracle E-Business Suite application products and the technology stack components are made for specific Oracle E-Business target instances to help reduce or eliminate the research work by administrators. These patch recommendations help to ensure all Oracle E-Business Suite systems are current with Oracle-recommended patches. You can deploy database and WLS patches by creating Oracle Enterprise Manager patch plans directly from the patch recommendations user interface.

Using the Patch Management dashboard, you can manage all patching activities of your Oracle E-Business Suite instances and view patch worker logs. Patch Manager simplifies the online patching process and reduces human intervention to a great extent, eliminating the need for continuous monitoring of the patching process.

Patch Manager makes it easier for administrators to check for prerequisite patches and to download the patches in offline mode or directly from My Oracle Support.

Customization Management

Create and deploy custom packages across Oracle E-Business Suite instances similar to Oracle-delivered patches. Customization Manager can help ensure all customizations follow Oracle development standards and/or user-defined customization standards specific to your IT organization. Easily integrate Customization Manager with any third-party source code version control software. This feature also supports the creation of National Language Support (NLS) patches.
Customization Manager can discover customizations within your Oracle E-Business Suite systems and provide detailed reports online or in a spreadsheet format. This helps your organization keep track of the inventory of customizations and use the data for further analysis.

Cloning

Automated Cloning allows you to clone your Oracle E-Business Suite applications for testing, training, or development purposes, leveraging the Oracle Enterprise Manager Cloud Control Provisioning Framework. A step-by-step interview guides an administrator through the cloning process and facilitates scheduling of Oracle E-Business Suite instance. Administrators can modify the standard cloning procedure to include pre- or post-custom steps, seamlessly automating their cloning process from beginning to end.

Instance Administration

Administrators can use the Instance Administration user interface when adding and removing (scale in and scale out) application services, application nodes, and managed servers.

Advertisement

Getting Started with OEM 13cR2 (13.2.2.0.0) Enterprise Manager

OEM13c Release 2 Enterprise Manager Summary Page

Relatively easy installation, a new learning curve to master for post-installation maintenance and patching.
The principal changes in Release 2 of 13c are the segregation of the Cloud Management Services plugins (and related functionality) to specific licensable plugin components – namely:

• Cloud Services Management – used for provisioning, allocation, and administration (think “infrastructure”) of Oracle Public Cloud-based resources (databases, development tools, middleware, etc.)
• Oracle Cloud Application – a portal punch-out to access Oracle Cloud-based applications such as hosted e-Business Suite, Cloud Content Management, Cloud SOA Suite, Cloud Identity Management, etc. (think “apps”.)
• Cloud Framework – a repackaging of the EM OMS framework components, and the only portion of the “cloud” components included in the base OEM licensing.
  Installation guide: http://www.oracle.com/technetwork/oem/grid-control/documentation/oem-091904.html

Cloud Management related plugins for OEM13cR2

This proof-of-concept testbed system was based upon the latest available installer set – Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (13.2.0.0) – we happen to be testing backwards-compatibility with some 10g databases before committing to the upgrade.  Direct (out-of-place) upgrades are supported for version of EM 12.1.0.4 and newer.

http://www.oracle.com/technetwork/oem/enterprise-manager/downloads

Fusion Middleware in this release is Weblogic 12.1.3
The host system is built upon Windows Server 2012 R2, single-tier, 16GB RAM, 2 cores, also hosting the OMS repository database in a 12.2.0.1.0 EE installation.  Since the DB Templates provided seemed to be specific for a 12.1.x database (and errors out in certain steps during the build), I simply created a generic database from the standard template with the pre-requisite parameter settings (e.g. _allow_insert_with_update_check=TRUE; sessions > 300; shared_pool_size (10% of SGA); etc.) , and pre-created the required tablespaces:

MGMT_AD4J_TS
MGMT_ECM_DEPOT_TS
MGMT_TABLESPACE

  During the installation, the (Oracle Universal Installer) OUI will create the following new users assigned to these tablespaces:

Open:
SYSMAN
SYSMAN_TYPES
SYSMAN_BIPLATFORM
SYSMAN_STB
SYSMAN_OPSS
SYSMAN_MDS
MGMT_VIEW

Expired & Locked:
CLOUD_ENGINE_USER
CLOUD_SWLIB_USER
EUS_ENGINE_USER
SYSMAN_RO

The standard download set includes a single executable file (.exe or .bin) and several additional ZIP archives.  These are to be staged in a single directory, and the ZIP files do NOT need to be pre-extracted.

Overall, the installation was straight-forward and as-documented (despite this being a Windows installation – which usually has its own quirks.) The portion that I feel has the steepest learning curve is post-installation maintenance, and particularly patching. The new tool in 13c, the OMSpatcher (Patchset 19999993 – think of it as a Java wrapper for opatch) was really designed for use specficially in a cloud-based environment and hasn’t been completely polished for use on-premise in smaller installations.

For example, it is designed to rely upon an active Weblogic AdminServer to instantiate it’s requests for inventory versioning information and component availability for patching. But if any of those components are actually in-use by the AdminServer itself, it will tend to fail to apply, and instead provide a lengthy step-by-step instruction set of how to accomplish all of the steps manually. Depending on the complexity of your OEM setup (e.g. the number of registered plugins and target types) this could take over an hour to get the patching utility to fail and then provide the manual steps.

The OEM-specific version of OPatch (Patchset 6880880) is quite different than its predecessors.  While the OMSPatcher uses the prior “unzip into the existing Oracle Home” installation technique, the OPatch utility itself now is a Java installer that relies on the OUI to allow updating of component versioning and inventory adjustments. Most importantly, you will need to “install” the new version of OPatch before being allowed to continue with your planned patching:

$JAVA_PATH\java -jar $PATCH_STAGE\6880880\opatch_generic.jar -silent oracle_home=$ORACLE_HOME (being updated for OPatch)

This OPatch update needs to be applied (before patching) to every Oracle Home using 13c technology (the OMS home, the Agent Home, other FMW homes at version 12.1.3 or newer, etc.)

See MOS EM 13c: How to Apply a Patch to the Enterprise Manager 13c Cloud Control OMS Oracle Home (Doc ID 2091619.1) for details about making an OMSPatcher property file, and why you want to create one.

Final steps executed:
Build general purpose 12.2.0.1.0 database with new tablespaces.
Install Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1 (13.2.0.0)
Patch 19999993: EM OMSPatcher latest version 13.8.0.0.2 (unzip to ORACLE_HOME)
Patch 6880880: EM OPatch latest version 13.9.0.0.0 (both OMS and Agent)

cd $PATCH_STAGE\<patch#>
$FMW_HOME\OMSPatcher\omspatcher apply -analyze -property_file 
$FMW_HOME\OMSPatcher\omspatcher apply -property_file  [OMS_DISABLE_HOST_CHECK=true  -- a useful option added to deal with virtual host names]

Patch 25163555: Tracking bug for Back-porting 24588124 oms side fix
Patch 25604219: MERGE REQUEST ON TOP OF 13.2.0.0.0 FOR BUGS 25497622 25497731 25506784
Patch 25387277: APR-2017 PSU OMS 13.2.0.0.170418
Patch 25162444: EM-BEACON Bundle Patch 13.2.0.0.161231 (Agent)
Patch 25580746: EM-AGENT Bundle Patch 13.2.0.0.170331 (Agent)

Other useful references:

Enterprise Manager 13.2 Master Bundle Patch List (Doc ID 2219797.1)
13.2.0.0.170418 Enterprise Manager Base Platform Patch Set Update (PSU) Readme for Oracle Management Server (OMS) (Doc ID 2246778.1)

What’s New Features in OEM AMS Applications Management Pack 13.1.1.1.0 for OEM 13c Released July 22

Per Getting Started with Oracle Application Management Pack (AMP) for Oracle E-Business Suite Release 13.1.1.1.0 (Doc ID 2045552.1)

Key Features

New EBS AMS Plugin Homepage Screenshot 13c 13.1.1.1.0

The following are brief descriptions of key features introduced in this release:

Enterprise Manager 13c Technology Uptake
Hybrid Cloud Management
System Management for Oracle E-Business Suite
Change Management for Oracle E-Business Suite

Enterprise Manager 13c Technology Uptake

Oracle Enterprise Manager 13c includes several enhancements including:

• A new Oracle Fusion Middleware technology stack
• The “Alta” user interface to enhance user experience

  

  New OEM Alta Interface 13c

• Edition Based Redefinition (EBR) support

Hybrid Cloud Managment

Oracle Application Management Suite now delivers several capabilities to monitor and manage Oracle E-Business Suite environments on Oracle Cloud and on-premises. These capabilities are delivered through the Enterprise Manager command line interface (EMCLI) for greater flexibility.

• Discovering Orchestration Virtual Machine
• Provisioning a New Oracle E-Business Suite 12.2 Instance
• Backup and Restore an Oracle E-Business Suite Instance
• Lift and Shift E-Business Suite instances from On-premise to Oracle Cloud
• Cloning E-Business Suite instances within Oracle Cloud
• Ability to run FS_Clone only option
• Apply E-Business Suite patches
• Start and Stop E-Business Suite instances on Cloud
• To create custom packages

System Managment for Oracle E-Business Suite

Discovery and Monitoring

• Real User Experience Insight (RUEI) regions integrated within the Oracle E-Business Suite Summary page
  • In this release, we have integrated five different RUEI regions within the Oracle E-Business Suite Summary page. Customers can personalize and activate one or all RUEI regions. These RUEI regions are specific to a given Oracle E-Business Suite instance and customers can view the data and drill down for detailed information.
• Auto Host Aliasing
  • Host aliasing is now automated and will attempt to match the host name in the Oracle E-Business Suite context file and the discovered host name in Oracle Enterprise Manager. If this attempt fails, you will be prompted to perform the mapping manually. Customization Manager and Patch Manager features now support host aliasing.

Change Management for Oracle E-Business Suite

Patch Manager – Oracle E-Business Suite Technology Stack Pack Deployments

• During the prior release of Application Management Suite, you had the ability to view patch recommendations for E-Business Suite applications and the technology stack. In this release, you now have the ability to deploy Database and WebLogic server patches using Enterprise Manager patch plans.

Customization Manager – Customization Discovery & Reporting

• Customization Discovery & Reporting process now includes discovery of database objects.

Cloning – Automated Cloning

• You now have the ability to save the cloning interview process as templates that can be used for future purposes for Smart Clone for R12, 12.2.x procedure.
• You can add custom parameters to the cloning procedure

 

Oracle Configuration Manager Quick Tip – EBS R12.2 Disconnected Collections

Collecting Disconnected Mode OCM data for e-Business Suite R12.2
per MOS Document ID 1447334.1 - Oracle E-Business Suite Release 12.2: Release Notes for Oracle Configuration Manager

MOS Oracle Configuration Manager Collector Download screen.

This method is used whenever your EBS instance is behind a firewall and no proxy server is available to reach the OCM collector end-point (ccr.oracle.com)

The reason you want to use OCM is not only because it provides a complete catalog of your Oracle software versions, modules and plug-ins, so that they can easily be associated with a Service Request (SR) instead of the analyst asking for every RDA script being run for every component in your application and database. 

It also enables the full functionality of that Systems Tab in My Oracle Support (buried just above your Settings menu) that allows you to do light versions of lifecycle management comparisons (such as what the instance looked like 2 weeks ago versus now) without having to own the OEM Lifecycle Management license. It comes compliments of your support agreement.

This also enables the full functionality of the patch and security recommendations section vital towards identifying high priority one-off patches specific to your particular configuration and platform.

Basically, instead of having a cron job periodically automatically upload the configurations to ccr.oracle.com, you will be creating the same staged configuration file, but uploading it to your Customer Support Identifier (CSI) via a Draft Service Request (SR) or a currently Open SR via the File Attachment... feature of the SRs.

In the background of MOS is a managed file transfer handler that looks for the specifically named "ocmconfig.jar" files and when detected, route them over to the Configuration Manager engine for automatic uploading to your CSI account.

While not as convenient as the automatic cron method, for those of use with no direct internet connectivity to ccr.oracle.com, our OEM and MOS functionalities become quite limited if the configurations aren't present under our CSI's.e.g. ORACLE_BASE=/u01/app/oracle

# Oracle DB Home cd $ORACLE_HOME unset ORACLE_CONFIG_HOME export JAVA_HOME=$ORACLE_HOME/jdk cd $ORACLE_HOME/ccr # Clear prior CCR installation, if required. rm -rf $ORACLE_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip cd $ORACLE_HOME/ccr/bin # Run setupCCR in Disconnected mode ./setupCCR -s -d # Create ORACLE_OCM user and related grants . $ORACLE_HOME/ccr/admin/scripts/installCCRSQL.sh collectconfig -s <SID> -r SYS # Additional grants for APPS EBS Collector . $ORACLE_HOME/ccr/admin/scripts/installCCRSQL.sh ebs_collectconfig -u <APPSUSERNAME> # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $ORACLE_HOME/ccr/hosts/$HOSTNAME/state/upload # Oracle 10.1.2 iAS Home . $ORACLE_BASE/EBSapps.env run export ORACLE_HOME=$ORACLE_HOME export ORACLE_CONFIG_HOME=$INST_TOP/ora/10.1.2 export JAVA_HOME=$ORACLE_HOME/jdk # Remove prior OCM config, if needed rm -rf $ORACLE_HOME/ccr/* rm -rf $ORACLE_CONFIG_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip # Bugfix specific to HP-UX Itanium - verify file permissions to execute chmod 754 $ORACLE_HOME/ccr/bin/deployPackages chmod 754 $ORACLE_HOME/ccr/bin/emSnapshotEnv cd $ORACLE_HOME/ccr/bin ./setupCCR -s -d # Disconnected mode # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $INST_TOP/ora/10.1.2/ccr/state/upload #Repeat for patch filesystem; . $ORACLE_BASE/EBSapps.env patch # Oracle FMW WebLogic Home # source the Domain environment . $ORACLE_BASE/EBSapps.env run . $EBS_DOMAIN_HOME/bin/setDomainEnv.sh export ORACLE_HOME=$FMW_HOME/utils export ORACLE_CONFIG_HOME=$INST_TOP/ora/FMW # Remove prior OCM config, if needed rm -rf $ORACLE_HOME/ccr/* rm -rf $ORACLE_CONFIG_HOME/ccr/* cd $ORACLE_HOME unzip <patch_stage_dir>/p5567658_<ver>_<platform>.zip # Bugfix specific to HP-UX Itanium - verify file permissions to execute chmod 754 $ORACLE_HOME/ccr/bin/deployPackages chmod 754 $ORACLE_HOME/ccr/bin/emSnapshotEnv cd $ORACLE_HOME/ccr/bin ./setupCCR -s -d # Disconnected mode # Run the collection and generate the ocmconfig.jar cd $ORACLE_HOME/ccr/bin ./emCCR collect # ocmconfig.jar location $INST_TOP/FMW/ccr/state/upload #Repeat for patch filesystem; . $ORACLE_BASE/EBSapps.env patch # Upload all of your ocmconfig.jar files (can be named uniquely - e.g. mydbocmconfig.jar) per MOS Document ID 763142.1 - How to upload the collection file ocmconfig.jar to My Oracle Support for Oracle Configuration Manager (OCM) running in Disconnected Mode.
# Also can ZIP up all of them at once and upload them to your Draft SR (or current open SR as long as you're happy with the CSI (Customer Support Identifier) that will receive the new configurations.
# Trick is have the string "ocmconfig" and .jar as the extension of each file
Have fun with your new Systems tab in My Oracle Support!

Oracle DataGuard and Standby Database Archive Logs

Users see your wonderful DataGuard implementation like this:

Simple Oracle Dataguard Architecture courtesy of https://appsdbatraining.files.wordpress.com

And yet, you know the actual picture looks more like this:

A map of Oracle DataGuard Architecture components

High-availability – the concept behind it makes every DBA shudder because every time it seems you deal with one element and have it protected, there’s another underlying component that also needs protection and redudancy, or else your solution is still insufficient.

Real Application Clusters (RAC) covers individual database host failures but is sensitive to failure of the storage subsystem or the network interconnections between the hosts.

Recovery Manager (RMAN) is your vital tool to keeping track of what’s backed up and where is it.  And its catalog of recovery information could reside locally in copies of the controlfiles, or centrally in another database.  Depends on your backup strategies, really.  Are you using SAN-based backups (snaps, virtual images, deduplicated block replication) or off-site methods that would have to be shipped back to start recovery?

But the typical first-time setup scenario, is you use the OEM-based jiffy whizbang method to setup your new DataGuard environment at the recommendation of one of the steps int the Maximum Availability Advisor (MAA), and everything’s up and running nicely.  You schedule a new weekly full backup, plus daily incremental backup as Oracle recommended practices prescribe, and notice everything’s running smoothly.

Except on your standby database, the archivelogs are piling up and not being deleted automatically.  What’s next?

This thread was a basic discussion in the Oracle Community forums of the topic:

https://community.oracle.com/thread/2388130?start=0&tstart=0

This is a typical RMAN-based configuration:

On Primary

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

On Standby ( Depends upon where backup is preformed )

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;

Or

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

/* if Standby Where Backups Are Not Performed  */

Source:

Data Guard Concepts and Administration 12c

http://docs.oracle.com/database/121/SBYDB/toc.htm

12 Using RMAN to Back Up and Restore FilesRMAN Configurations at the Primary Database
12.3.4 RMAN Configurations at a Standby Where Backups Are Not Performed

The following RMAN configurations are recommended at a standby database where backups are not done:

1. Connect RMAN to the standby database as target, and to the recovery catalog.
2. Enable automatic deletion of archived logs once they are applied at the standby database (this is also applicable to all terminal databases when the cascading or far sync instance features are in use):
3. CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

 

** However, that doesn’t really take into consideration what might happen if a final archivelog before switchover of roles doesn’t quite make it on the standby (for whatever reason, it gets corrupted during playback or something similar which results in a Database Needs More Recovery error.)

Based upon:

12.3.3 RMAN Configurations at a Standby Database Where Backups are Performed

The following RMAN configurations are recommended at a standby database where backups are done:

1. Connect RMAN to the standby database (where backups are performed) as target, and to the recovery catalog.
2. Enable automatic backup of the control file and the server parameter file:

3.  RMAN > CONFIGURE CONTROLFILE AUTOBACKUP ON;

4. Skip backing up data files for which there already exists a valid backup with the same checkpoint:

5.  RMAN > CONFIGURE BACKUP OPTIMIZATION ON;

6. Configure the tape channels to create backups as required by media management software:

7.  RMAN > CONFIGURE CHANNEL DEVICE TYPE SBT PARMS ‘<channel parameters>’;

8. Because the archived logs are backed up at the standby database, Oracle recommends that you configure the BACKED UP option for the log deletion policy:

9.  RMAN > CONFIGURE ARCHIVELOG DELETION POLICY BACKED UP n TIMES TO [DEVICE TYPE SBT];

 

I came up with the configuration of:

 

On Standby ( Depends upon where backup is preformed )

# If no DataGuard is present (single DB host):

# Ensure daily RMAN backup job is being executed in OEM or via cron.

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;

 

And setup a 2^nd backup set for the Standby Host DB.  This puts the backups into the defined Fast Recovery Area and manages both the backups and archivelog retention in the same mountpoint.

 

Each has its pros and cons depending on the scenario.You need to lay out your entire architecture scheme including backup solutions and play out the various scenarios that you’re required to cover as far as your Quality of Service (QOS) guarantee to your end-user population.

And of course, if you’re using the advanced cross-WAN FarSync DataGuard implementation architecture (wherein there’s a separate Failover Archive Log (FAL) standby database whose sole purpose in life is to cache archivelogs in case the data replication stream is too much for WAN bandwidth to handle in real-time.) this all still applies because the FAL server is basically just another standby target which needs managing just as much as any regular LAN-based full DataGuard standby instance (it’s just missing the big datafiles and handles all the archivelog traffic.)  You’ll just have even more servers and services involved in keeping the whole thing running (like your Global Names Service servers and databases, which might also be RAC and DataGuard protected, or your Single-Sign On authentication services, or even the OEM Cloud Control OMS itself orchestrating all of that.)

 

DataGuard and OEM 12c OMS DB Failover Configuration

# When Oracle DataGuard high-availability for the OMS database is configured using the OEM DataGuard Administration Wizard, and fast-start failover is configured, fail-overs automatically rename the standby as primary, and vice-versa and establish the change-over in roles.  While this accomplishes the database staying online and available on the secondary host (or all other databases in the DG group), the Enterprise Manager OMS must be told how to connect to it – preferably transparently.

# DataGuard OMS Registration
# Enterprise Manager Grid Control 11g: How to Configure the OMS Connect String when Repository is in a Dataguard setup (Doc ID 1328768.1)
# OEMPR11 is our primary DB SID/Service Name
# OEMPR11_DGMGRL is our alias for the fail-over service (pointing to all DG instances)

SQLPLUS as SYS:
SQL> exec DBMS_SERVICE.CREATE_SERVICE (service_name => ‘OEMPR11_DGMGRL’,network_name => ‘OEMPR11_DGMGRL’,aq_ha_notifications => true,failover_method => ‘BASIC’,failover_type => ‘SELECT’,failover_retries => 180,failover_delay => 1);

SQL> exec dbms_service.start_service(‘OEMPR11_DGMGRL’);

# Verify operation:

$> lsnrctl services     # Should see the new OEMPR11_DGMGRL service listed

# Create a Database Trigger so that the service can be stopped when the Database role becomes standby and started only when the Database role is Primary:

SQL> CREATE OR REPLACE TRIGGER manage_OCIservice after startup on database
DECLARE
role VARCHAR(30);
BEGIN
SELECT DATABASE_ROLE INTO role FROM V$DATABASE;
IF role = ‘PRIMARY’ THEN
DBMS_SERVICE.START_SERVICE(‘OEMPR11_DGMGRL’);
ELSE
DBMS_SERVICE.STOP_SERVICE(‘OEMPR11_DGMGRL’);
END IF;
END;

# Re-configure the OMS (All MT hosts) to have the connection string as:
$>  cd <OMS_HOME>/bin
# Following is a single-line command (basically an entire JDBC style connect string)
$> ./emctl config oms -store_repos_details -repos_conndesc ‘(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))’ -repos_user sysman

# Example output
$> ./emctl config oms <…> TYPE=select)(METHOD=basic)))’ -repos_user sysman               <
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Repository User’s Password :
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using ’emctl stop oms -all’ and ’emctl start oms’.
It is also necessary to restart the BI Publisher Managed Server.

# Add the tnsnames.ora entry (all DB hosts at minimum)
OEMPR11_DGMGRL=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))
(FAILOVER_MODE=(TYPE=select)(METHOD=basic))
)

# Testing connectivity:

$> sqlplus sysman/$SYSMAN_PW@'(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))’

# Test the Failover Service:

# Connect to the Database from sqlplus using SYSMAN user via the new service created above:

$> sqlplus sysman/$SYSMAN_PW@OEMPR11_DGMGRL

# Execute these queries to verify the Database name and service names:

SQL> select db_unique_name from v$database;

DB_UNIQUE_NAME
——————-
OEMPR11

SQL> show parameter service_names

NAME            TYPE     VALUE
————-   ——-  ————————————
service_names   string   OEMPR11, OEMPR11_DGMGRL

# Re-start the OMS once so that the connection string change is saved:

cd <OMS_HOME>/bin
./emctl stop oms -all    #on AdminServer MT
./emctl stop oms         #other MTs
./emctl start oms

Investigating an OEM 12c E-Business Suite Alert (AMS 12.1.0.x)

OEM 12 Home Page

The Applications Management Suite plug-in for Oracle Enterprise Manager simplifies discovery of the myriad of subtargets that make up an Oracle E-Business Suite instance.  Correspondingly, the number of alerts sent out can rise dramatically because of the inter-relationships between the components.

For example, the outage of a single Apache process triggers all of the following associated targets also to flag as a service Down status:

INSTANCE-Oracle E-Business Suite
INSTANCE-Infrastructure INSTANCE_host-APPL_TOP Context
HTTP_Server

The number of downed targets increases if a subcomponent of a primary component (such as a single JVM thread under the OACore process) experiences an outage.

This is a simple walk-through of navigating one of the e-mail alerts to start figuring out what happened.

The e-mail alert looks like this:

From: OEM12 Burbank
Sent: Monday, December 07, 2015 8:30 AM
To: DBAs
Subject: EM Event: Fatal:INSTANCE-Oracle E-Business Suite – Target is down; 1 member is down: INSTANCE_EBS Availability System

Host=hostname
Target type=Oracle E-Business Suite
Target name=INSTANCE-Oracle E-Business Suite
Categories=Availability
Message=Target is down; 1 member is down: INSTANCE_EBS Availability System
Severity=Fatal
Event reported time=Dec 7, 2015 8:29:14 AM PST
Target Lifecycle Status=Production
Operating System=Linux
Platform=x86_64
Associated Incident Id=390885
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Target Availability
Event name=Status
Availability status=Down
Root Cause Analysis Status=Symptom
Rule Name=EBS Notifications,Rule_EBS_Notifications
(to get notified, you set up Rule Sets that tell OEM when and what to notify you about)
Rule Owner=DBA
Update Details:
Target is down; 1 member is down: INSTANCE_EBS Availability System
Incident created by rule (Name = Incident management rule set for all targets, Incident creation rule for a Target Down availability status [System generated rule]).

---

To investigate an event alert, click on the Associated Incident ID (e.g. the 390885 which on your system will be a URL taking you into OEM) which will take you to the associated Incident Summary page.

Click on Related Events to investigate what raised the event alert (there may be more than one cause):

OEM 12c AMS 12.1.0.4 – Incident Details

From the screen, it shows the red mark on PRODARMK-Infrastructure PRODARMK_ascopofinm01-APPL_TOP Context (Oracle E-Business Suite Node).

Click on that link in the list of Targets.

Navigate to Monitoring -> Status History:

OEM 12c AMS 12.1.0.4 – Navigation Target: Monitoring -> Status History

Change the Availability History view to All History (the related underlying event caused is displayed.)

OEM 12c AMS 12.1.0.4 – Target: Status History Details

If you click on the related Message (e.g. Target is down; 1 member is down: INSTANCE_hostname.auca.corp_oacore_JVM_…); you will then be shown the related Event page for that target:

OEM 12c AMS 12.1.0.4 – Target: Event Details

Click on the Related Events tab for this target, to confirm the service alert recorded:

OEM 12c AMS 12.1.0.4 – Target: Event Details -> Related Events Timeline

If this is a recurring issue, by sliding the timeline back and forth (and adjusting the period view to a larger sample) you can see if there are any associated time-related occurrences that can be used to identify root cause.

For the specific issue, login to the associated host, and view the output and error logs for the process itself to determine what triggered the alert (in this case, the JVM automatically restarted the OACore process that had run out of memory.)

What’s New Features in Oracle Enterprise Manager (OEM) Cloud Control 12c Release 5 (12.1.0.5)

I kept getting asked what’s new in Release 5 – so here’s the summary of what’s changed since 12.1.0.4.0 (the prior release for the past 12 months):

http://www.oracle.com/technetwork/oem/grid-control/downloads/index.html

http://docs.oracle.com/cd/E24628_01/doc.121/e25353

What’s New in Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5)

•     Oracle Cloud Management (Hybrid Cloud)
•     Database Plug-in 12.1.0.8 Features
•     Fusion Middleware Plug-in 12.1.0.8 Features
•     Cloud Management Plug-in 12.1.0.10 Features
•     Chargeback and Consolidation Planner Plug-in 12.1.0.7 Features
•     Virtual Infrastructure Plug-in 12.1.0.2

Oracle Cloud Management (Hybrid Cloud) – This enhancement is new in Enterprise Manager Cloud Control Release 5 (12.1.0.5).
Enterprise Manager Cloud Control now provides you with a single pane of glass for monitoring and managing both your on-premise and Oracle Cloud deployments, all from the same management console. By deploying Management Agents onto the Oracle Cloud virtual hosts serving your Oracle Cloud services, you are able to manage Oracle Cloud targets just as you would any other targets. The communication between Management Agents and your on-premise Oracle management service instances is secure from external interference. Support is provided for managing Oracle Database and Fusion Middleware PaaS targets, as well as JVMD support for monitoring JVMs on your Oracle Cloud virtual hosts.

Oracle Cloud Management includes the following key features:

•     Automated agent deployment and configuration
•     Database and Java PaaS instances monitoring
•     Incident management including notifications and ticketing integration
•     Configuration management including Search and Inventory, comparison between on-premise and cloud instances, configuration history, and compliance
•     Cloning between on-premise and Oracle Cloud
•     One-off patching of Oracle Cloud database instances

Database Plug-in 12.1.0.8 Features

• Snap Clone Leveraging Sparse Clones on Exadata
  You can now create a Test Master pluggable database (PDB) to use as the snapshot source for thin cloning on Exadata ACFS.
  For functional testing scenarios, for example on development or testing systems, business IT users now have on-demand access to production data copies without incurring the penalty of multi-terrabyte storage.
• Data Cloning to Oracle Cloud
  With this release you have the following data cloning to Oracle Cloud options:
  • Like-to-like cloning: on premise to-from Oracle Private Cloud:
    Enterprise Manager12c PDB to 12c PDB, assuming the containers exist on both sides
    Regular non-container dedicated database to dedicated database
  • Like-to-unlike data migration:
    Regular to PDB migration where the source is an on-premise non-PDB database and the data is migrated to a cloud based PDB
    PDB to normal database migration where the source is a PDB on an Oracle PaaS cloud and data and schema are migrated to a non-container CDB on premise

Fusion Middleware Plug-in 12.1.0.8 Features

• Generic JVM Provisioning for Private Cloud
  This feature enables self-service users to provision non-Oracle middleware components as cloud services. It also uses Enterprise Manager’s chargeback and quota management capabilities to enable administrators to limit service usage based on organizational policies.
• JVMD Support for PaaS
  Enterprise Manager Cloud Control now enables you to deploy JVMD agents on your Oracle Cloud virtual hosts. These deployed JVMD agents can report to a JVMD engine deployed in your private network. This feature enables you to monitor the JVMs deployed on the virtual hosts running your Oracle Cloud services as well as the JVMs deployed on the hosts in your private network using a single console, for example Enterprise Manager Cloud Control deployed in your private network.
• Middleware Self Service Portal Support for Virtual Java as a Service with Exalogic Systems
  WebLogic services that have been provisioned on Exalogic systems using Service Manager can now be viewed in the Cloud Self Service Portal as part of the Exalogic Private Cloud service family.
• MWaaS on Solaris SPARC
  This project integrates MWaaS with the Oracle Enterprise Manager Ops Center. It provides the ability to dynamically add or remove the virtual hardware in a MWaaS cloud.

Cloud Management Plug-in 12.1.0.10 Features

• Dynamic Scaling of PaaS Zone Through Integration with the Sun Solaris SPARC Ops Center 12c Adapter
  Enterprise Manager 12c is integrated with the Ops Center through the Sun Solaris SPARC adapter. An instance of this registered adapter is used by the associated PaaS Infrastructure zone to provision or delete VMs on demand. The communication with the Ops Center server is achieved through the use of a client kit.
  This integration enables customers to provide layered services capability in a private cloud, for example DBaaS using a Solaris VM based IaaS. Any spike in compute demand can now be handled instantly by cloud administrators.
• DBaaS PaaS Provider Pools Integration with Ops Center
  Administrators can now increase the capacity of a DBaaS pool by clicking a single button. Doing this transparently provisions a virtual machine or cluster through the associated Ops Center adapter attached to the PaaS zone. It also deploys the requisite software components, for example Enterprise Manager Agent, Grid Infrastructure, Oracle Database and so on, through the gold image provisioning or by cloning from an existing reference member of the pool.
  Compute resource scarcity at PaaS provider level results in the failure of self service provisioning requests. This integration helps SSA administrators to respond to such failures promptly.
• Clone to Oracle Cloud
  Administrators now have the ability to create a full clone of an Enterprise Manager 12c pluggable database amongst existing container databases within the on-premise IT infrastructure or to an Oracle Cloud. Administrators can also clone a service from on-premise to Oracle PaaS and vice-versa. The cloning is supported in two broad categories:
  •     Peer-peer clone: Clones directly from an on-premise target to Oracle Cloud.
  •     Clone through the Software Library: Administrators can archive a gold image into the Software Library in one step and then deploy it in a separate step.

Chargeback and Consolidation Planner Plug-in 12.1.0.7 Features

• Host Consolidation Support for Oracle Cloud Shapes
  Oracle Enterprise Manager now provides host consolidation support for Oracle Cloud shapes. When creating scenarios for physical server to physical server (P2P) consolidations that target new or phantom destinations, you can choose physical machines configured in the Oracle cloud by selecting the cloud computing configuration, or shape, to use as the destination. Oracle provides a wide range of shapes to help you select a combination of processing power and memory for your instances that best suits your business requirements.

Virtual Infrastructure Plug-in 12.1.0.2

• Dynamic Resource Provisioning Support
  Through integration with Oracle Enterprise Manager Ops Center, dynamic on-demand resource provisioning enables Oracle Cloud and self service administrators to dynamically increase or decrease resources in their private cloud setup based on usage requirements. Resources can either be infrastructure resources such as hosts (based on virtual machines) or platform resources such as middleware and database Oracle homes. Resources can be dynamically added when additional service instances need to be provisioned due to high usage or decreased if existing resources are not being used.

That’s it, in a nutshell.

R12 e-Business Suite and OEM Monitoring – Oracle Spins Freezes

Every so often, system load on an e-Business Suite instance ramps up and response time to users starts climbing, often resulting in user observed errors such as:

• FRM-92100 Your connect to the server has been interrupted
• FRM-92102 A network error has occurred

  

  That dreaded FRM-91201 / FRM-91200 error causing you to restart your session.

Or sometimes, the screen just freezes (aka spins, stops, is broken, stuck, motionless, looks like a screen saver,can’t do anything, won’t work, froze-up, etc.) and the person has to close their browser, or even shut-down their workstation and restart.

It’s simply not doing anything – Nothing to see here, just move your cursor around. And wait… and wait.

Old technology often barks with unrelated error messages to the actual cause.  If there’s a lot going on with concurrent requests, or interfaces, or analytic extracts running, the front-end response-time slows down, sometimes sufficiently to trigger these kinds of Form errors, even though technically there was no interruption to the network connectivity, either between the hosts, nor the workstation and the middle-tier application server.

However, on the database, the user-experience can be seen, although not necessarily in the place you might expect.  OEM  had introduced it’s Adaptive Metric Thresholds technology back in OEM 11g (in a slightly different place than in 12c (in Oracle Management Server/OMS 12.1.0.4.0).  In OEM 11g, they were a link under the AWR Baseline Reports page.

See the Baseline Metric Thresholds link at the bottom.

In OEM 12c, you’ll find them under

Targets -> Database -> Peformance -> Adaptive Thresholds -> Baseline Metric Thresholds -> Edit Thresholds:

Where those adaptive metric thresholds moved in 12c.

 

 

On this page and in the list of Baseline Metrics, when you click into them, you can access the trending statistics being gathered for each metric.  Many times this will provide direct insight into what a user experiences as the “the system is frozen” translates into “the back-end database response time is incredibly bad.”

See the spikes around 7AM and 11:30AM? Those are being associated with “System Froze” reports.

 

In the example here, the database experienced a dramatic slow-down in response almost 5 to 10 times slower than usual, which only lasted a few seconds. But that can be enough to show up in many users’ sessions who might have just kicked off a query, or were trying to save something.  Based upon the information gathered, we set the Warning and Critical thresholds to 1500ms and 2000ms respectively to start sending e-mail alert notifications upon breach of the levels. If the settings are left at “None”, no incident would be raised, and thus, no notification would be sent.

If you’re experiencing odd transient outages or sluggish behavior that defies the normal AWR and ADDM snapshot analysis, go take a look at what OEM has been gathering in the background over time and see if the statistics correlate to any of your issues.  There’s value in that data. Just mine it.

Run #em12c on #db12c? – Discussion from Oracle Open World (MOS Note: 1920632.1)

We also discussed this during the EM4APPS SIG Meeting
https://jhlui1.wordpress.com/2014/08/26/oow14-oracle-enterprise-management-for-applications-sig-mtg-928-mw3003-330p-415p/

Ok Folks, I’ve been here are Oracle Open World for a few days now.  In that time, I’ve had numerous conversations about running Oracle Enterprise Manager 12c on Oracle Database 12c.  I will be honest and say that I’ve enjoyed these conversations; however, after about the fourth time I decided I need to write a quick post on the explanation discussed in these conversations.  

Early this year (August) I wrote a post about the what came out of the OEM CAB in May 2014 and how to get OEM 12c to work on DB12c.  The concept of running OEM 12c on DB12c, pluggable or not, have many people excited and looking forward to configuring OEM to do that very configuration.  Heck, I’ve even installed it for a few customers in that configuration (non-PDB).  So I’m a bit sad in having to say this:  ORACLE DATABASE 12c SUPPORT FOR…

View original post 211 more words

Enterprise Manager 12.1.0.4.0 – Post Upgrade Tasks

Must keep up with OEM 12c Grid Control updates…

Maaz Anjum's Blog

With the new release of Enterprise Manager, I upgraded my 12.1.0.3.0 installation successfully using the superb documentation here. It is however, important to note that the post-steps are also important. For example, upgrading (at the very least) the Management Agent on the OMS host, and de-installation of the old-OMS home. Gokhan Atil wrote a good post on the actual upgrade process which is available on his blog.

Management Agent Upgrade on the OMS Host:

The upgrade process for agents has not changed since EM’s previous 12c releases.

1. Navigate to the “Upgrade Agents” link under the Setup -> Manage Cloud Control menu.

2. Select the agent(s) for the OMS Hosts and click “OK”.

3. With the Agents selected, click on “Submit”. You have the option to override the “Privileged Credentials” to run the root.sh script.

4. The submitted job takes a few minutes before it completes (hopefully successfully!).

View original post 307 more words

How to migrate EM12c R3 OMS and repository to a new host

Very useful, even if you’re moving parts of OEM 12c from one host to another, or modifying the filesystem mounts.

Pardy DBA

(EDIT 20130917: If you simply need to change the IP address of your OEM server, please review MOS note 1562029.1.  The procedure in that note may allow you to change your OEM server’s IP address without following the lengthy process I describe below.)

In order to save power in our data center, I need to migrate my EM12c R3 environment from the host where it currently runs to a new host.  I have a simple configuration, with a single OMS, no load balancer, and the repository database runs on the same host as EM12c R3 itself.  I also have BI Publisher installed and integrated with EM12c, and a few third party plugins as I’ve detailed elsewhere on this blog.  If you use an OS other than Linux x86-64 I suggest you research thoroughly as this procedure may or may not apply to your environment.  Further, if you have a multi-OMS…

View original post 3,770 more words