Okay, an Autonomous Database is one that Wanders Off by Itself.

Used without express permission from windowsinstructed.com

In a 2014 VoucherCloud.net (a coupon website) survey of the non-technical U.S. general public:

• 11% believed HTML wss a sexually-transmitted disease
• 51% believed a stormy weather condition would affect their access to the Cloud
• 27% thought a gigabyte was a common insect in South America
• 18% thought Blu-ray was a marine animal
• 23% thought an MP3 was a Star Wars robot
• 12% thought USB is the acronym for a European country
• 42% said they believed a motherboard was “the deck of a cruise ship”
• 77% could not identify what SEO means
• 15% say software is comffortable clothing

However, 61% of the 2,392 respondents (18 and older) all thought it was important to have a good knowledge of technology.

That explains why in ZDNet’s Oracle’s Next Chapter: The Autonomous Database and the DBA (https://www.zdnet.com/article/oracles-next-chapter-the-autonomous-database-and-the-dba/) takes a bit of chewing to understand that when a vendor says “you’ll save $250K by moving to the Cloud,” that’s akin to someone saying, “You’ll save $40,000 by not buying a car, but renting it at $40/day from Hertz/Avis/Thrifty/Dollar.”

Well, yes, but not quite.  Rental agencies have those add-ons such as a Collision Damage Waiver, which can be thought of as the same as that 15% mandatory support fee needed on top of the core/sizing/machine-size fee, providing coverage in case something goes awry (or buggy, or BSOD’s for no known reason.)

And simply, if you stop paying, you don’t have a car/database/middleware/website.  If you do decide to opt for the rent-to-own option, just like the ubiquitous furniture rentals used by many seasoned relocation workers, doing so does cost much more than buying the furniture outright (but you don’t have to move it, and you get to turn it in, and trade-up or down when you wish, subject to the terms of your rental agreement.)

[Terms are important, as you will notice in the new Cloud On-Premise agreement, it does have a 4-year minimum term – similar to a limited term car lease.  And it comes with an early termination cost. And similarly it has “limited mileage” conditions, which if you go over your CPU/sizing/feature limits, you’ll simply be billed extra for that.  Convenience has costs.]

An autonomous database at this stage is similar to self-driving cars – given super-precise limitations, on a controlled environment, with well-defined conditions, yes, the Optimizer stays within the lanes and keeps the database engine humming along. Whence the odd situation is encountered, back to the driver/DBA to figure out what to-do and what went wrong.

The LA TImes article:
http://www.latimes.com/business/technology/la-fi-tn-1-10-americans-html-std-study-finds-20140304-story.html

The full VoucherCloud.net survey results:
https://drive.google.com/file/d/0B9HJeR-F9NIeczNDb2hVb2p6UTQ/edit

In closing, in case you missed it, Japan created a banana with edible peels: https://news.nationalgeographic.com/2018/01/edible-peel-bananas-created-japan-food-spd/

Advertisement

Oracle OpenWorld 2016 in 5 Pages

 

Oracle OpenWorld 2016 Cloud Services and Products Summary

4 Days (almost 7 for me) of Product Directors and Managers spinning as many plates in the air as humanly possible, all being tossed into the Cloud with a fervor only matched by the lack of compensation being provided for conventional technology stack sales, left me with some specific value-adds for my time spent (not including the Safe Harbor information, we can’t say anything about….sshhhh.)

Note: (I’m EBS-centric and we have very little budget for hardware, so while I was amazed at the new SPARC S7 processors with on-board encryption and J/SQL optimization, alas they didn’t make it into my summary pages this year.)

AD/TXK Delta.8
Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.8 and R12.TXK.C.Delta.8 (Doc ID 2159750.1)

Configuring Oracle E-Business Suite Release 12.x Using Logical Host Names (Doc ID 1968231.1)

• e-Mail Notifications at Parent and Child Levels for Request Sets
• Save and Re-use Concurrent Program Parameters (CP Templates)
• Regex Support in Profile Option Values and FND_MESSAGES
• Multiple OA Framework Windows and Tabs in Multiple Browser Windows
• EBS Healthcheck Diagnostics
• 64-bit JDK’s on CM Tier
• adop Improvements
• Delta Sync Mode (syncmode=patch/delta) – changed rows only
• Rsync support in phase=prepare
• Custom module sync auto-supported (no custom drivers req’d)
• Standardized logging SessionID/ExecutionID/phase/node

EBS 12.2.6.AD Oracle E-Business Suite 12.2.6 Readme (Doc ID 2114016.1)

• Security Configuration Console
• JSP Whitelisting
• Redirection Whitelisting
• AD/TKX.Delta.8 incorporated
• More HTML User Interfaces: HTML alternatives to several existing Forms; interfaces offer simpler layouts and more modern user interactions.

–Examples include the Update Assets UI in Enterprise Asset Management, and AOL setup UIs for Users, Responsibilities, and Flexfields.

SecureFiles does not allow shrinking of the tablespace after deletions (AP Imaging)

Available Interconnect Cloud Service adapters:

• –PL/SQL
• –Java Beans
• –Apps Modules

Roadmap ICS adapters:

• –REST
• –Concurrent Programs
• –XML Gateway
• –BPEL/Business Services

CAAS (Cloud Adaptive Assistant Service) – The widget Larry was text-chatting with.

Mobile Foundation and Mobile Application Framework

Foundation v6 released August 2016

–Incorporates MAF 2.3.2

–Enhancements for Approvals (HR, Projects, Service Centers, Quoting, Custom Approvals)

Foundation v7 scheduled for April 2017

MAF Roadmap items:

–Airwatch and MobileIron support

–Mobile REST API’s

–EBS Integrated Service Gateway support (SOA)

PDF Version for download:       oracle_oow16_summary_jlui

TRCSESS – Trace Information for Various Oracle Sessions or Modules – Quick & Dirty Tip

Database Sessions view from Oracle SQL Developer

— Find your session

SQL> /* a Dell TOAD type query */

SELECT ROUND (BITAND (s.ownerid, 65535)) parent_session_sid,
ROUND (BITAND (s.ownerid, 16711680) / 65536) parent_session_instid,
RAWTOHEX (SADDR) AS saddr,
s.SID,
s.SERIAL#,
s.AUDSID,
RAWTOHEX (PADDR) AS paddr,
s.USER#,
s.USERNAME,
s.COMMAND,
s.OWNERID,
s.TADDR,
s.LOCKWAIT,
s.STATUS,
s.SERVER,
s.SCHEMA#,
s.SCHEMANAME,
s.OSUSER,
s.PROCESS,
s.MACHINE,
s.PORT,
s.TERMINAL,
UPPER (s.PROGRAM) PROGRAM,
s.TYPE,
s.SQL_ADDRESS,
s.SQL_HASH_VALUE,
s.SQL_ID,
s.SQL_CHILD_NUMBER,
s.SQL_EXEC_START,
s.SQL_EXEC_ID,
s.PREV_SQL_ADDR,
s.PREV_HASH_VALUE,
s.PREV_SQL_ID,
s.PREV_CHILD_NUMBER,
s.PREV_EXEC_START,
s.PREV_EXEC_ID,
s.PLSQL_ENTRY_OBJECT_ID,
s.PLSQL_ENTRY_SUBPROGRAM_ID,
s.PLSQL_OBJECT_ID,
s.PLSQL_SUBPROGRAM_ID,
s.MODULE,
s.MODULE_HASH,
s.ACTION,
s.ACTION_HASH,
s.CLIENT_INFO,
s.FIXED_TABLE_SEQUENCE,
s.ROW_WAIT_OBJ#,
s.ROW_WAIT_FILE#,
s.ROW_WAIT_BLOCK#,
s.ROW_WAIT_ROW#,
s.TOP_LEVEL_CALL#,
s.LOGON_TIME,
s.LAST_CALL_ET,
s.PDML_ENABLED,
s.FAILOVER_TYPE,
s.FAILOVER_METHOD,
s.FAILED_OVER,
s.RESOURCE_CONSUMER_GROUP,
s.PDML_STATUS,
s.PDDL_STATUS,
s.PQ_STATUS,
s.CURRENT_QUEUE_DURATION,
s.CLIENT_IDENTIFIER,
s.BLOCKING_SESSION_STATUS,
s.BLOCKING_INSTANCE,
s.BLOCKING_SESSION,
s.FINAL_BLOCKING_SESSION_STATUS,
s.FINAL_BLOCKING_INSTANCE,
s.FINAL_BLOCKING_SESSION,
s.SEQ#,
s.EVENT#,
s.EVENT,
s.P1TEXT,
s.P1,
s.P1RAW,
s.P2TEXT,
s.P2,
s.P2RAW,
s.P3TEXT,
s.P3,
s.P3RAW,
s.WAIT_CLASS_ID,
s.WAIT_CLASS#,
s.WAIT_CLASS,
s.WAIT_TIME,
s.SECONDS_IN_WAIT,
s.STATE,
s.WAIT_TIME_MICRO,
s.TIME_REMAINING_MICRO,
s.TIME_SINCE_LAST_WAIT_MICRO,
s.SERVICE_NAME,
s.SQL_TRACE,
s.SQL_TRACE_WAITS,
s.SQL_TRACE_BINDS,
s.SQL_TRACE_PLAN_STATS,
s.SESSION_EDITION_ID,
s.CREATOR_ADDR,
s.CREATOR_SERIAL#,
s.ECID,
stat.cpu – stat.CPU_this_call_start cpu_this_call,
stat.CPU,
stat.UGA_memory,
stat.PGA_memory,
stat.Commits,
stat.Rollbacks,
si.Block_Gets,
si.Consistent_Gets,
si.Physical_Reads,
si.Block_Changes,
si.Consistent_Changes,
p.program “OSProgram”,
p.spid,
p.pid
FROM V$SESSION S,
V$SESS_IO si,
V$PROCESS p,
(  SELECT ss.sid stat_sid,
SUM (
DECODE (sn.name, ‘CPU used when call started’, ss.VALUE, 0))
CPU_this_call_start,
SUM (
DECODE (sn.name, ‘CPU used by this session’, ss.VALUE, 0))
CPU,
SUM (DECODE (sn.name, ‘session uga memory’, ss.VALUE, 0))
uga_memory,
SUM (DECODE (sn.name, ‘session pga memory’, ss.VALUE, 0))
pga_memory,
SUM (DECODE (sn.name, ‘user commits’, ss.VALUE, 0)) commits,
SUM (DECODE (sn.name, ‘user rollbacks’, ss.VALUE, 0))
rollbacks
FROM v$sesstat ss, v$statname sn
WHERE     ss.STATISTIC# = sn.STATISTIC#
AND (   sn.name = ‘CPU used when call started’
OR sn.name = ‘CPU used by this session’
OR sn.name = ‘session uga memory’
OR sn.name = ‘session pga memory’
OR sn.name = ‘user commits’
OR sn.name = ‘user rollbacks’)
GROUP BY ss.sid) stat
WHERE     (    (s.USERNAME IS NOT NULL)
AND (NVL (s.osuser, ‘x’) <> ‘SYSTEM’)
AND (s.TYPE <> ‘BACKGROUND’))
AND si.sid(+) = s.sid
AND p.addr(+) = s.paddr
AND stat.stat_sid = s.sid

— Setup a Client Trace:

SQL> exec dbms_monitor.client_id_trace_enable (client_id=>'<terminal or client string>’);
PL/SQL procedure successfully completed.

— Setup a trace for all matching session threads for a module:
SQL> exec dbms_monitor.serv_mod_act_trace_enable (service_name=>'<ORACLE_SID or SERVICE_NAME>’, module_name=>'<MODULE from V$SESSION>’);
PL/SQL procedure successfully completed.

Visit your trace directory (in 11gR2 and up, you can consolidate the various dump locations using the init.ora parameter diagnostic_dest='<filepath>’)

# *.trc looks through every trace file in the directory for the matching module string
# Can also use any of the other options to search for specific sessions:
# [action=action_name]
# [clientid=client_id]
# [session=session_id]

oracle > trcsess output=mytrace.trc module=”<MODULE from V$SESSION>” *.trc

# Then just execute your normal tkprof command to format the consolidated output

oracle > tkprof mytrace.trc myformattedtrace.txt sys=no sort=exeela explain=<user/pwd>

# Voila!  All the traces for a specific session, module, or action at a specific time.

— Don’t forget to disable your tracing afterwards!

SQL> exec dbms_monitor.client_id_trace_disable (client_id=>'<terminal or client string>’);
PL/SQL procedure successfully completed.

— Setup a trace for all matching session threads for a module:
SQL> exec dbms_monitor.serv_mod_act_trace_disable (service_name=>'<ORACLE_SID or SERVICE_NAME>’, module_name=>'<MODULE from V$SESSION>’);
PL/SQL procedure successfully completed.

Oracle DataGuard and Standby Database Archive Logs

Users see your wonderful DataGuard implementation like this:

Simple Oracle Dataguard Architecture courtesy of https://appsdbatraining.files.wordpress.com

And yet, you know the actual picture looks more like this:

A map of Oracle DataGuard Architecture components

High-availability – the concept behind it makes every DBA shudder because every time it seems you deal with one element and have it protected, there’s another underlying component that also needs protection and redudancy, or else your solution is still insufficient.

Real Application Clusters (RAC) covers individual database host failures but is sensitive to failure of the storage subsystem or the network interconnections between the hosts.

Recovery Manager (RMAN) is your vital tool to keeping track of what’s backed up and where is it.  And its catalog of recovery information could reside locally in copies of the controlfiles, or centrally in another database.  Depends on your backup strategies, really.  Are you using SAN-based backups (snaps, virtual images, deduplicated block replication) or off-site methods that would have to be shipped back to start recovery?

But the typical first-time setup scenario, is you use the OEM-based jiffy whizbang method to setup your new DataGuard environment at the recommendation of one of the steps int the Maximum Availability Advisor (MAA), and everything’s up and running nicely.  You schedule a new weekly full backup, plus daily incremental backup as Oracle recommended practices prescribe, and notice everything’s running smoothly.

Except on your standby database, the archivelogs are piling up and not being deleted automatically.  What’s next?

This thread was a basic discussion in the Oracle Community forums of the topic:

https://community.oracle.com/thread/2388130?start=0&tstart=0

This is a typical RMAN-based configuration:

On Primary

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

On Standby ( Depends upon where backup is preformed )

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;

Or

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

/* if Standby Where Backups Are Not Performed  */

Source:

Data Guard Concepts and Administration 12c

http://docs.oracle.com/database/121/SBYDB/toc.htm

12 Using RMAN to Back Up and Restore FilesRMAN Configurations at the Primary Database
12.3.4 RMAN Configurations at a Standby Where Backups Are Not Performed

The following RMAN configurations are recommended at a standby database where backups are not done:

1. Connect RMAN to the standby database as target, and to the recovery catalog.
2. Enable automatic deletion of archived logs once they are applied at the standby database (this is also applicable to all terminal databases when the cascading or far sync instance features are in use):
3. CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

 

** However, that doesn’t really take into consideration what might happen if a final archivelog before switchover of roles doesn’t quite make it on the standby (for whatever reason, it gets corrupted during playback or something similar which results in a Database Needs More Recovery error.)

Based upon:

12.3.3 RMAN Configurations at a Standby Database Where Backups are Performed

The following RMAN configurations are recommended at a standby database where backups are done:

1. Connect RMAN to the standby database (where backups are performed) as target, and to the recovery catalog.
2. Enable automatic backup of the control file and the server parameter file:

3.  RMAN > CONFIGURE CONTROLFILE AUTOBACKUP ON;

4. Skip backing up data files for which there already exists a valid backup with the same checkpoint:

5.  RMAN > CONFIGURE BACKUP OPTIMIZATION ON;

6. Configure the tape channels to create backups as required by media management software:

7.  RMAN > CONFIGURE CHANNEL DEVICE TYPE SBT PARMS ‘<channel parameters>’;

8. Because the archived logs are backed up at the standby database, Oracle recommends that you configure the BACKED UP option for the log deletion policy:

9.  RMAN > CONFIGURE ARCHIVELOG DELETION POLICY BACKED UP n TIMES TO [DEVICE TYPE SBT];

 

I came up with the configuration of:

 

On Standby ( Depends upon where backup is preformed )

# If no DataGuard is present (single DB host):

# Ensure daily RMAN backup job is being executed in OEM or via cron.

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;

 

And setup a 2^nd backup set for the Standby Host DB.  This puts the backups into the defined Fast Recovery Area and manages both the backups and archivelog retention in the same mountpoint.

 

Each has its pros and cons depending on the scenario.You need to lay out your entire architecture scheme including backup solutions and play out the various scenarios that you’re required to cover as far as your Quality of Service (QOS) guarantee to your end-user population.

And of course, if you’re using the advanced cross-WAN FarSync DataGuard implementation architecture (wherein there’s a separate Failover Archive Log (FAL) standby database whose sole purpose in life is to cache archivelogs in case the data replication stream is too much for WAN bandwidth to handle in real-time.) this all still applies because the FAL server is basically just another standby target which needs managing just as much as any regular LAN-based full DataGuard standby instance (it’s just missing the big datafiles and handles all the archivelog traffic.)  You’ll just have even more servers and services involved in keeping the whole thing running (like your Global Names Service servers and databases, which might also be RAC and DataGuard protected, or your Single-Sign On authentication services, or even the OEM Cloud Control OMS itself orchestrating all of that.)

 

DataGuard and OEM 12c OMS DB Failover Configuration

# When Oracle DataGuard high-availability for the OMS database is configured using the OEM DataGuard Administration Wizard, and fast-start failover is configured, fail-overs automatically rename the standby as primary, and vice-versa and establish the change-over in roles.  While this accomplishes the database staying online and available on the secondary host (or all other databases in the DG group), the Enterprise Manager OMS must be told how to connect to it – preferably transparently.

# DataGuard OMS Registration
# Enterprise Manager Grid Control 11g: How to Configure the OMS Connect String when Repository is in a Dataguard setup (Doc ID 1328768.1)
# OEMPR11 is our primary DB SID/Service Name
# OEMPR11_DGMGRL is our alias for the fail-over service (pointing to all DG instances)

SQLPLUS as SYS:
SQL> exec DBMS_SERVICE.CREATE_SERVICE (service_name => ‘OEMPR11_DGMGRL’,network_name => ‘OEMPR11_DGMGRL’,aq_ha_notifications => true,failover_method => ‘BASIC’,failover_type => ‘SELECT’,failover_retries => 180,failover_delay => 1);

SQL> exec dbms_service.start_service(‘OEMPR11_DGMGRL’);

# Verify operation:

$> lsnrctl services     # Should see the new OEMPR11_DGMGRL service listed

# Create a Database Trigger so that the service can be stopped when the Database role becomes standby and started only when the Database role is Primary:

SQL> CREATE OR REPLACE TRIGGER manage_OCIservice after startup on database
DECLARE
role VARCHAR(30);
BEGIN
SELECT DATABASE_ROLE INTO role FROM V$DATABASE;
IF role = ‘PRIMARY’ THEN
DBMS_SERVICE.START_SERVICE(‘OEMPR11_DGMGRL’);
ELSE
DBMS_SERVICE.STOP_SERVICE(‘OEMPR11_DGMGRL’);
END IF;
END;

# Re-configure the OMS (All MT hosts) to have the connection string as:
$>  cd <OMS_HOME>/bin
# Following is a single-line command (basically an entire JDBC style connect string)
$> ./emctl config oms -store_repos_details -repos_conndesc ‘(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))’ -repos_user sysman

# Example output
$> ./emctl config oms <…> TYPE=select)(METHOD=basic)))’ -repos_user sysman               <
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Repository User’s Password :
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using ’emctl stop oms -all’ and ’emctl start oms’.
It is also necessary to restart the BI Publisher Managed Server.

# Add the tnsnames.ora entry (all DB hosts at minimum)
OEMPR11_DGMGRL=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))
(FAILOVER_MODE=(TYPE=select)(METHOD=basic))
)

# Testing connectivity:

$> sqlplus sysman/$SYSMAN_PW@'(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=<primaryDBHostnameFQDN>)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=<secondaryDBHostnameFQDN>)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=OEMPR11_DGMGRL))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))’

# Test the Failover Service:

# Connect to the Database from sqlplus using SYSMAN user via the new service created above:

$> sqlplus sysman/$SYSMAN_PW@OEMPR11_DGMGRL

# Execute these queries to verify the Database name and service names:

SQL> select db_unique_name from v$database;

DB_UNIQUE_NAME
——————-
OEMPR11

SQL> show parameter service_names

NAME            TYPE     VALUE
————-   ——-  ————————————
service_names   string   OEMPR11, OEMPR11_DGMGRL

# Re-start the OMS once so that the connection string change is saved:

cd <OMS_HOME>/bin
./emctl stop oms -all    #on AdminServer MT
./emctl stop oms         #other MTs
./emctl start oms

Slow Oracle Database Performance on a NetApp (10g, 11g, 12c) Tips

http://www.netapp.com

If you have a NetApp storage appliance, device, SAN, whatever you want to call it, review this first (NetApp Best Practices for Oracle Databases – published March 2014): http://www.netapp.com/us/media/tr-3633.pdf

The quick and dirty (for those experiencing Production issues right now):

The database-side init.ora:

grep filesystemio $ORACLE_HOME/dbs/initSID.ora

*.filesystemio_options=’SETALL’

The options for filesystemio_options can be summarized as follows:

• ASYNCH: Asynchronous I/O. Oracle should submit I/O requests to the operating system for processing. This permits Oracle to carry on with other work rather than waiting for I/O completion and increases parallelization of I/O.

• DIRECTIO: Direct I/O. Oracle should perform I/O directly against physical files rather than routing I/O through the host operating system cache.

• NONE: Use synchronous and buffered I/O. In these configurations, the choice between shared and dedicated server processes and the number of dbwriters will become more important.

• SETALL: Use both asynchronous and direct I/O. (preferred for NetApp)

Note: The filesystemio_options parameter has no effect in DNFS and ASM environments. The use of Direct NFS (DNFS) or Automatic Storage Management (ASM) automatically results in the use of both asynchronous and direct I/O.

However…. Because you might end up cloning to an environment that doesn’t support DNFS (remember it’s stuck in the ORACLE_HOME/rdbms/lib object binary), you should have it set to SETALL anyway to allow fall-back to standard NFS (but with asynch and DI/O.).

[This refers to the: cd $ORACLE_HOME/rdbms/lib ; make -f ins_rdbms.mk [dnfs_on / dnfs_off] used to enable and disable DNFS (you also need the /etc/oranfstab created to support this – server:   local:   path: export: mount: etc…)]

If you have 11.2.0.4.x or later (12c) there’s a new DNFS-related init.ora parameter critical for managing packet requests by the database to avoid packet flooding the NFS file server:  Without it (i.e. older versions) DNFS only works well on dedicated DNFS storage because an oracle database on modern hardware can easily over-whelm the NetApp’s ability to service TCP packets (can send upwards of 4,000+ requests per second)

$> grep dnfs $ORACLE_HOME/dbs/initSID.ora

*.dnfs_batch_size=128

On the Linux OS side:

$> grep tcp /etc/sysctl.conf

sunrpc.tcp_slot_table_entries = 128

$> grep nosharecache /etc/fstab

hostname.corp:/vol/testoracled/d01     /u01/app/Oracle              nfs rw,bg,hard,vers=3,proto=tcp,timeo=600,rsize=65536,wsize=65536,nointr,nosharecache

hostname.corp:/vol/testdata            /u01/app/Oracle/oradata     nfs rw,bg,hard,vers=3,proto=tcp,timeo=600,rsize=65536,wsize=65536,nointr,nosharecache

Incidentally, if you want to measure your I/O rates, I have been using SLOB (Silly Little Oracle Basher – http://kevinclosson.wordpress.com/slob/ ) with one small modification to the iostat command in the runit.sh script to capture the I/O activity on all the shares (iostat -mn instead of -xm):

#      ( iostat -xm 3 > iostat.out 2>&1 ) &
( iostat -mn 3 > iostat.out 2>&1 ) &
misc_pids=”${misc_pids} $!”
( vmstat 3 > vmstat.out 2>&1 ) &
misc_pids=”${misc_pids} $!”
( mpstat -P ALL 3  > mpstat.out 2>&1) &
misc_pids=”${misc_pids} $!”

All this besides the usual OEM recommendations (bigger log_buffer, manage SGA size, get block sizes correct), dropped latency from over 40000ms to < 80ms on heavy load, and is producing 12000 IOPS on the same device.

This is an R12 e-Business Suite environment running on an 11.2.0.3.13 (PSU JAN2015) database (5TB).

What triggered this investigation was that the Log Writer process (LGWR) began dumping trace files about 6 months ago with entries that look like:

*** 2014-12-21 19:49:29.827
Warning: log write elapsed time 9988ms, size 1KB

After investigating all the usual suspects about slow disk, I happened upon the aforementioned NetApp white paper. This occurs when multiple hosts are competing for resources under a single NetApp appliance environment (multiple enclosures, multiple dedicated or shared aggregates, etc.)

Situation addressed and under control.

R12 e-Business Suite and OEM Monitoring – Oracle Spins Freezes

Every so often, system load on an e-Business Suite instance ramps up and response time to users starts climbing, often resulting in user observed errors such as:

• FRM-92100 Your connect to the server has been interrupted
• FRM-92102 A network error has occurred

  

  That dreaded FRM-91201 / FRM-91200 error causing you to restart your session.

Or sometimes, the screen just freezes (aka spins, stops, is broken, stuck, motionless, looks like a screen saver,can’t do anything, won’t work, froze-up, etc.) and the person has to close their browser, or even shut-down their workstation and restart.

It’s simply not doing anything – Nothing to see here, just move your cursor around. And wait… and wait.

Old technology often barks with unrelated error messages to the actual cause.  If there’s a lot going on with concurrent requests, or interfaces, or analytic extracts running, the front-end response-time slows down, sometimes sufficiently to trigger these kinds of Form errors, even though technically there was no interruption to the network connectivity, either between the hosts, nor the workstation and the middle-tier application server.

However, on the database, the user-experience can be seen, although not necessarily in the place you might expect.  OEM  had introduced it’s Adaptive Metric Thresholds technology back in OEM 11g (in a slightly different place than in 12c (in Oracle Management Server/OMS 12.1.0.4.0).  In OEM 11g, they were a link under the AWR Baseline Reports page.

See the Baseline Metric Thresholds link at the bottom.

In OEM 12c, you’ll find them under

Targets -> Database -> Peformance -> Adaptive Thresholds -> Baseline Metric Thresholds -> Edit Thresholds:

Where those adaptive metric thresholds moved in 12c.

 

 

On this page and in the list of Baseline Metrics, when you click into them, you can access the trending statistics being gathered for each metric.  Many times this will provide direct insight into what a user experiences as the “the system is frozen” translates into “the back-end database response time is incredibly bad.”

See the spikes around 7AM and 11:30AM? Those are being associated with “System Froze” reports.

 

In the example here, the database experienced a dramatic slow-down in response almost 5 to 10 times slower than usual, which only lasted a few seconds. But that can be enough to show up in many users’ sessions who might have just kicked off a query, or were trying to save something.  Based upon the information gathered, we set the Warning and Critical thresholds to 1500ms and 2000ms respectively to start sending e-mail alert notifications upon breach of the levels. If the settings are left at “None”, no incident would be raised, and thus, no notification would be sent.

If you’re experiencing odd transient outages or sluggish behavior that defies the normal AWR and ADDM snapshot analysis, go take a look at what OEM has been gathering in the background over time and see if the statistics correlate to any of your issues.  There’s value in that data. Just mine it.

Using Demo Data for Oracle Data Mining Tools

With the forthcoming (but already available) SQLDeveloper 4.1 edition, an improved version of the Oracle Data Miner tools is incorporated into the SQLDeveloper console.  However, I found that there were a number of steps needed to actually use this new data modeling product other than just responding ‘Yes’ to the “Do you wish to enable the Data Miner Repository on this database?” prompt.

Here’s what I ended up doing to get things up and running (so that I could play with data modeling and visualization using Excel and the new SQLDeveloper DM extensions.)

#In this case, I’m adding back the demonstration data (i.e. EMP, DEPTNO type tables; the SH, OE, HR, et.al. schemas) into an existing R12 e-Business Suite (12.1.3) instance.

# Installing the Oracle Demo data in an R12 instance.

# Use the runInstaller from the R12 $ORACLE_HOME
cd $ORACLE_HOME/oui/bin
export DISPLAY=<workstation IP>:0.0
./runInstaller

# Choose the source products.xml from the staging area – Download and stage the DB Examples CD from OTN
/mnt/nfs/database/11203/examples/stage/products.xml

# Complete the OUI installation through [Finish]

cd $ORACLE_HOME/demo/schema
mkdir -p $ORACLE_HOME/demo/schema/log
echo $ORACLE_HOME/demo/schema/log/     ## used to respond to the Log Directory prompt during mksample.sql

sqlplus “/ as sysdba”
— will need passwords for: SYS/SYSTEM and APPS (used for all of the demo schemas, some of which pre-exist such as, HR, OE (PM, IX, SH and BI were okay for 12.1.3).

— ## Be sure to comment out any DROP USER <HR, OE, etc.) commands in this script (or you will be restoring your EBS instance from a backup because it just dropped your Module schema tables…) ##
— They look like this:
/*
mksample.sql:– DROP USER hr CASCADE;
mksample.sql:– DROP USER oe CASCADE;
mksample.sql:DROP USER pm CASCADE;
mksample.sql:DROP USER ix CASCADE;
mksample.sql:DROP USER sh CASCADE;
mksample.sql:DROP USER bi CASCADE;
*/

–Similarly – if/when you decide you no longer need the data – do NOT just use the $ORACLE_HOME/demo/schema/drop_sch.sql script
–or you just dropped your HR/OE/BI EBS schemas; don’t do that.
/*
drop_sch.sql:PROMPT Dropping Sample Schemas
drop_sch.sql:– DROP USER hr CASCADE;
drop_sch.sql:– DROP USER oe CASCADE;
drop_sch.sql:DROP USER pm CASCADE;
drop_sch.sql:DROP USER ix CASCADE;
drop_sch.sql:DROP USER sh CASCADE;
drop_sch.sql:DROP USER bi CASCADE;

order_entry/oe_main.sql:– Dropping the user with all its objects
order_entry/oe_main.sql:– DROP USER oe CASCADE;
order_entry/oe_main.sql:– ALTER USER oe DEFAULT TABLESPACE &tbs QUOTA UNLIMITED ON &tbs;

*/
— in this instance the $APPS_PW is synchronized to all application module schemas (i.e. AR, HR, GL, etc.)
— log directory would be the actual path from echo $ORACLE_HOME/demo/schema/log/ (including the trailing slash)
SQL> @mksample.sql

# to additionally create the Data Mining user (DM in this case)

create user &&dmuser identified by &&dmuserpwd
default tablespace &&usertblspc
temporary tablespace &&temptblspc
quota unlimited on &&usertblspc;

GRANT CREATE JOB TO &&dmuser;
GRANT CREATE MINING MODEL TO &&dmuser;       — required for creating models
GRANT CREATE PROCEDURE TO &&dmuser;
GRANT CREATE SEQUENCE TO &&dmuser;
GRANT CREATE SESSION TO &&dmuser;
GRANT CREATE SYNONYM TO &&dmuser;
GRANT CREATE TABLE TO &&dmuser;
GRANT CREATE TYPE TO &&dmuser;
GRANT CREATE VIEW TO &&dmuser;
GRANT EXECUTE ON ctxsys.ctx_ddl TO &&dmuser;
GRANT CREATE ANY DIRECTORY TO &&dmuser;
— Grant the SH Demo table and package objects to the DM user
@?/rdbms/demo/dmshgrants.sql &&dmuser

connect &&dmuser/&&dmuserpwd
— Create the Data Mining Views against the  SH Demo table and package objects
@?/rdbms/demo/dmsh.sql

@?/rdbms/demo/dmabdemo.sql — Builds the Adaptive Baynes Model demo
@?/rdbms/demo/dmaidemo.sql — Builds the Attribute Importance demo
@?/rdbms/demo/dmardemo.sql — Builds the Association Rules demo
@?/rdbms/demo/dmdtdemo.sql — Builds the Decision Tree demo
@?/rdbms/demo/dmdtxvlddemo.sql — Builds the Cross Validation demo
@?/rdbms/demo/dmglcdem.sql — Builds the Generalized Linear model demo
@?/rdbms/demo/dmglrdem.sql — Builds the General Linear Regression model demo
@?/rdbms/demo/dmhpdemo.sql — not a Data Mining program – Hierarchical Profiler
@?/rdbms/demo/dmkmdemo.sql — Builds the K-Means Clustering model demo
@?/rdbms/demo/dmnbdemo.sql — Builds the Naive Baynes Model data
@?/rdbms/demo/dmnmdemo.sql — Builds the Non-negative Matrix Factorization model
@?/rdbms/demo/dmocdemo.sql — Builds the O-Cluster model Demo
@?/rdbms/demo/dmsvcdem.sql — Builds the Support Vector Machine model demo
@?/rdbms/demo/dmsvodem.sql — Builds the One-Class Support Vector Machine model demo
@?/rdbms/demo/dmsvrdem.sql — Builds the Support Vector Regression model demo
@?/rdbms/demo/dmtxtfe.sql — Builds the Oracle Text Term Feature Extractor demo
@?/rdbms/demo/dmtxtnmf.sql — Builds the Text Mining Non-Negative Matrix Factorization model demo
@?/rdbms/demo/dmtxtsvm.sql — Builds the Text Mining Support Vector Machine model demo

## End of Data Mining Demo user (DM) setup and configuration for use of Oracle Demo Data

Run #em12c on #db12c? – Discussion from Oracle Open World (MOS Note: 1920632.1)

We also discussed this during the EM4APPS SIG Meeting
https://jhlui1.wordpress.com/2014/08/26/oow14-oracle-enterprise-management-for-applications-sig-mtg-928-mw3003-330p-415p/

Ok Folks, I’ve been here are Oracle Open World for a few days now.  In that time, I’ve had numerous conversations about running Oracle Enterprise Manager 12c on Oracle Database 12c.  I will be honest and say that I’ve enjoyed these conversations; however, after about the fourth time I decided I need to write a quick post on the explanation discussed in these conversations.  

Early this year (August) I wrote a post about the what came out of the OEM CAB in May 2014 and how to get OEM 12c to work on DB12c.  The concept of running OEM 12c on DB12c, pluggable or not, have many people excited and looking forward to configuring OEM to do that very configuration.  Heck, I’ve even installed it for a few customers in that configuration (non-PDB).  So I’m a bit sad in having to say this:  ORACLE DATABASE 12c SUPPORT FOR…

View original post 211 more words

Oracle Database 12c – Interactive Quick Reference

Happened to run into this while Googling around.  It’s quite useful and well-produced for anyone interested in learning about the 12c RDBMS architecture, V$ views, how the processes relate to each other, and need an overview of how the Container and/or Pluggable multitenant features work.

http://www.oracle.com/webfolder/technetwork/tutorials/obe/db/12c/r1/poster/OUTPUT_poster/poster.html

Sample Oracle 12c Database Interactive Quick Reference Pages

Configure ASMM and AMM in Oracle

ORA-06512 ‘DBSNMP.BSLN_INTERNAL’

Periodic Stats Gathering Fails in 11gR2 Oracle Database – found in alert.log: ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073

The Fix:

SQL> @?/rdbms/admin/catnsnmp.sql

SQL> @?/rdbms/admin/catsnmp.sql

Dbhk's Blog

Symptoms

1.  BSLN_MAINTAIN_STATS_JOB failed at every SUNDAY

2.  Alert messages got the following

Errors in file /opt/oracle/diag/rdbms/vstbpro/vstbpro1/trace/vstbpro1_j000_7796.trc:
ORA-12012: error on auto execute of job 11762
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073

3. Details of the trace file

Trace file /opt/oracle/diag/rdbms/vstbpro/vstbpro1/trace/vstbpro1_j000_7796.trc
Oracle Database 11g Release 11.1.0.7.0 – 64bit Production
With the Real Application Clusters option
ORACLE_HOME = /opt/oracle/product/11g
System name:    SunOS
Node name:      wvpdb09
Release:        5.10
Version:        Generic_138888-01
Machine:        sun4u
Instance name: vstbpro1
Redo thread mounted by this instance: 1
Oracle process number: 73
Unix process pid: 7796, image: oracle@wvpdb09 (J000)

*** 2010-04-11 15:00:04.639
*** SESSION ID:(1041.32020) 2010-04-11 15:00:04.639
*** CLIENT ID:() 2010-04-11 15:00:04.639
*** SERVICE NAME:(SYS$USERS) 2010-04-11 15:00:04.639
*** MODULE NAME:(DBMS_SCHEDULER) 2010-04-11 15:00:04.639
*** ACTION NAME:(BSLN_MAINTAIN_STATS_JOB) 2010-04-11 15:00:04.639

ORA-12012: error on auto execute of job 11762
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at “DBSNMP.BSLN_INTERNAL”, line 2073
ORA-06512: at line 1

Cause

Table DBSNMP.BSLN_BASELINES contains…

View original post 35 more words