Oracle DataGuard and Standby Database Archive Logs

Users see your wonderful DataGuard implementation like this:

Simple Oracle Dataguard Architecture
Simple Oracle Dataguard Architecture courtesy of https://appsdbatraining.files.wordpress.com

And yet, you know the actual picture looks more like this:

A map of Oracle DataGuard Architecture components
A map of Oracle DataGuard Architecture components

High-availability – the concept behind it makes every DBA shudder because every time it seems you deal with one element and have it protected, there’s another underlying component that also needs protection and redudancy, or else your solution is still insufficient.

Real Application Clusters (RAC) covers individual database host failures but is sensitive to failure of the storage subsystem or the network interconnections between the hosts.

Recovery Manager (RMAN) is your vital tool to keeping track of what’s backed up and where is it.  And its catalog of recovery information could reside locally in copies of the controlfiles, or centrally in another database.  Depends on your backup strategies, really.  Are you using SAN-based backups (snaps, virtual images, deduplicated block replication) or off-site methods that would have to be shipped back to start recovery?

But the typical first-time setup scenario, is you use the OEM-based jiffy whizbang method to setup your new DataGuard environment at the recommendation of one of the steps int the Maximum Availability Advisor (MAA), and everything’s up and running nicely.  You schedule a new weekly full backup, plus daily incremental backup as Oracle recommended practices prescribe, and notice everything’s running smoothly.

Except on your standby database, the archivelogs are piling up and not being deleted automatically.  What’s next?

This thread was a basic discussion in the Oracle Community forums of the topic:

https://community.oracle.com/thread/2388130?start=0&tstart=0

This is a typical RMAN-based configuration:

On Primary

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

On Standby ( Depends upon where backup is preformed )

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;

Or

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

/* if Standby Where Backups Are Not Performed  */

Source:

Data Guard Concepts and Administration 12c

http://docs.oracle.com/database/121/SBYDB/toc.htm

12 Using RMAN to Back Up and Restore FilesRMAN Configurations at the Primary Database
12.3.4 RMAN Configurations at a Standby Where Backups Are Not Performed

The following RMAN configurations are recommended at a standby database where backups are not done:

  1. Connect RMAN to the standby database as target, and to the recovery catalog.
  2. Enable automatic deletion of archived logs once they are applied at the standby database (this is also applicable to all terminal databases when the cascading or far sync instance features are in use):
  3. CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

 

** However, that doesn’t really take into consideration what might happen if a final archivelog before switchover of roles doesn’t quite make it on the standby (for whatever reason, it gets corrupted during playback or something similar which results in a Database Needs More Recovery error.)

Based upon:

12.3.3 RMAN Configurations at a Standby Database Where Backups are Performed

The following RMAN configurations are recommended at a standby database where backups are done:

  1. Connect RMAN to the standby database (where backups are performed) as target, and to the recovery catalog.
  2. Enable automatic backup of the control file and the server parameter file:

3.  RMAN > CONFIGURE CONTROLFILE AUTOBACKUP ON;

  1. Skip backing up data files for which there already exists a valid backup with the same checkpoint:

5.  RMAN > CONFIGURE BACKUP OPTIMIZATION ON;

  1. Configure the tape channels to create backups as required by media management software:

7.  RMAN > CONFIGURE CHANNEL DEVICE TYPE SBT PARMS ‘<channel parameters>’;

  1. Because the archived logs are backed up at the standby database, Oracle recommends that you configure the BACKED UP option for the log deletion policy:

9.  RMAN > CONFIGURE ARCHIVELOG DELETION POLICY BACKED UP n TIMES TO [DEVICE TYPE SBT];

 

I came up with the configuration of:

 

On Standby ( Depends upon where backup is preformed )

# If no DataGuard is present (single DB host):

# Ensure daily RMAN backup job is being executed in OEM or via cron.

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;

 

And setup a 2nd backup set for the Standby Host DB.  This puts the backups into the defined Fast Recovery Area and manages both the backups and archivelog retention in the same mountpoint.

 

Each has its pros and cons depending on the scenario.You need to lay out your entire architecture scheme including backup solutions and play out the various scenarios that you’re required to cover as far as your Quality of Service (QOS) guarantee to your end-user population.

And of course, if you’re using the advanced cross-WAN FarSync DataGuard implementation architecture (wherein there’s a separate Failover Archive Log (FAL) standby database whose sole purpose in life is to cache archivelogs in case the data replication stream is too much for WAN bandwidth to handle in real-time.) this all still applies because the FAL server is basically just another standby target which needs managing just as much as any regular LAN-based full DataGuard standby instance (it’s just missing the big datafiles and handles all the archivelog traffic.)  You’ll just have even more servers and services involved in keeping the whole thing running (like your Global Names Service servers and databases, which might also be RAC and DataGuard protected, or your Single-Sign On authentication services, or even the OEM Cloud Control OMS itself orchestrating all of that.)

 

Advertisements

Nespresso Vertuoline Coffee Capsule Brew Formulas

I like Nespresso’s Vertuoline single brew appliance – not because it’s convenient (which it is), nor because it’s quick (ditto). But because of inventive others like My-Cap.com, the rather otherwise wasteful aluminum capsules can be re-used indefinitely (as long as you’re careful not to pierce or dent them too much through handling.)

My-Cap.com foil kit for Nespresso Vertuoline
My-Cap.com foil kit for Nespresso Vertuoline

My-Cap.com makes foils, replacement capsules, little plastic quick caps, and all sorts of accessories for single-use pod/capsule coffee makers which definitely extends their environmental friendliness geometrically (which otherwise, at 1 plastic or aluminum pod per cup creates a ridiculous amount of landfill over time, if not properly recycled – Nespresso is one of the only company that provides free shipping for recycling capsules.)

My full user review is up at Amazon.com:

Nespresso Vertuoline Finished Lungo Custom Deathwish

https://www.amazon.com/review/R2CA7L5M1YTU1M/ref=cm_cr_rdp_perm

But this post is about a more in-depth feature about the Vertuoline capsule brewing formulation coded into the barcodes that surround the bottoms of each capsule.

Only recently has Nespresso started revealing the numerous formulas used in each version of the capsules, allowing those of us doing the re-use/re-pack/re-foil thing, to properly select a barcode that will work best with the coffee being refilled.

I’ll try to keep the following table updated as new information arrives on this subject:

Capsule Type Color Prewetting Flow Temperature
Altissio Espresso Dk Purple Short Slow High
Diavolitto Espresso Dk Blue Long Slow High
Voltesso Espresso Bright Gold Short Slow Low
Decaf Intenso Espresso Dk Red Long Fast High
Giornio Lungo Orange Long Slow Low
Solelio Lungo Yellow Short Fast Low
Intensio Lungo Dk Brown Long Med High
Stormio Lungo Dk Green Long Slow High
Odacio Lungo Med Blue Long Med High
Melozio Lungo Dk Gold Short Fast High
Elvazio Lungo Pink Short Slow->Fast Low->High
Decaffeinato Lungo Red Short Med Low
Half Caffeinato Lungo Red/Black Short Med Low
Cafe de Cuba Lungo White/Red script Short Slow Low
Flavored Lungo Various Short Med Low
My-Cap.com custom Deathwish after brewing
My-Cap.com custom Deathwish after brewing

I use #2 espresso grind for the Lungo size capsules (taking 10-12g of coffee grounds), and #1 espresso for the Espresso size (which take from 5-8g by comparison).  I pack each to within 1mm of the top of the flat rim of the capsule, which allows plenty of expansion room during the pre-wetting stage.

To remove the original foil, just hobby knife around the rounded part of the rim inside the flatter portion that the foil is glued to leaving a nice flat foil ring for the reusable foils to adhere to during extraction.

A cardboard my-cap storage sleeve
A cardboard my-cap storage sleeve

I also made up a few cardboard sleeves (ala Pringles style cans) that close-fit to the edges of the capsules to better keep the re-use foils on the capsules (the adhesive works well during extraction, but I disliked all the extra clamping and crimping others were doing to attempt to seal them better.  I find just placing them on top,  and running around the edge with the fan-brush handle is fine, then just gently fold over the edges.  The adhesive is enough to seal against the rim lip during extraction, and as long as you keep the capsules upright, they won’t spill.)

The price for convenience is that these little capsules do use about 200% more coffee per cup in order to reach the strength level of a standard French press style cup of coffee, but produce a good 2cm of crema in the process (more like a Vev-Vigano stove-top pressure coffee extractor.)  The espresso versions are a little lighter,  in that the brewing process is modified and thus consume closer to the “normal” puck’s worth of coffee per espresso (and they can be double-brewed – reset the cycle by turning the lock open, but don’t open it, then re-lock again to allow the 2nd button press, and re-trigger the pre-wetting cycle, if you prefer something like a 1-1/2 Espresso.)

OEM 12c SSL Certificate Swapping (HowTo)

"This Connection is Untrusted" error message
“This Connection is Untrusted” error message

Oracle Enterprise Manager out of the box, comes with demonstration SSL certificates that are generally okay for getting the basic system up and running, but should not be left as your long-term solution for SSL/HTTPS connections to your Oracle Management Server (OMS).

Similar to how e-Business Suite installations delivered a DEMO Certificate Authority certificate with the bundled Internet Application Server (iAS) installation, OEM packages do the same thing.  But eventually, forced by browser and client workstation OS upgrades, you will eventually need to install “real” certificates by a true trusted Root Certification Authority (RCA) so that your client browsers don’t begin rejecting encrypted connections to your OMS.

If you search for SSL Certificate authorities, there are many well-known public RCA’s  such as, DigiCert, Verisign, Thawte, GeoTrust, and others, or even those available from your domain registrar.  Larger organizations probably have their own Certificate Authority signing server on-premise that allow generation of trusted certificates, as well. The only really important thing is that the CA is actually available in your browser and OS as a Trusted Root Authority, and that the signing chain is verifiable to prevent issues with SSL/HTTPS handshaking.  For each middle-tier OMS host, or virtual host if you are set up for high-availability with multiple WebLogic servers, a certificate request is generated, signed and then imported back into the keychains related to the OMS Weblogic hosts, and the OMS Servicing Agents (the OEM Agents installedo on the OMS middle-tier hosts.)

Once a new certificate is installed to the OMS itself (in WebLogic), you will also need to install the related RCA to the OMS-side OEM Agent servicing all of the connections to the other OEM Agents, so that they too, will be SSL enabled.

# OEM SSL Certificate swapping
#    EM 12c Cloud Control: How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS Console application ? (Doc ID 1937457.1)
#     EM 12c: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager 12c Cloud Control Installation (Doc ID 1527874.1)
STAGE_DIR=/mnt/nfs/FMW/certs
EM_INSTANCE_HOME=/oemgc/Oracle/gc_inst2/em/EMGC_OMS1      #WebTierIH2 OHS/ohs2 on (alternate hostname

oemmgr@(primary hostname) $> cd $STAGE_DIR

oemmgr@(primary hostname) $> cat import.sh
export JAVA_JREBIN=$JAVA_HOME/jre/bin
export CERTS=/mnt/nfs/FMW/certs
$JAVA_JREBIN/keytool -import -file $CERTS/ORGPOLICYCA.cer -trustcacerts -alias ORGROOTCA -storepass changeit -noprompt  -keystore $JAVA_HOME/jre/lib/security/cacerts
$JAVA_JREBIN/keytool -import -file $CERTS/ORGROOTCA.cer -trustcacerts -alias ORGRootPolicyCA -storepass changeit -noprompt   -keystore $JAVA_HOME/jre/lib/security/cacerts
$JAVA_JREBIN/keytool -import -file $CERTS/ORGHOSTISSUECA1.cer -trustcacerts -alias ORGHOSTissueca1 -storepass changeit -noprompt  -keystore  $JAVA_HOME/jre/lib/security/cacerts

oemmgr@(primary hostname) $> . ./import.sh
Certificate was added to keystore
Certificate was added to keystore
Certificate was added to keystore

# Determine keystore directory (found by locating your Oracle Home Service (OHS) installation filesystem)
oemmgr@(primary hostname) $> ps -ef | grep ohs
oemmgr     873  9334  0 Jan13 ?        00:47:48 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9334  9305  0 Jan13 ?        00:00:06 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9342  9334  0 Jan13 ?        00:00:11 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs -l /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/ohs1-%Y%m%d%H%M%S.log 10M 70M
oemmgr    9344  9334  0 Jan13 ?        00:00:11 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/access_log 10M 100M
oemmgr    9345  9334  0 Jan13 ?        00:00:01 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_http_access_log 10M 100M
oemmgr    9346  9334  0 Jan13 ?        00:02:13 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M
oemmgr    9349  9334  0 Jan13 ?        00:00:07 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs.log 10M 100M
oemmgr    9350  9334  0 Jan13 ?        00:00:00 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs -l -h:/oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/component_events.xml_ohs1 /oemgc/Oracle/gc_inst2/WebTierIH1/auditlogs/OHS/ohs1/audit-pid9334-%Y%m%d%H%M%S.log 1M 4M
oemmgr    9351  9334  0 Jan13 ?        00:00:28 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9352  9334  0 Jan13 ?        00:47:55 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9353  9334  0 Jan13 ?        00:47:52 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL

# Confirm settings
oemmgr@(primary hostname) $> grep keystore /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/ssl.conf
SSLWallet file:/oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/console

# Stage copy of revised CA wallet                                  #WebTierIH2/config/OHS/ohs2 on (secondary hostname)
oemmgr@(primary hostname) $> cp -r /mnt/nfs/FMW/certs/oemgc.domain /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores
cp -r /mnt/nfs/FMW/certs/oemgc.domain /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores

# Check permissions 770 on wallet dir, 600 on wallets
oemmgr@(secondary hostname) $> ls -la /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores/oemgc.domain
ls -la /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
total 32
drwxrwx— 2 oemmgr oinstall  4096 Mar  9 12:50 .
drwx—— 7 oemmgr oinstall  4096 Mar  9 12:50 ..
-rw——- 1 oemmgr oinstall 11653 Mar  9 12:50 cwallet.sso
-rw——- 1 oemmgr oinstall 11576 Mar  9 12:50 ewallet.p12

# Primary wallet for the OMS console
oemmgr@(primary hostname) $> cd /oemgc/Oracle/MW3/oracle_common/bin
oemmgr@(primary hostname) $> ./orapki wallet display -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/console
Oracle PKI Tool : Version 11.1.1.7.0
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=oemgc.domain
Trusted Certificates:
Subject:        CN=(primary hostname).domain,C=US,ST=CA,L=EnterpriseManager on (primary hostname).domain,OU=EnterpriseManager on (primary hostname).domain,O=EnterpriseManager on (primary hostname).domain

# Confirm new wallet contents
oemmgr@(primary hostname) $> ./orapki wallet display -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
Oracle PKI Tool : Version 11.1.1.7.0
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=*.domain,OU=Information Technology,O=ORG My Org,L=My City,ST=California,C=US
Trusted Certificates:
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=ORG POLICY CA
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Subject:        CN=ORG ROOT CA
Subject:        CN=HOSTISSUECA1,DC=fss,DC=ORG,DC=com
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

#Add certificates to monitoring agent for the OMS:
cd $AGENT_HOME/bin
./emctl stop agent
# Default jks keyring password – welcome
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGROOTCA.cer -alias ORGROOTCA
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGPOLICYCA.cer -alias ORGPOLICYCA
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGHOSTISSUECA1.cer -alias HOSTISSUECA1
./emctl start agent

# Support virtual host ignore hostname verification
export EM_COMMON_JAVA_OPTIONS=”-Dweblogic.security.SSL.ignoreHostnameVerification=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.log.FileName=/oemgc/Oracle/gc_inst2/em/EMGC_OMS1/sysman/log/wls.log”

# Backup the EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properites file
cd /oemgc/Oracle/gc_inst2/em/EMGC_OMS1                               #OMS2 on (secondary hostname)
cp emgc.properties emgc.properties_selfsign

# Requires SYSMAN password
# (secondary hostname)
$OMS_TOP/bin/emctl secure console -wallet /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores/oemgc.domain

# Example output
# Oracle Enterprise Manager Cloud Control 12c Release 5
# Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
# Securing Console… Started.
# Enter Enterprise Manager Root (SYSMAN) Password :
# Securing Console… Successful
# Restart OMS

# (primary hostname)
$OMS_TOP/bin/emctl secure console -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
$OMS_TOP/bin/emctl stop oms
$OMS_TOP/bin/emctl start oms

# 10-JUN-2016 Addendum – enable emcli login by establishing trust for the new certificate

oemmgr@(primary hostname)$> ./emcli setup -url=https://oemgc.auca.corp:7799/em -username=”SYSMAN” -password=$SYSMAN_PW

Oracle Enterprise Manager 12c Release 5.
Copyright (c) 1996, 2015 Oracle Corporation and/or its affiliates. All rights reserved.

The configuration directory “/home/oemmgr” may not be local. See the “dir” option in the help for the setup command.
Do you want to continue using this directory? [yes/no] yes

Warning: This certificate has not been identified as trusted in the local trust store
————————————–
[<blah – lots of cryptic information about the new certificate>
]
————————————–
Do you trust the certificate chain? [yes/no] yes
Emcli setup successful

# Test emcli connectivity

oemmgr@(primary hostname)$> ./emcli login -username=”SYSMAN” -password=$SYSMAN_PW

Login successful

#Section 4: Rolling back to the Demonstration WLS Certificate
#If you need to switch back the WLS components in the OMS installation to use the default WebLogic Server demonstration certificates, execute the following steps on each OMS.

1.Stop the OMS:

cd <OMS_HOME>/bin
emctl stop oms

2.Run the following command:

cd <OMS_HOME>/bin>
emctl secure wls -use_demo_cert
emctl secure console -self_signed

3.Stop the OMS:

cd <OMS_Home>/bin
emctl stop oms -all

4.Start the OMS:

cd <OMS_Home>/bin
emctl start oms