Tag Archives: SSL

OEM 12c SSL Certificate Swapping (HowTo)

"This Connection is Untrusted" error message
“This Connection is Untrusted” error message

Oracle Enterprise Manager out of the box, comes with demonstration SSL certificates that are generally okay for getting the basic system up and running, but should not be left as your long-term solution for SSL/HTTPS connections to your Oracle Management Server (OMS).

Similar to how e-Business Suite installations delivered a DEMO Certificate Authority certificate with the bundled Internet Application Server (iAS) installation, OEM packages do the same thing.  But eventually, forced by browser and client workstation OS upgrades, you will eventually need to install “real” certificates by a true trusted Root Certification Authority (RCA) so that your client browsers don’t begin rejecting encrypted connections to your OMS.

If you search for SSL Certificate authorities, there are many well-known public RCA’s  such as, DigiCert, Verisign, Thawte, GeoTrust, and others, or even those available from your domain registrar.  Larger organizations probably have their own Certificate Authority signing server on-premise that allow generation of trusted certificates, as well. The only really important thing is that the CA is actually available in your browser and OS as a Trusted Root Authority, and that the signing chain is verifiable to prevent issues with SSL/HTTPS handshaking.  For each middle-tier OMS host, or virtual host if you are set up for high-availability with multiple WebLogic servers, a certificate request is generated, signed and then imported back into the keychains related to the OMS Weblogic hosts, and the OMS Servicing Agents (the OEM Agents installedo on the OMS middle-tier hosts.)

Once a new certificate is installed to the OMS itself (in WebLogic), you will also need to install the related RCA to the OMS-side OEM Agent servicing all of the connections to the other OEM Agents, so that they too, will be SSL enabled.

# OEM SSL Certificate swapping
#    EM 12c Cloud Control: How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS Console application ? (Doc ID 1937457.1)
#     EM 12c: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager 12c Cloud Control Installation (Doc ID 1527874.1)
EM_INSTANCE_HOME=/oemgc/Oracle/gc_inst2/em/EMGC_OMS1      #WebTierIH2 OHS/ohs2 on (alternate hostname

oemmgr@(primary hostname) $> cd $STAGE_DIR

oemmgr@(primary hostname) $> cat import.sh
export JAVA_JREBIN=$JAVA_HOME/jre/bin
export CERTS=/mnt/nfs/FMW/certs
$JAVA_JREBIN/keytool -import -file $CERTS/ORGPOLICYCA.cer -trustcacerts -alias ORGROOTCA -storepass changeit -noprompt  -keystore $JAVA_HOME/jre/lib/security/cacerts
$JAVA_JREBIN/keytool -import -file $CERTS/ORGROOTCA.cer -trustcacerts -alias ORGRootPolicyCA -storepass changeit -noprompt   -keystore $JAVA_HOME/jre/lib/security/cacerts
$JAVA_JREBIN/keytool -import -file $CERTS/ORGHOSTISSUECA1.cer -trustcacerts -alias ORGHOSTissueca1 -storepass changeit -noprompt  -keystore  $JAVA_HOME/jre/lib/security/cacerts

oemmgr@(primary hostname) $> . ./import.sh
Certificate was added to keystore
Certificate was added to keystore
Certificate was added to keystore

# Determine keystore directory (found by locating your Oracle Home Service (OHS) installation filesystem)
oemmgr@(primary hostname) $> ps -ef | grep ohs
oemmgr     873  9334  0 Jan13 ?        00:47:48 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9334  9305  0 Jan13 ?        00:00:06 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9342  9334  0 Jan13 ?        00:00:11 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs -l /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/ohs1-%Y%m%d%H%M%S.log 10M 70M
oemmgr    9344  9334  0 Jan13 ?        00:00:11 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/access_log 10M 100M
oemmgr    9345  9334  0 Jan13 ?        00:00:01 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_http_access_log 10M 100M
oemmgr    9346  9334  0 Jan13 ?        00:02:13 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M
oemmgr    9349  9334  0 Jan13 ?        00:00:07 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs /oemgc/Oracle/gc_inst2/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs.log 10M 100M
oemmgr    9350  9334  0 Jan13 ?        00:00:00 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/odl_rotatelogs -l -h:/oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/component_events.xml_ohs1 /oemgc/Oracle/gc_inst2/WebTierIH1/auditlogs/OHS/ohs1/audit-pid9334-%Y%m%d%H%M%S.log 1M 4M
oemmgr    9351  9334  0 Jan13 ?        00:00:28 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9352  9334  0 Jan13 ?        00:47:55 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL
oemmgr    9353  9334  0 Jan13 ?        00:47:52 /oemgc/Oracle/MW3/Oracle_WT/ohs/bin/httpd.worker -DSSL

# Confirm settings
oemmgr@(primary hostname) $> grep keystore /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/ssl.conf
SSLWallet file:/oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/console

# Stage copy of revised CA wallet                                  #WebTierIH2/config/OHS/ohs2 on (secondary hostname)
oemmgr@(primary hostname) $> cp -r /mnt/nfs/FMW/certs/oemgc.domain /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores
cp -r /mnt/nfs/FMW/certs/oemgc.domain /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores

# Check permissions 770 on wallet dir, 600 on wallets
oemmgr@(secondary hostname) $> ls -la /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores/oemgc.domain
ls -la /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
total 32
drwxrwx— 2 oemmgr oinstall  4096 Mar  9 12:50 .
drwx—— 7 oemmgr oinstall  4096 Mar  9 12:50 ..
-rw——- 1 oemmgr oinstall 11653 Mar  9 12:50 cwallet.sso
-rw——- 1 oemmgr oinstall 11576 Mar  9 12:50 ewallet.p12

# Primary wallet for the OMS console
oemmgr@(primary hostname) $> cd /oemgc/Oracle/MW3/oracle_common/bin
oemmgr@(primary hostname) $> ./orapki wallet display -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/console
Oracle PKI Tool : Version
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=oemgc.domain
Trusted Certificates:
Subject:        CN=(primary hostname).domain,C=US,ST=CA,L=EnterpriseManager on (primary hostname).domain,OU=EnterpriseManager on (primary hostname).domain,O=EnterpriseManager on (primary hostname).domain

# Confirm new wallet contents
oemmgr@(primary hostname) $> ./orapki wallet display -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
Oracle PKI Tool : Version
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=*.domain,OU=Information Technology,O=ORG My Org,L=My City,ST=California,C=US
Trusted Certificates:
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=ORG POLICY CA
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Subject:        CN=ORG ROOT CA
Subject:        CN=HOSTISSUECA1,DC=fss,DC=ORG,DC=com
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

#Add certificates to monitoring agent for the OMS:
cd $AGENT_HOME/bin
./emctl stop agent
# Default jks keyring password – welcome
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGROOTCA.cer -alias ORGROOTCA
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGPOLICYCA.cer -alias ORGPOLICYCA
./emctl secure add_trust_cert_to_jks -trust_certs_loc /mnt/nfs/FMW/certs/ORGHOSTISSUECA1.cer -alias HOSTISSUECA1
./emctl start agent

# Support virtual host ignore hostname verification
export EM_COMMON_JAVA_OPTIONS=”-Dweblogic.security.SSL.ignoreHostnameVerification=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.log.FileName=/oemgc/Oracle/gc_inst2/em/EMGC_OMS1/sysman/log/wls.log”

# Backup the EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properites file
cd /oemgc/Oracle/gc_inst2/em/EMGC_OMS1                               #OMS2 on (secondary hostname)
cp emgc.properties emgc.properties_selfsign

# Requires SYSMAN password
# (secondary hostname)
$OMS_TOP/bin/emctl secure console -wallet /oemgc/Oracle/gc_inst2/WebTierIH2/config/OHS/ohs2/keystores/oemgc.domain

# Example output
# Oracle Enterprise Manager Cloud Control 12c Release 5
# Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
# Securing Console… Started.
# Enter Enterprise Manager Root (SYSMAN) Password :
# Securing Console… Successful
# Restart OMS

# (primary hostname)
$OMS_TOP/bin/emctl secure console -wallet /oemgc/Oracle/gc_inst2/WebTierIH1/config/OHS/ohs1/keystores/oemgc.domain
$OMS_TOP/bin/emctl stop oms
$OMS_TOP/bin/emctl start oms

# 10-JUN-2016 Addendum – enable emcli login by establishing trust for the new certificate

oemmgr@(primary hostname)$> ./emcli setup -url=https://oemgc.auca.corp:7799/em -username=”SYSMAN” -password=$SYSMAN_PW

Oracle Enterprise Manager 12c Release 5.
Copyright (c) 1996, 2015 Oracle Corporation and/or its affiliates. All rights reserved.

The configuration directory “/home/oemmgr” may not be local. See the “dir” option in the help for the setup command.
Do you want to continue using this directory? [yes/no] yes

Warning: This certificate has not been identified as trusted in the local trust store
[<blah – lots of cryptic information about the new certificate>
Do you trust the certificate chain? [yes/no] yes
Emcli setup successful

# Test emcli connectivity

oemmgr@(primary hostname)$> ./emcli login -username=”SYSMAN” -password=$SYSMAN_PW

Login successful

#Section 4: Rolling back to the Demonstration WLS Certificate
#If you need to switch back the WLS components in the OMS installation to use the default WebLogic Server demonstration certificates, execute the following steps on each OMS.

1.Stop the OMS:

cd <OMS_HOME>/bin
emctl stop oms

2.Run the following command:

cd <OMS_HOME>/bin>
emctl secure wls -use_demo_cert
emctl secure console -self_signed

3.Stop the OMS:

cd <OMS_Home>/bin
emctl stop oms -all

4.Start the OMS:

cd <OMS_Home>/bin
emctl start oms


Big Data Cloud Storage Framework Solutions

Based upon a recent meeting of minds at MESS (Media Entertainment & Scientific Systems) http://www.meetup.com/MESS-LA the challenge facing the industries is how to deal with petabyte-sized amassed data that still needs to be accessible in real-time for secured editing purposes by downstream customers and suppliers.
Here’s a multi-phase solution idea:

My idea for a secure big file delivery model. Figured by J. H. Lui (c) 2016

Using torrent technology for access, with authenticated peer-to-peer hosts. private SSL-encrypted trackers/announcers, and encrypted bit streams, this maintains access to the fundamental data source using minimal infrastructure.
Add two-factor authentication to the authentication protocol to allow time- and role-based security to be enforced (so-and-so p2p host is authorized to connect to the torrent during X days/N hours per day/etc.)
Use generic two-factor authentication providers (e.g. Symantec VIP or SAASPass) to allow the small service providers to access data without excessive overhead cost, or dedicated hardware solutions.
Store the data source files using a torrent+sharding+bit-slicing protocol (similar to the Facebook imaging storage model.) Without authenticated access to the cloud torrent, any individual data chunk or shard grabbed by a sniffer becomes useless.
Segregate and divide the data files using a role-based security architecture (e.g. Scene 1 needed by X post-production editor, during N time-period.) Individual torrent participants can select the individual virtual file segments they need for work, without downloading the data chunks unrelated to them. Similarly, the above described time+role based security prevents access to/from data segments that are not authorized for that endpoint. Could even add password-protection to individual sensitive segments to provide one more level of turn-key security.
Use a Google Drive/Dropbox style OS protocol to allow mounting of the torrent sources to the end-user workstations with transparent access.  Whichever mechanism can provide adequate latency for the block replication should be sufficient.  Rather than mounting the same cloud torrent to every local workstation, use local NFS servers to provide local home-basing of the cloud mount (WAN speed), then export that mount locally (LAN speed) to the various workstations that need access to it.  That way, there’s only one penetration point to/from the cloud torrent, which can be adequately firewalled locally by the end-user. This is a solution for the end consumers that need access to the largest portion of the cloud data set.
The source data hives can use multi-path networking protocol ( https://jhlui1.wordpress.com/2015/05/21/multi-path-multiplexed-network-protocol-tcpip-over-mmnp-redundant-connections ) to further split and sub-divide the data streams (which are already encrypted), to maximize performance to bandwidth-limited consumer endpoints.
Media companies have a rather different data value model to deal with because during pre-production the data value is extremely high, but it drops off rapidly post-production release once the market consumes it. But the same model at a lower protection level would work for actual distribution – wherein end subscribers are authenticated for access to a particular resolution or feature set of the original cloud segments (e.g. 8K versus 1K media, or audio-only, or with or without Special Features access.)

@Oracle R12 EBS HTTP Server Error Child Could Not Open Sslmutex Lockfile

HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).

mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/ (System error follows)
System: No such file or directory (errno: 2)

And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).

applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: ENV_ID.myhostname
ias-component                    | process-type       |     pid | status 
OC4JGroup:default_group          | OC4J:oafm          |   16788 | Alive  
OC4JGroup:default_group          | OC4J:forms         |   16709 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |   21689 | Alive  
HTTP_Server                      | HTTP_Server        |   16494 | Stop

OEM will show the HTTP service as being down, even though the pages are probably still responding:

R12 EBS OEM AMS monitor HTTP Service down screenshot
R12 EBS OEM AMS monitor HTTP Service down screenshot

Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.

These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration.  These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:

# ###############################################################
# This file is automatically generated by AutoConfig.  It will be read and
# overwritten.  If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
# ###############################################################

## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop

#  Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *

ssl.conf:SSLMutex  file:/ptcharmk/inst/apps/${ENV_ID}/logs/ora/10.1.3/Apache/ssl_mutex

# Append a line to the existing custom template
echo -e “\nAcceptMutex pthread\n” >> $FND_TOP/admin/template/custom_conf_1013.tmp

# Modify the existing SSL template (doesn’t seem to have a custom include on this one)
vi $FND_TOP/admin/template/ssl_conf_1013.tmp

Comment:  # SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
Insert:     SSLMutex sem

# Execute AutoConfig

$SCRIPT_TOP/adautocfg.sh -appspass=$APPS_PW

# Re-Start Apache
$SCRIPT_TOP/adapcctl.sh start

# After Change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep -i utex $INST_TOP/ora/10.1.3/Apache/Apache/conf/*

custom.conf:AcceptMutex pthread
ssl.conf:# SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
ssl.conf:SSLMutex  sem

# Confirm semaphore mode is activated

applmgr@myhostname $> grep -i mutex $INST_TOP/logs/ora/10.1.3/Apache/*
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 11:58:14 2015] [notice] Accept mutex: pthread (Default: fcntl)
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 12:12:12 2015] [notice] Accept mutex: pthread (Default: fcntl)

As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.

SFO Airport Free WiFi and an Android Samsung S4


So you might have found it annoying
that your fuzzy new Android phone/tablet doesn’t like to connect to WiFi in certain places with the terms and conditions to accept before allowing you to connect.
For me and my Samsung S4 Mini, it turned out to be a minor issue with Chrome’s pop up blocking.


After getting the usual error page like this: [SSL CONNECTION ERROR]


I opened another window and happened to be on the flysfo.com


website going through the Samsung instructions when I noticed the page suddenly changed and up popped the Accept Terms and Conditions page.
Accepted it and off I went on SFO Free WiFi (yea.)
You’ll also notice the screenshot shows the gateway address for their portal service:
– hitting that link may have helped, too.