@Oracle R12 EBS HTTP Server Error Child Could Not Open Sslmutex Lockfile

HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).

mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/10.1.3.1/Apache/Apache/logs/ssl_mutex.10549 (System error follows)
System: No such file or directory (errno: 2)

And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).

applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: ENV_ID.myhostname
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status 
---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm          |   16788 | Alive  
OC4JGroup:default_group          | OC4J:forms         |   16709 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |   21689 | Alive  
HTTP_Server                      | HTTP_Server        |   16494 | Stop

OEM will show the HTTP service as being down, even though the pages are probably still responding:

R12 EBS OEM AMS monitor HTTP Service down screenshot

Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.

These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration.  These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:

# ###############################################################
#
# This file is automatically generated by AutoConfig.  It will be read and
# overwritten.  If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
)

## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop

#  Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *

ssl.conf:SSLMutex  file:/ptcharmk/inst/apps/${ENV_ID}/logs/ora/10.1.3/Apache/ssl_mutex

# Append a line to the existing custom template
echo -e “\nAcceptMutex pthread\n” >> $FND_TOP/admin/template/custom_conf_1013.tmp

# Modify the existing SSL template (doesn’t seem to have a custom include on this one)
vi $FND_TOP/admin/template/ssl_conf_1013.tmp
=============

Comment:  # SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
Insert:     SSLMutex sem

# Execute AutoConfig

$SCRIPT_TOP/adautocfg.sh -appspass=$APPS_PW

# Re-Start Apache
$SCRIPT_TOP/adapcctl.sh start

# After Change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep -i utex $INST_TOP/ora/10.1.3/Apache/Apache/conf/*

custom.conf:AcceptMutex pthread
ssl.conf:# SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
ssl.conf:SSLMutex  sem

# Confirm semaphore mode is activated

applmgr@myhostname $> grep -i mutex $INST_TOP/logs/ora/10.1.3/Apache/*
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 11:58:14 2015] [notice] Accept mutex: pthread (Default: fcntl)
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 12:12:12 2015] [notice] Accept mutex: pthread (Default: fcntl)

As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.

Advertisement

e-Business Suite Patches ONE-OFF, or Family Pack – My @Oracle Support Searches

One-off patch, then [Search]

Patches & Updates Name or Bug Number (Simple)

Click on the Patch# (link):

Patch Search Results

Showing that this particular patch was not superceded by a Family pack release:But in the README.txt – the giveaway is the “Target Mini-pack/Family Pack/Release” – 12.1.4/12.2 means this won’t be incorporated into a 12.1.3 Family pack and will only be included in an eventual 12.1.4 upgrade, or in the 12.2 upgrade.  It would remain a ONE-OFF status for 12.1.3.

Patch README.txt

The other approach is going from the RPC downward:

Patches & Updates Product or Family (Advanced)

For a Functional Overview of this patch please review EBS: R12.1: Receivables

Recommended Patch Collection (RPC), Feb 2015  (Doc ID 1980300.1)

Refer to Doc ID 954704.1 for all Financials Feb 2015 RPCs.

nb – In case it’s useful – you can make a browser shortcut for any My Oracle Support Document ID’s by using the search URL: https://support.oracle.com/epmos/faces/DocContentDisplay?id=(documentID#)

R12.2.3 / R12.2.x Oracle e-Business Suite – Resetting the WebLogic Password

…because occasionally you either forget it (or in my case, a technology patch reset it to something I wasn’t sure about.)

There are 2 basic routes:

---

The “short” process – re-run the post-clone config (adcfgclone.pl):

# Have to remove the existing FMW home because it will re-unzip a new one from the clone staging archive
applptch@aburpaofinm01 $> cd /u01/fs2
applptch@aburpaofinm01 $> ls
EBSapps  FMW_Home  inst
applptch@aburpaofinm01 $> ls -latr
total 24
drwxrwxr-x  8 applptch oinstall 4096 May  8 21:49 ..
drwxr-xr-x  5 applptch oinstall 4096 May  9 12:33 EBSapps
drwxr-xr-x  3 applptch oinstall 4096 Jul 16 15:30 inst
drwxr-xr-x  6 applptch oinstall 4096 Jul 16 15:41 .
drwxr-x— 13 applptch oinstall 4096 Jul 16 16:16 FMW_Home
applptch@aburpaofinm01 $> rm -rf FMW_HOME_BAK
# Check that the Oracle Homes are not in the OraInventory
# Or you will receive the error:
# Below Oracle Homes are already registered in the global inventory:
#/u01/fs2/FMW_Home/Oracle_EBS-app1
#/u01/fs2/FMW_Home/webtier
#/u01/fs2/FMW_Home/oracle_common
#
# Exiting Cloning…
#
# Ensure that the above Oracle Homes are unregistered from the global inventory “/u01/oraInventory/ContentsXML/inventory.xml” and re-run adcfgclone.pl script
vi /u01/oraInventory/ContentsXML/inventory.xml
# and remove the <HOME> through </HOME> entries for the 3 listed entries
# Begin the clone – needs the APPS password
applptch@aburpaofinm01 $> cd $COMMON_TOP/clone/bin
applptch@aburpaofinm01 $> perl adcfgclone.pl appsTier

---

The “long” process – by manually manipulating the admin account in the /u01/<fs#>/FMW_Home:

# Verify your Weblogic Environment home
applptch@aburpaofinm01 $> set | grep EBS

EBS_DOMAIN_HOME=/u01/fs2/FMW_Home/user_projects/domains/EBS_domain_PTCHARMK

# Stop Weblogic
$ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PW
Weblogic Password: [anything because you don’t know it]

# Set your environment
cd $EBS_DOMAIN_HOME/bin
. setDomainEnv.sh

# Remove the existing LDIFT’s, boot.properties and LDAP directory
rm -rf $EBS_DOMAIN_HOME/security/DefaultAuthenticatorInit.ldift ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/ldap

# Start AdminServer from command line manually
cd $EBS_DOMAIN_HOME ;\
startWebLogic.sh

# Enter new credentials as above
<Sep 16, 2014 4:58:09 PM PDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:

Change NodeManager password

Log in to the WebLogic Administration console.
Click the Lock & Edit button.
In the left panel, click on the EBS Domain link.
Select the Security tab.
Click on the ‘Advanced’ link.
Edit the ‘Node Manager password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Edit the ‘Confirm NodeManager Password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Save and activate the changes.

Shut down AdminServer from the console. For the first time, AdminServer has to be stopped from the Admin console. Follow these steps:

Log in to the WebLogic Administration console.
Shut down AdminServer.

Shut down NodeManager using the normal AD script.

$ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

Copy the boot.properties file for all managed servers.

cp $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties \
$EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties

The EBS WebLogic Server domain password has now been changed, and all servers can now be started using the normal AD scripts.

To start AdminServer:

$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

# Start it all back up again
$ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PW
Weblogic Password: [weblogic pwd]

Now your password change is complete (or at least you can get back into your Admin Console).

# Clone the working result from the run filesystem to the patch filesystem
$ adop phase=fs_clone

** Based upon a regular WebLogic post by:

http://shivabizint.wordpress.com/2012/10/15/how-to-reset-the-lost-weblogic-admin-user-password/

**And an R12.2 version by:

http://nagendrachillale.wordpress.com/2014/08/22/if-the-admin-password-of-an-ebs-weblogic-domain-is-lost-or-forgotten/

#OOW14 – @Oracle Enterprise Management for Applications SIG Mtg – 9/28 MW3003 3:30P-4:15P

http://em4apps.oaug.org/News.php

OOW14 Sunday Schedule 28-SEP-2014

We’ll be at OpenWorld 2014 in San Francisco on User Group Sunday, September 28 (Session Time Mostone West – 3003; 15:30 – 16:15): SIG9177 – OAUG Enterprise Manager for Applications SIG Meeting

Click Here to Add to Outlook/iCalendar

LinkedIn Discussion Forum created by consensus of the attendees:

https://www.linkedin.com/groups/Oracle-Enterprise-Manager-Applications-OEM4APPS-6780481

R12.2 Upgrade – Registering New Application Modules – r122_new_custsch.ksh

Note: This script is based upon an OEL 5.9 environment with a completed first-stage R12.2 e-Business Suite upgrade from 12.1.3. Using a pre-existing registered module (ZFA – Oracle Financial Analyzer in our case), this clones the settings required to fulfill the R12.2 upgrade pre-requisite check for additonal applications, which had been added since the 12.1.3 release.) It is referenced in the primary 12.2.3 Upgrade Post and fully listed herein.

cat /mnt/nfs/config/r122_new_custsch.ksh

#!/bin/ksh
# Create new Custom Schema registration files
# Use r122_new_custsch.ksh <APP> <APPID> <APPL_NAME>
# e.g. r122_new_custsch.ksh ABC 123 My_New_App
if ! [[ ${#} = 3 ]]; then
echo -e “\n Use ${0} <APP> <APPID> <APPL_NAME>\n”
echo -e “e.g. r122_new_custsch.ksh ABC 123 My_New_App\n”
exit
fi
if [[ $RUN_BASE = ” ]]; then
echo -e “\n RUN_BASE not set! Cannot continue. Exiting…\n”
exit
fi
echo -e “\n RUN_BASE: ” $RUN_BASE
cd $RUN_BASE/EBSapps/appl/admin/
OLDAPPL_TOP=/ptcharmk/apps/apps_st/appl
NEWAPPL=${1}
NEWAPPL_LC=`echo ${NEWAPPL} | tr ‘[A-Z]’ ‘[a-z]’`
NEWAPPLID=${2}
NEWAPPL_NAME=${3}
echo -e “\nCreating new ${NEWAPPL_LC}prod.txt and ${NEWAPPL_LC}terr.txt”
echo -e “for Application ${NEWAPPL} which is APPLICATION_ID = ${NEWAPPLID}\n”
cp zfaprod.txt ${NEWAPPL_LC}prod.txt
cp zfaterr.txt ${NEWAPPL_LC}terr.txt
echo -e “Creating driver directories…\n”
mkdir -p $RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/admin/driver
mkdir -p $RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/admin/sql
mkdir -p $RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/sql
echo -e “Copying dummy driver files…\n”
cp $OLDAPPL_TOP/xxad/12.0.0/admin/driver/xxadfile.drv \
$RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/admin/driver/${NEWAPPL_LC}file.drv
cp $OLDAPPL_TOP/xxad/12.0.0/sql/XXADNLINS.sql \
$RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/sql/${NEWAPPL}NLINS.sql
cp $OLDAPPL_TOP/xxad/12.0.0/admin/sql/XXADNLADD.sql \
$RUN_BASE/EBSapps/appl/${NEWAPPL_LC}/12.0.0/admin/sql/${NEWAPPL}NLADD.sql
echo -e “Replacing key values…\n”
perl -p -i.bak -e “s:zfa:${NEWAPPL_LC}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}prod.txt
perl -p -i.bak -e “s:ZFA:${NEWAPPL}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}prod.txt
perl -p -i.bak -e “s:505:${NEWAPPLID}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}prod.txt
perl -p -i.bak -e “s:zfa:${NEWAPPL_LC}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}terr.txt
perl -p -i.bak -e “s:ZFA:${NEWAPPL}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}terr.txt
perl -p -i.bak -e “s:505:${NEWAPPLID}:g” $RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}terr.txt
perl -p -i.bak -e “s:Oracle_Financial_Analyzer:${NEWAPPL_NAME}:g” \
$RUN_BASE/EBSapps/appl/admin/${NEWAPPL_LC}terr.txt
echo -e “\nFinished creating new ${NEWAPPL_LC}prod.txt and ${NEWAPPL_LC}terr.txt\n”
# echo -e “Contents ${NEWAPPL_LC}prod.txt:\n”
# cat ${NEWAPPL_LC}prod.txt
# echo -e “Contents ${NEWAPPL_LC}terr.txt:\n”
# cat ${NEWAPPL_LC}terr.txt

Top 10 Signs You’ve Been Working with Oracle Technology and Larry Ellison Too Long

–Note: Mr. Ellison is CEO of Oracle Corporation, the world’s foremost database, middleware and data appliance vendor, long-known for ability to facilitate deployment of massive applications which need unheard of expertise to ensure proper operation and stability (e.g. healthcare.gov)

10) You’re thinking about buying a Ferrari. But you need 15 of them. In black carbon fiber. And are hiring NZ Kiwis to drive them for you.

9) You’ve asked your local real estate broker to price the 6 houses on either side of yours to see how much it would cost to own your entire neighborhood.

8) You’ve just applied 100 sequential Windows Updates (each mentioning something non-specifically horrible will happen to Windows if don’t apply them) to your PC, some of which you had to fix and re-write, and rebooted each time in-between and don’t see any problem with that.

7) Your burger and fries order at McDonalds has a problem, but instead of talking to a manager, you want to go online and open a Sev 2.1 Service Request, with immediate cell phone call-back requested.

6) The new iPhone App you just purchased comes delivered on 100 DVD-ROM‘s, or downloadable digital archive images, and again, you don’t see an issue with that.

5) Your car stereo only has one song playing on infinite repeat: AC/DC’s Back in Black.

4) For some reason every article of clothing you wear has your company logo on it (including your undies), and you see that as completely normal.

3) That new food processor you bought for US$2000 keeps having bizarre breakdowns, but you’re pretty sure it will be fixed in next month’s firmware upgrade.

2) You seriously consider purchasing Ni’ihau (in the Hawaiian islands)

1) Your 13 year-old daughter helped her school create a complex summer school registration system, but you just smirk and snicker that she chose to host it using Microsoft SQLServer, and mutter about read-consistency and parallel block reads.

12c) You incessantly fixate on figuring out how that family potato farmer in western North Dakota could increase his crop yield, if he only used a database.

Healthcare.gov and You – An @IOUG IT Project Perspective

http://www.ioug.org/p/bl/ar/blogaid=147

How Lucky Do You Feel today to be working on lesser-known web projects instead of healthcare.gov?

(photo By Charles Dharapak, AP) Would you care to be in Mr. Jeffery Zients shoes as the latest consultant brought in to fix a website application that has garnered the attention of world-wide press, and has the public chattering about a mis-fired IT project implementation? — http://hosted.ap.org/dynamic/stories/U/US_HEALTH_OVERHAUL_PROBLEMS – New boss for fixing the balky health care website

(ed – nevermind that spelling and grammar checks appear to be missing from the AP News articles in light of getting the story out expediently…)

How about being in the place of Quality Software Services Inc.(QSSI) owned by a unit of UnitedHealth Group (http://www.qssinc.com/industryPartners.html), with the original role as lead web application designer, now being directed to become the equivalent of a Construction Manager, to orchestrate the other subcontractors towards a viable and scalable solution?

A big data back-end powered under Metropolis (http://www.palantir.com/platforms/#metropolis) which is a Hadoop extension set. Thought this was far away from your own sphere of knowledge?  Nope – it’s Oracle-based, in a lot of areas.  Notice the Platinum Partner status on QSSI’s page?

I’m quite sure that all of this will be closely watched by the IT public community at-large as a primary example of watching our federal government, normally not known for subscribing to AGILE and RAD practices in its projects, suddenly become responsible for a monster of a custom web application with a zero-defects requirement due-date of 12/15/2013 .

All of you have been part of post-Go-Live (10/2/2013) projects wherein the defect lists soar, and post-implementation patches, data changes, and no-bounce needed updates get immediately Sev 1 prioritized. But how many of you have had to deal with the public skewering your work through dozens of daily news articles and political representatives posturing to ensure “the person to be blamed will be fired?” —   http://bigstory.ap.org/article/insurers-others-say-obamacare-glitches-fixable

How about having your User Experience testing shot at by the general public, when your UAT instance was never allowed to see the light-of-day by end-users (public) prior to Go-Live? — http://bigstory.ap.org/article/questionable-design-blamed-health-website-woes

Of course, back a month earlier were some buried stories wherein the major software and service contractors were already expressing guarded caution for what would be a “rocky launch” based upon compressed delivery schedules, and mitigated test plans foreshortened to tighten the Go-Live date.  And even though there were plans for alternative sign-ups through call-in centers, in-person, and by-mail application, the one that got 100% of the public’s attention was: the website (http://www.healthcare.gov) — http://bigstory.ap.org/article/obamacare-contractors-project-confidence

So, how lucky do you feel not being a software consultant like QSSI, or CGI Federal (http://www.cgi.com/en/us-federal/services-solutions), now being brought in front of congressional committees to explain why a lack of appropriate IT project management by the end-user customer (the Office of Management and Budget) caused two otherwise operationally-compliant application systems (one a e-commerce website, and the other a commodity trading center/cafeteria application) to be unable to scale to demand when combined together for the first time, without adequate regression testing (plus a pushed Go-Live date, of course)?

How would you solve this project debacle – you’re in IT, and an Oracle expert aren’t you?

Related articles

• Obamacare Website to Be Fixed by December, Zients Says (bloomberg.com)
• UPDATE 2-White House says ‘Obamacare’ website will be fixed by end of … – Reuters (in.reuters.com)
• Healthcare.gov fixer’s one month estimate spurs new analysis of site’s problems (medcitynews.com)
• Healthcare.gov contractors point fingers back at Obama administration in House testimony (hotair.com)
• Obama admin. taps contractor to fix health website (thenewstribune.com)
• New boss for fixing the balky health care website (kansascity.com)