@Oracle R12 EBS HTTP Server Error Child Could Not Open Sslmutex Lockfile

HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).

mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/10.1.3.1/Apache/Apache/logs/ssl_mutex.10549 (System error follows)
System: No such file or directory (errno: 2)

And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).

applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: ENV_ID.myhostname
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status 
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   16788 | Alive  
OC4JGroup:default_group          | OC4J:forms         |   16709 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |   21689 | Alive  
HTTP_Server                      | HTTP_Server        |   16494 | Stop

OEM will show the HTTP service as being down, even though the pages are probably still responding:

R12 EBS OEM AMS monitor HTTP Service down screenshot
R12 EBS OEM AMS monitor HTTP Service down screenshot

Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.

These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration.  These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:

# ###############################################################
#
# This file is automatically generated by AutoConfig.  It will be read and
# overwritten.  If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
)

## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop

#  Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *

ssl.conf:SSLMutex  file:/ptcharmk/inst/apps/${ENV_ID}/logs/ora/10.1.3/Apache/ssl_mutex

# Append a line to the existing custom template
echo -e “\nAcceptMutex pthread\n” >> $FND_TOP/admin/template/custom_conf_1013.tmp

# Modify the existing SSL template (doesn’t seem to have a custom include on this one)
vi $FND_TOP/admin/template/ssl_conf_1013.tmp
=============

Comment:  # SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
Insert:     SSLMutex sem

# Execute AutoConfig

$SCRIPT_TOP/adautocfg.sh -appspass=$APPS_PW

# Re-Start Apache
$SCRIPT_TOP/adapcctl.sh start

# After Change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep -i utex $INST_TOP/ora/10.1.3/Apache/Apache/conf/*

custom.conf:AcceptMutex pthread
ssl.conf:# SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
ssl.conf:SSLMutex  sem

# Confirm semaphore mode is activated

applmgr@myhostname $> grep -i mutex $INST_TOP/logs/ora/10.1.3/Apache/*
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 11:58:14 2015] [notice] Accept mutex: pthread (Default: fcntl)
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 12:12:12 2015] [notice] Accept mutex: pthread (Default: fcntl)

As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s