Tag Archives: Server

@Oracle R12 EBS HTTP Server Error Child Could Not Open Sslmutex Lockfile

HTTP Server Error “Child Could Not Open Sslmutex Lockfile” (Doc ID 562624.1) addresses a non-critical, but annoying issue (because it tends to fill up the $INST_TOP/logs/ora/10.1.3/Apache directory with big error logs every day – every mouse click on a OA Framework page generates an error entry).

mod_ossl: Child could not open SSLMutex lockfile /apps/oracle/oas/10.1.3.1/Apache/Apache/logs/ssl_mutex.10549 (System error follows)
System: No such file or directory (errno: 2)

And sort of randomly, your HTTP service will show “Stop” on the adapcctl.sh status screen. But it’s probably still actually running (the processes still show up when ps -ef | grep httpd – and the OA Framework still seems to be responding).

applmgr@myhostname$> $SCRIPT_TOP/adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: ENV_ID.myhostname
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status 
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   16788 | Alive  
OC4JGroup:default_group          | OC4J:forms         |   16709 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |   21689 | Alive  
HTTP_Server                      | HTTP_Server        |   16494 | Stop

OEM will show the HTTP service as being down, even though the pages are probably still responding:

R12 EBS OEM AMS monitor HTTP Service down screenshot
R12 EBS OEM AMS monitor HTTP Service down screenshot

Implementation for e-Business Suite R12 is relatively easy (just modify 2 configuration files and bounce Apache), but making sure the changes stick after AutoConfig runs means you do need to learn about the configuration templates in $FND_TOP/admin/templates.

These are the skeletal files used during AutoConfig (adautocfg.sh) that regenerate all those files used to control the middleware and database software configuration.  These are the files you that contain warning headers written into each and every one of them warning you not to make direct changes to the configuration file itself like:

# ###############################################################
#
# This file is automatically generated by AutoConfig.  It will be read and
# overwritten.  If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#
# ###############################################################
)

## Execute on every R12 MT server running HTTP/web services – Apache downtime required during change.
# Stop Apache
$SCRIPT_TOP/adapcctl.sh stop

#  Before change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep mutex *

ssl.conf:SSLMutex  file:/ptcharmk/inst/apps/${ENV_ID}/logs/ora/10.1.3/Apache/ssl_mutex

# Append a line to the existing custom template
echo -e “\nAcceptMutex pthread\n” >> $FND_TOP/admin/template/custom_conf_1013.tmp

# Modify the existing SSL template (doesn’t seem to have a custom include on this one)
vi $FND_TOP/admin/template/ssl_conf_1013.tmp
=============

Comment:  # SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
Insert:     SSLMutex sem

# Execute AutoConfig

$SCRIPT_TOP/adautocfg.sh -appspass=$APPS_PW

# Re-Start Apache
$SCRIPT_TOP/adapcctl.sh start

# After Change
applmgr@myhostname $> cd $INST_TOP/ora/10.1.3/Apache/Apache/conf
applmgr@myhostname $> grep -i utex $INST_TOP/ora/10.1.3/Apache/Apache/conf/*

custom.conf:AcceptMutex pthread
ssl.conf:# SSLMutex  file:${INST_TOP}/logs/ora/10.1.3/Apache/ssl_mutex
ssl.conf:SSLMutex  sem

# Confirm semaphore mode is activated

applmgr@myhostname $> grep -i mutex $INST_TOP/logs/ora/10.1.3/Apache/*
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 11:58:14 2015] [notice] Accept mutex: pthread (Default: fcntl)
$INST_TOP/logs/ora/10.1.3/Apache/error_log.1437091200:[Fri Jul 17 12:12:12 2015] [notice] Accept mutex: pthread (Default: fcntl)

As mentioned in the MOS document (Doc ID 562624.1), you may need to use ipcs -a to view hanging semaphore processes after making this change, and use ipcrm -s to kill the hanging processes if Apache refuses to startup after making this change and a different issue causes Apache’s processes to hang.

Advertisement

R12.2.3 / R12.2.x Oracle e-Business Suite – Resetting the WebLogic Password

…because occasionally you either forget it (or in my case, a technology patch reset it to something I wasn’t sure about.)wls_denied_ss

There are 2 basic routes:


The “short” process – re-run the post-clone config (adcfgclone.pl):

# Have to remove the existing FMW home because it will re-unzip a new one from the clone staging archive
applptch@aburpaofinm01 $> cd /u01/fs2
applptch@aburpaofinm01 $> ls
EBSapps  FMW_Home  inst
applptch@aburpaofinm01 $> ls -latr
total 24
drwxrwxr-x  8 applptch oinstall 4096 May  8 21:49 ..
drwxr-xr-x  5 applptch oinstall 4096 May  9 12:33 EBSapps
drwxr-xr-x  3 applptch oinstall 4096 Jul 16 15:30 inst
drwxr-xr-x  6 applptch oinstall 4096 Jul 16 15:41 .
drwxr-x— 13 applptch oinstall 4096 Jul 16 16:16 FMW_Home
applptch@aburpaofinm01 $> rm -rf FMW_HOME_BAK
# Check that the Oracle Homes are not in the OraInventory
# Or you will receive the error:
# Below Oracle Homes are already registered in the global inventory:
#/u01/fs2/FMW_Home/Oracle_EBS-app1
#/u01/fs2/FMW_Home/webtier
#/u01/fs2/FMW_Home/oracle_common
#
# Exiting Cloning…
#
# Ensure that the above Oracle Homes are unregistered from the global inventory “/u01/oraInventory/ContentsXML/inventory.xml” and re-run adcfgclone.pl script
vi /u01/oraInventory/ContentsXML/inventory.xml
# and remove the <HOME> through </HOME> entries for the 3 listed entries
# Begin the clone – needs the APPS password
applptch@aburpaofinm01 $> cd $COMMON_TOP/clone/bin
applptch@aburpaofinm01 $> perl adcfgclone.pl appsTier


The “long” process – by manually manipulating the admin account in the /u01/<fs#>/FMW_Home:

# Verify your Weblogic Environment home
applptch@aburpaofinm01 $> set | grep EBS

EBS_DOMAIN_HOME=/u01/fs2/FMW_Home/user_projects/domains/EBS_domain_PTCHARMK

# Stop Weblogic
$ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PW
Weblogic Password: [anything because you don’t know it]

# Set your environment
cd $EBS_DOMAIN_HOME/bin
. setDomainEnv.sh

# Remove the existing LDIFT’s, boot.properties and LDAP directory
rm -rf $EBS_DOMAIN_HOME/security/DefaultAuthenticatorInit.ldift ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties ; \
rm -rf $EBS_DOMAIN_HOME/servers/AdminServer/data/ldap

# Start AdminServer from command line manually
cd $EBS_DOMAIN_HOME ;\
startWebLogic.sh

# Enter new credentials as above
<Sep 16, 2014 4:58:09 PM PDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:

Change NodeManager password

Log in to the WebLogic Administration console.
Click the Lock & Edit button.
In the left panel, click on the EBS Domain link.
Select the Security tab.
Click on the ‘Advanced’ link.
Edit the ‘Node Manager password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Edit the ‘Confirm NodeManager Password’ field and set it to the new WebLogic Server password. The password should be same as set in Step 3.
Save and activate the changes.

Shut down AdminServer from the console. For the first time, AdminServer has to be stopped from the Admin console. Follow these steps:

Log in to the WebLogic Administration console.
Shut down AdminServer.

Shut down NodeManager using the normal AD script.

$ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

Copy the boot.properties file for all managed servers.

cp $EBS_DOMAIN_HOME/servers/AdminServer/data/nodemanager/boot.properties \
$EBS_DOMAIN_HOME/servers/oacore_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oafm_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/forms-c4ws_server?/security/boot.properties  \
$EBS_DOMAIN_HOME/servers/oaea_server?/security/boot.properties

The EBS WebLogic Server domain password has now been changed, and all servers can now be started using the normal AD scripts.

To start AdminServer:

$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

# Start it all back up again
$ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PW
Weblogic Password: [weblogic pwd]

Now your password change is complete (or at least you can get back into your Admin Console).

# Clone the working result from the run filesystem to the patch filesystem
$ adop phase=fs_clone

** Based upon a regular WebLogic post by:

http://shivabizint.wordpress.com/2012/10/15/how-to-reset-the-lost-weblogic-admin-user-password/

**And an R12.2 version by:

http://nagendrachillale.wordpress.com/2014/08/22/if-the-admin-password-of-an-ebs-weblogic-domain-is-lost-or-forgotten/