Apache Lock and Mutex files

We had issues in one of our Production Apache. The root cause is that the apache lock files are present on NFS mount:

[Sun Jan 11 03:44:37 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:38 2009] [alert] Child 26287 returned a Fatal error...
Apache is exiting!
[Sun Jan 11 03:44:39 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [emerg] (5)I/O error: fcntl: F_SETLKW: Error freeing accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [error] OPM:hc:Bad chunk-size value in response body
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:08 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:47 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 18:20:40 2009] [warn] pid file /erp11i/erpapp/comn/conf/erp11i_app11i/iAS/Apache/Apache/logs/erp11i_erp11i
overwritten -- Unclean shutdown of previous Apache run?
[Sun Jan 11 18:20:40 2009] [notice] FastCGI: process manager initialized (pid 9972)
[Sun Jan 11 18:20:41 2009] [notice] Oracle HTTP Server Powered by Apache/1.3.19 configured -- resuming normal operations

It is Oracle and Apache’s recommendation that we put Apache Lock and Mutex files on local disk instead of NFS mount (Metalink Note 233428.1):

Apache Lock and OPM Mutex Files
Oracle HTTP Server and the Oracle Process Manager processes create temporary lock files for their internal operations. The location of these lock files are specified in httpd.conf by the LockFile and OpmMtxFile respectively. You must ensure that value of the autoconfig variable s_lock_pid_dir is set to a location on the local file system to avoid file locking issues on the network file system.

http://httpd.apache.org/docs/1.3/mod/core.html#lockfile
LockFile directive
Syntax: LockFile file-path
Default: LockFile logs/accept.lock
Context: server config
Status: core
The LockFile directive sets the path to the lockfile used when Apache is compiled with either USE_FCNTL_SERIALIZED_ACCEPT or USE_FLOCK_SERIALIZED_ACCEPT. This directive should normally be left at its default value. The main reason for changing it is if the logs directory is NFS mounted, since the lockfile must be stored on a local disk. The PID of the main server process is automatically appended to the filename.
SECURITY: It is best to avoid putting this file in a world writable directory such as /var/tmp because someone could create a denial of service attack and prevent the server from starting by creating a lockfile with the same name as the one the server will try to create.
Apache works fine on systems that mount disks via NFS. However there are some files which Apache uses which should not be stored on NFS mounts. The most important is the "lock file" which is used by Apache to efficiently let multiple processes access the same network socket. This default location for this file is the "logs" directory under the server root, unless changed by the "LockFile" (autoconfig variable lock_pid_dir) directive. If the server root or the logs directory is NFS mounted, the location of the lock file must be changed. A directory such as /tmp or /var/tmp is often a good location. This lock file should not be NFS mounted because many implementations of NFS do not lock files properly.

For achieving this, we need to do the following:
1. Make certain that the following Autoconfig variables point to a "local disk" (/export/home/applmgr). If one has multiple application servers then this must be checked and changed on each application tier.
s_lock_pid_dir
s_web_pid_file
2. Run Autoconfig after the change
3. Restart your middle tier services
References: Metalink Notes: 560853.1, 605596.1, 738612.1, 732857.1

Start Apache on port 80

Oracle HTTP Server will not start on ports below 1024 because on Unix systems only users with superuser authorizations can bind these ports.

Perform the following steps to enable Oracle HTTP Server to run as root on ports below 1024:

Log in as root.

Run the following commands in the middle-tier Oracle home:

cd ORACLE_HOME/Apache/Apache/bin
chown root .apachectl
chmod 6750 .apachectl

oracle.apps.fnd.sso.AppsLogin was not found

In an instance, when I clicked on the Oracle Applications Manager link: http://www.justanexample.com:8000/servlets/weboam/oam/oamLogin

The following error appeared:

HTTP 404 Not Found
The requested URL /oracle.apps.fnd.sso.AppsLogin was not found on this server.

The browser showed this in the site address field:

http://www.justanexample.com:8000/oracle.apps.fnd.sso.AppsLogin?requestUrl=/oa_servlets/weboam/oam/oamLogin&cancelUrl=http://11ioltp.justanexample.com:8069/oracle.apps.fnd.sso.AppsLogin

I compared this with a working instance where the address filed shows this:

http://www.anotherexample.com:8001/OA_HTML/AppsLocalLogin.jsp?requestUrl=
http%3A%2F%2Fwww.anotherexample.com%3A8068%2Foa_servlets%2Fweboam
%2Foam%2FoamLogin&cancelUrl=http%3A%2F%2Fwww.anotherexample.com
%3A8068%2Foa_servlets%2Foracle.apps.fnd.sso.AppsLogin&s2=
498FD27D425A66475CD8CD04DEE6BE5659B6DA334C5F84414D175249D8DFEB4C

Examining the Apache logs revealed this error:

[Thu Feb 7 23:25:54 2008] [error] [client 192.168.10.38] File does not exist: $COMMON_TOP/portal/$CONTEXT_NAME/oracle.apps.fnd.sso.AppsLogin

Why is it not giving the oa_servlets by default. Why is it going to portal ? It should go to http://justanexample.com:8000/oa_servlets/oracle.apps.fnd.sso.AppsLogin, but instead it is going to http://justanexample.com:8000/oracle.apps.fnd.sso.AppsLogin. This is resulting in apps looking for this class in $COMMON_TOP/portal instead of looking for it in $COMMON_TOP/java.

I logged an SR with Oracle for this and they suggested :

1. Please change the profile option "Apps Servlet Agent" at server level to have the same value a
s at Site level, i.e. it should be:
http://justanexample.com:8000/oa_servlets

2. Bounce Apache.

3. Retest the issue and advice the results.

Even without bouncing Apache, it started working. However the error still came when we try to logout of OAM or Self Service:

HTTP-404 Not Found
The requested URL /oracle.apps.fnd.sso.AppsLogout was not found on this server.

Oracle asked me to do this:

ACTION PLAN
============
1. Please review Note 364439.1 - "Tips and Queries for Troubleshooting Advanced Topologies", section "The Profile Options Query".

After I ran this query in the instance where everything was fine and the instance where it was failing, I realized that all the 130 profile values under the profile (D)-Apps Servlet Agent was missing /oa_servlets

Boiling it down to this query:

Instance where this error was occuring:

select profile_option_value
from fnd_profile_option_values
where profile_option_id=3804
and profile_option_value='http://justanexample.com:8000';

PROFILE_OPTION_VALUE
--------------------------------------
http://justanexample.com:8000
http://justanexample.com:8000
http://justanexample.com:8000
.
.
.
34 rows selected

Instance which was showing OAM page ok the result of query was:

PROFILE_OPTION_VALUE
----------------------------------------------------
http://www.anotherexample.com:8001/oa_servlets
http://www.anotherexample.com:8001/oa_servlets
http://www.anotherexample.com:8001/oa_servlets
http://www.anotherexample.com:8001/oa_servlets
http://www.anotherexample.com:8001/oa_servlets

5 rows selected

So the simple solution is to run an update command:

update fnd_profile_option_values
set profile_option_value='http://justanexample.com:8000/oa_servlets'
where profile_option_id=3804
and profile_option_value='http://justanexample.com:8000'

Bounce Apache after this and it works.