Apache Lock and Mutex files

We had issues in one of our Production Apache. The root cause is that the apache lock files are present on NFS mount:

[Sun Jan 11 03:44:37 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:38 2009] [alert] Child 26287 returned a Fatal error...
Apache is exiting!
[Sun Jan 11 03:44:39 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [emerg] (5)I/O error: fcntl: F_SETLKW: Error freeing accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [error] OPM:hc:Bad chunk-size value in response body
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:08 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:47 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 18:20:40 2009] [warn] pid file /erp11i/erpapp/comn/conf/erp11i_app11i/iAS/Apache/Apache/logs/erp11i_erp11i
overwritten -- Unclean shutdown of previous Apache run?
[Sun Jan 11 18:20:40 2009] [notice] FastCGI: process manager initialized (pid 9972)
[Sun Jan 11 18:20:41 2009] [notice] Oracle HTTP Server Powered by Apache/1.3.19 configured -- resuming normal operations

It is Oracle and Apache’s recommendation that we put Apache Lock and Mutex files on local disk instead of NFS mount (Metalink Note 233428.1):

Apache Lock and OPM Mutex Files
Oracle HTTP Server and the Oracle Process Manager processes create temporary lock files for their internal operations. The location of these lock files are specified in httpd.conf by the LockFile and OpmMtxFile respectively. You must ensure that value of the autoconfig variable s_lock_pid_dir is set to a location on the local file system to avoid file locking issues on the network file system.

http://httpd.apache.org/docs/1.3/mod/core.html#lockfile
LockFile directive
Syntax: LockFile file-path
Default: LockFile logs/accept.lock
Context: server config
Status: core
The LockFile directive sets the path to the lockfile used when Apache is compiled with either USE_FCNTL_SERIALIZED_ACCEPT or USE_FLOCK_SERIALIZED_ACCEPT. This directive should normally be left at its default value. The main reason for changing it is if the logs directory is NFS mounted, since the lockfile must be stored on a local disk. The PID of the main server process is automatically appended to the filename.
SECURITY: It is best to avoid putting this file in a world writable directory such as /var/tmp because someone could create a denial of service attack and prevent the server from starting by creating a lockfile with the same name as the one the server will try to create.
Apache works fine on systems that mount disks via NFS. However there are some files which Apache uses which should not be stored on NFS mounts. The most important is the "lock file" which is used by Apache to efficiently let multiple processes access the same network socket. This default location for this file is the "logs" directory under the server root, unless changed by the "LockFile" (autoconfig variable lock_pid_dir) directive. If the server root or the logs directory is NFS mounted, the location of the lock file must be changed. A directory such as /tmp or /var/tmp is often a good location. This lock file should not be NFS mounted because many implementations of NFS do not lock files properly.

For achieving this, we need to do the following:
1. Make certain that the following Autoconfig variables point to a "local disk" (/export/home/applmgr). If one has multiple application servers then this must be checked and changed on each application tier.
s_lock_pid_dir
s_web_pid_file
2. Run Autoconfig after the change
3. Restart your middle tier services
References: Metalink Notes: 560853.1, 605596.1, 738612.1, 732857.1