AC-00002: Error: Unable to create log file

Makarand was doing a Fresh R12 install. The install was successful on the Database Node and Primary Application Node. On Primary Application node only 'Batch Processing Services' are enabled.

However while installing on the additonal application node the rapidwiz was giving errors

'AC-00002: Error: Unable to create log file - $INST_TOP/$CONTEXT_NAME/logs/08031631.log
Exception in thread "AWT-EventQueue-0" java.lang.RuntimeException: AC-00002: Unable to create log file -
$INST_TOP/$CONTEXT_NAME/logs/08031631.log

Makarand had tried installing multiple times but every time it failed with the same message

The truss of the rapidwiz process showed this:

/40: open64("$INST_TOP/$CONTEXT_NAME/logs/08200953.log",
O_WRONLY|O_APPEND|O_CREAT, 0666) Err#13 EACCES


Here’s the reason for the failure:

apps11i is the app tier and you are trying to write to an NFS mount. If you access an NFS mount from a server other than its origin, the root user of the application server is treated as nobody user. I verified this by touching a file while logged in as root:

-rw-r--r-- 1 nobody nobody 0 Aug 20 10:59 abc

This happens if the NFS share is shared without root= in the NFS mount command.

An examination of the /etc/dfs/dfstab file revealed that:

share -F nfs -o rw=visionerp.justanexample.com,root=visionerp.justanexample.com /gpshpc76/custom
share -F nfs -o rw=erpr12.justanexample.com,root=erpr12.justanexample.com /erpr12/erpapp
share -F nfs -o rw=erp11i.justanexample.com,root=erp11i.justanexample.com /erp11i/erpapp

If you compare the last line with a line for other instances it clearly shows that the application server has no read write rights and no root equivalence. I corrected this by executing this command:

share -F nfs -o rw=apps11i.justanexample.com,root=apps11i.justanexample.com /arpsysd1/erpapp

I also corrected it in the /etc/dfs/dfstab.

After this we retried the rapidwiz and it worked.

Apache Lock and Mutex files

We had issues in one of our Production Apache. The root cause is that the apache lock files are present on NFS mount:

[Sun Jan 11 03:44:37 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:38 2009] [alert] Child 26287 returned a Fatal error...
Apache is exiting!
[Sun Jan 11 03:44:39 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [emerg] (5)I/O error: fcntl: F_SETLKW: Error freeing accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:44 2009] [error] OPM:hc:Bad chunk-size value in response body
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:56 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:44:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:08 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:47 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 03:45:57 2009] [emerg] (151)Stale NFS file handle: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
[Sun Jan 11 18:20:40 2009] [warn] pid file /erp11i/erpapp/comn/conf/erp11i_app11i/iAS/Apache/Apache/logs/erp11i_erp11i
overwritten -- Unclean shutdown of previous Apache run?
[Sun Jan 11 18:20:40 2009] [notice] FastCGI: process manager initialized (pid 9972)
[Sun Jan 11 18:20:41 2009] [notice] Oracle HTTP Server Powered by Apache/1.3.19 configured -- resuming normal operations

It is Oracle and Apache’s recommendation that we put Apache Lock and Mutex files on local disk instead of NFS mount (Metalink Note 233428.1):

Apache Lock and OPM Mutex Files
Oracle HTTP Server and the Oracle Process Manager processes create temporary lock files for their internal operations. The location of these lock files are specified in httpd.conf by the LockFile and OpmMtxFile respectively. You must ensure that value of the autoconfig variable s_lock_pid_dir is set to a location on the local file system to avoid file locking issues on the network file system.

http://httpd.apache.org/docs/1.3/mod/core.html#lockfile
LockFile directive
Syntax: LockFile file-path
Default: LockFile logs/accept.lock
Context: server config
Status: core
The LockFile directive sets the path to the lockfile used when Apache is compiled with either USE_FCNTL_SERIALIZED_ACCEPT or USE_FLOCK_SERIALIZED_ACCEPT. This directive should normally be left at its default value. The main reason for changing it is if the logs directory is NFS mounted, since the lockfile must be stored on a local disk. The PID of the main server process is automatically appended to the filename.
SECURITY: It is best to avoid putting this file in a world writable directory such as /var/tmp because someone could create a denial of service attack and prevent the server from starting by creating a lockfile with the same name as the one the server will try to create.
Apache works fine on systems that mount disks via NFS. However there are some files which Apache uses which should not be stored on NFS mounts. The most important is the "lock file" which is used by Apache to efficiently let multiple processes access the same network socket. This default location for this file is the "logs" directory under the server root, unless changed by the "LockFile" (autoconfig variable lock_pid_dir) directive. If the server root or the logs directory is NFS mounted, the location of the lock file must be changed. A directory such as /tmp or /var/tmp is often a good location. This lock file should not be NFS mounted because many implementations of NFS do not lock files properly.

For achieving this, we need to do the following:
1. Make certain that the following Autoconfig variables point to a "local disk" (/export/home/applmgr). If one has multiple application servers then this must be checked and changed on each application tier.
s_lock_pid_dir
s_web_pid_file
2. Run Autoconfig after the change
3. Restart your middle tier services
References: Metalink Notes: 560853.1, 605596.1, 738612.1, 732857.1

ERROR: Please check the jri clone stage directory

Sreenivas reported this error while merging application tier file systems:

Executing merge appl top for JRI merge

ERROR: Please check the jri clone stage directory
Number of java directories in clone stage for jri merge are incorrect
adjcopy needs atleast two directories to merge jri files i.e., -masterArchive and -deltaArchives

Bug 3804082 says:

$OA_JAVA/META-INF/JRIMETA.DAT does not exist in clone directory of any server.

However this file was present.

In an OTN forum, Sreenivas found the advise to copy $COMMON_TOP/clone directory from all source nodes.

This time adcfgclone.pl ran fine:

Executing merge appl top for JRI merge

 merge jri files with command :

 JRI log file located at, $APPL_TOP/admin/$TWO_TASK/out/jri_logfile.txt

 JRI report file located at, $APPL_TOP/admin/$TWO_TASK/out/jri_report.txt

 Copying master archive from $COMMON_TOP/clone/bin/../appl/webserver1/java/* to $COMMON_TOP/java

 Completed copying master archive to  $COMMON_TOP/java

root has no powers on NFS drive

In our shared application file system instance, where a SAN mount on database server is NFS mounted on 4 application servers, we were installing Olite.  Olite requires running root.sh script. But the root.sh always gave permission denied whenever we ran it from the application tier. This was happening because the uid and gid of root did not match on the application server and the database server.  Application Server was installed as Solaris 10 where root's gid is 0.  DB server was upgraded from Solaris 8 to Solaris 10.  In Solaris 8 the root user's gid is 1.  After changing the root user's gid to 0, the script ran fine.

NFS does NOT behave the same as a local filesystem, especially for root! NFS is a very non-secure method to share filesystems and unless the NFS server exports the filesystem with a special root option, the root user on the local computer is nobody (literally!). That's why there is a 'nobody' user and it has a user ID of -2. This user has NO special privileges. Th1e reason is fairly obvious. If a server were to blindly export filesystems to any computer on the network (with root=root privileges), then the contents of the filesystem could NEVER be trusted as any machine could trash any file or directory.

So even if you were root on your local box, chown will return EPERM (errno 1) because the NFS filesystem says you are nobody and therefore not the file's owner. /etc/exports on the NFS server can allow root access but one should be very careful to restrict root privileges for the above mentioned reasons.

FRM-407530 ORA-06512 after implementing shared APPL_TOP

Recently an instance was upgraded from 11.5.8 to 11.5.10 and shared APPL_TOP was implemented. After the shared appl_top steps, forms wouldn't launch. Every time a form was accessed it would throw up the FRM-40735 ORA-06512 error. If the forms were manually compiled, they would not compile and throw up errors. After a lot of research, it was found that all the forms were calling a CUSTOM binary which was not copied by the merge process of shared APPL_TOP as it was not a part of core Apps. Once this binary was copied, all the forms could be compiled without issues.