Forbidden HTTP-403 in Apache access_log

This morning, I got a call from the DBAs. A new Apache/Forms node was added to an instance. During sanity checks, whenever this node was accessed through this navigation:

System Administration -> PL/SQL ping

the browser showed HTTP-403 and the apache access log showed:

192.168.7.81 - - [20/Jul/2008:01:50:53 -0400] "GET /pls/apps11i/OracleSSWA.Execu
te?E=%7B!38FC0AD8B864E929E00AF2CBA3052B695A3C266C98FB2A33&P= HTTP/1.1" 403 241

192.168.7.81 - - [20/Jul/2008:02:44:18 -0400] "GET /pls/apps11i/fnd_icx_launch.l
aunch?resp_app=FND&resp_key=PS_EOPS_DBA&secgrp_key=STANDARD&start_func=FND_FNDRS
RUN&other_params= HTTP/1.1" 403 244

I checked /etc/resolv.conf and found that the domain was different from the other nodes:

cat /etc/resolv.conf
domain prod.justanexample.com

In other nodes it was cat /etc/resolv.conf
domain justanexample.com

What this means is that whenver the server is trying to identify itself it identifies itself with the new domain name instead of the old one. After adding the line

Allow apps11i.prod.justanexample.com

in httpd_pls.conf and bouncing Apache, the PL/SQL ping succeeded and the error HTTP-403 did not appear.

I asked the DBAs to add the line
Allow apps11i.prod.justanexample.com

in these four .conf files in $IAS_ORACLE_HOME/Apache/Apache/conf/ :

httpd_pls.conf
apps.conf
oprocmgr.conf
trusted.conf

In shared APPL_TOP instances:

cd $IAS_CONFIG_HOME/Apache/Apache/conf

grep apps11i.prod.justanexample.com *.conf  

The above grep command should return these 4 files:

httpd_pls.conf
apps.conf
oprocmgr.conf
trusted.conf

Once this is in place, if you bounce apache and forms, it will start working.

AppsLocallogin shows blank page or HTTP-403

An instance which had a new virtual name was recently cloned. Whenever the E-Business Home Page link (which points to http://www.justanexample.com/OA_HTML/AppsLocallogin.jsp) was clicked it showed up a blank page or HTTP-403 (If you go to Internet Explorer -> Tools Menu -> Advanced -> uncheck Show friendly HTTP error messages).

mod_jserv.log showed a lot of errors like this one:

..] [error] OPM:Can not find one alive process

These errors indicate that Jserv is not up-and-running. This is confirmed by the following entry in the error_log which indicates that oprocmgr is not able to find a active Jserv JVM.

If you check the Apache access log :

grep 403 $IAS_ORACLE_HOME/Apache/Apache/access_log

It would return a lot of lines similar to this one:

physicalservername.com - - [12/Sep/2007:17:25:55 -0400] "GET /oprocmgr-service?cmd=Register&index=0&modName=JServ&grpName=DiscoGroup&port=17140 HTTP/1.1" 403 226

This means that oprocmgr-service doesn't trust physicalservername.com

I checked the "s_oacore_trusted_oproc_nodes" variable in xml file and it pointed to the new virtual name only. I added the physicalservername.com also

"s_oacore_trusted_oproc_nodes=justanexample.com,physicalservername.com

and ran autoconfig. The problem was solved. If you do not want to run autoconfig, the file to modify is $IAS_ORACLE_HOME/Apache/Apache/conf/oprocmgr.conf

======================================
Port 1234

Order Deny,Allow
Deny from all
Allow from localhost
Allow from justanexample
Allow from justanexample.com
Allow from physicalservername.com -- Added this line
======================================

This problem and its solution is well documented in metalink note 372096.1.

You should add this line in these files present in $IAS_ORACLE_HOME/Apache/Apache/conf/

trusted.conf
apps.conf
oprocmgr.conf
httpd_pls.conf

Allow from physicalservername.com