11.5.5 java.lang.RuntimeException: Cannot get Connection to the Database

Sadiq called me for an issue with an 11.5.5 instance in which clicking on Diagnostics or other tabs in CRM HTML Administrator responsibility showed this error:

Exception:

java.lang.RuntimeException: Cannot get Connection to the Database
at oracle.apps.fnd.sso.Utils.getConnection(Utils.java:222)
at _oa__html._fndvald._jspService(_fndvald.java:83)
at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java)
at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java)
at oracle.jsp.JspServlet.doDispatch(JspServlet.java)
at oracle.jsp.JspServlet.internalService(JspServlet.java)
at oracle.jsp.JspServlet.service(JspServlet.java)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)
at org.apache.jserv.JServConnection.processRequest(JServConnection.java:314)
at org.apache.jserv.JServConnection.run(JServConnection.java:188)
at java.lang.Thread.run(Thread.java:595)

aoljtest failed on Verify DBC

Request URI:/OA_HTML/jsp/fnd/aolj_verify_dbc_content.jsp

Exception:
java.lang.NoClassDefFoundError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:168)
at oracle.jsp.app.JspClassLoader.loadClass(JspClassLoader.java)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at _oa__html._jsp._fnd._aolj__verify__dbc__content._jspService(_aolj__verify__dbc__content.java:575)
at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java)
at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java)
at oracle.jsp.JspServlet.doDispatch(JspServlet.java)
at oracle.jsp.JspServlet.internalService(JspServlet.java)
at oracle.jsp.JspServlet.service(JspServlet.java)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)
at org.apache.jserv.JServConnection.processRequest(JServConnection.java:314)
at org.apache.jserv.JServConnection.run(JServConnection.java:188)
at java.lang.Thread.run(Thread.java:595)

http://server:port/OA_HTML/jtflogin.jsp did not show the jtf login page and gave errors.

This is definitely a problem with the DBC file. Sadiq hand built it but the error did not resolve. Guest password is Oracle in dbc file, It is Oracle in the GUEST_USER_PASSWORD profile also.

I have asked him to bounce database based on metalink note 250624.1.

Metalink Note 178437.1 has these steps for jtflogin:

Login: Cannot access jtflogin.jsp nor jtfdefaultlogin.jsp:
You are trying to log into the html stack via jtflogin.jsp or jtfdefaultlogin.jsp.

Jtflogin.jsp does not display the page and returns message 'error in error page'.

Jtfdefaultlogin.jsp also does not display the page and returns message like 'system error' followed by several java messages, the most significant
being 'oracle.apps.jtf.base.session.ServletSessionManagerException'.

You have also tried to run the diagnostics via jtfqalgn.htm and the database connectivity tests fail

Cause #1:
There can be several reasons for this problem scenario, including configuration files not being correct for httpd.conf, jserv.conf,
jserv.properties, and zone.properties.

The following solution is for correcting one specific condition where your .dbc file has been manually edited, looks correct but the
connectivity issue persists.

To eliminate this possibility, please perform the following procedure:

Why do we want to do this procedure ?

Because if you have ever manually edited the .dbc file, you may have unknowingly inserted/deleted
some character/space/line feed/carriage return/tab/etc.

Do not manually edit your .dbc file. Results are unpredictable and can take a long time to diagnose.

Resolution:
Perform the following query:

SQL> select * from fnd_application_servers;

The query should return one (1) row. If the query returns more than one row, delete the row(s) that is/are most apparently incorrect. You will probably be able to recognize the invalid row(s) by the IP address column (server_address) that points to an invalid host database tier.

The authentication string in this table (column server_id) must match the same string in your 'host_sid'.dbc file (APPL_SERVER_ID).

Make a backup copy of your .dbc file.

To insure that everything is in synch, run the adgendbc.sh script located in your $COMMON_TOP\admin\install directory. This script takes no parameters except '-deinstall' which you probably do not want to do. This script will assign a new authentication string (server_id), update your .dbc file and update your fnd_application_servers table.

It's OK if you get an error message related to unique key constraint (read the script file).

Double check the information in your .dbc file and fnd_application_servers table. It should be in synch. Display or print the .dbc file. Please do not use an editor because you may forget the rule and save it.

This is still not solved. I'll update this post once I learn more.

How to find out what webserver is running on remote server

Vivek Gite has a nice article here on how to fingerprint/identify web server.  You can do it from your unix server with curl command:

$ curl -I http://windows.justanexample.com/

HTTP/1.1 403 Forbidden

Content-Length: 218

Content-Type: text/html

Server: Microsoft-IIS/6.0

Date: Tue, 07 Apr 2009 17:26:05 GMT

Here's an excerpt from man entry for curl

DESCRIPTION

curl is a tool to transfer data from or to a  server,  using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or  FILE).   The  command is designed to work without user interaction.

curl offers a busload of useful tricks like  proxy  support,user authentication, ftp upload, HTTP post, SSL connections, cookies, file transfer resume and  more. The amount of features will make your head spin!

I'll update this post with more curl examples.

Redirect URL to login page in IBM Websphere

Jim asked me whether I knew anything about IBM Websphere. He had a problem where he wanted the application's login page to appear whenever the primary URL like http://websphere.justanexample.com is accessed. The login page was http://websphere.justanexample.com/BusinessApp/AppLogin.html

I worked with Sai to learn that IBM Websphere used IBM HTTPServer as the Web Server.
As per http://www-01.ibm.com/software/webservers/httpservers/:

IBM HTTP Server is based on the Apache HTTP Server (httpd.apache.org), developed by the Apache Software Foundation.

So it has the same files like httpd.conf like the Apache in 11i. If we changed the Document Root to point to the login folder, it was causing other issues like images were not being rendered as they referred to relative path.

So we did the next best thing. We created a new file index.html in the DocumentRoot directory which redirected to http://websphere.justanexample.com/BusinessApp/AppLogin.html

Here's the code:

ld: fatal: library -ljava: not found

Akhilesh reported these errors when they were relinking 10g RDBMS ORACLE_HOME:

$ORACLE_HOME/lib32:/opt/ORCLcluster/lib/ -Y P,:/opt/SUNWcluster/lib:/opt/ORCLcluster/lib/:/usr/ccs/lib:/usr/lib -Qy -lc -laio `cat $ORACLE_HOME/lib32/sysliblist` -R$ORACLE_HOME/lib32 -lm
ld: fatal: library -ljava: not found
ld: fatal: library -ljvm: not found
ld: fatal: File processing errors. No output written to $ORACLE_HOME/sysman/lib/libnmemso.so
*** Error code 1
make: Fatal error: Command failed for target `$ORACLE_HOME/sysman/lib/libnmemso.so'
Current working directory $ORACLE_HOME/sysman/lib
*** Error code 1
make: Fatal error: Command failed for target `libnmemso'
chmod 755 $ORACLE_HOME/bin
/usr/ccs/bin/make -f $ORACLE_HOME/sysman/lib/ins_sysman.mk relink_sharedobj SHAREDOBJ=libnmemso
/usr/ccs/bin/ld -o $ORACLE_HOME/sysman/lib/libnmemso.so -dy -M $ORACLE_HOME/sysman/lib32/libnmemso.def -G -z text -L$ORACLE_HOME/lib32 -L$ORACLE_HOME/lib -L$ORACLE_HOME/sysman/lib32/ -R$ORACLE_HOME/lib32/:$ORACLE_HOME/lib/:$ORACLE_HOME/sysman/lib32/ $ORACLE_HOME/sysman/lib32/libnmem.a $ORACLE_HOME/sysman/lib32/libnmer.a $ORACLE_HOME/sysman/lib32/libnmeu.a $ORACLE_HOME/sysman/lib32/libnmt.a $ORACLE_HOME/sysman/lib32/libnml.a $ORACLE_HOME/sysman/lib32/libnmee.a $ORACLE_HOME/sysman/lib32/libnmefmgr.a $ORACLE_HOME/sysman/lib32/libnmevr.a $ORACLE_HOME/sysman/lib32/libnmehu.a $ORACLE_HOME/sysman/lib32/libnmehl.a $ORACLE_HOME/sysman/lib32/libnmepm.a $ORACLE_HOME/sysman/lib32/libnmedc.a $ORACLE_HOME/sysman/lib32/libnmedt.a $ORACLE_HOME/sysman/lib32/libnmedm.a $ORACLE_HOME/sysman/lib32/libnmedr.a $ORACLE_HOME/sysman/lib32/libnmeb.a $ORACLE_HOME/sysman/lib32/libnmec.a $ORACLE_HOME/sysman/lib32/libnmesched.a $ORACLE_HOME/sysman/lib32/libnmectl.a $ORACLE_HOME/sysman/lib32/libnmejc.a $ORACLE_HOME/sysman/lib32/libnmej.a -L/erppgrt1/oracle/10.2.0/jdk/jre/lib/sparc/client -L/erppgrt1/oracle/10.2.0/jdk/jre/lib/sparc -z lazyload -ljava -ljvm -z nolazyload $ORACLE_HOME/sysman/lib32/libnmeclumgr.a $ORACLE_HOME/sysman/lib32/libnmecluster.a $ORACLE_HOME/sysman/lib32/libnmexml.a $ORACLE_HOME/sysman/lib32/libnmo.a -lxml10 -lcore10 -lnls10 -lcore10 -lnls10 -lcore10 -lunls10 -lclntsh -lnnz10 -lthread -lsocket -lnsl -lm -ldl -lkvm -lkstat -lelf -laio -lposix4 -R /opt/SUNWcluster/lib:$ORACLE_HOME/lib32:/opt/ORCLcluster/lib/ -Y P,:/opt/SUNWcluster/lib:/opt/ORCLcluster/lib/:/usr/ccs/lib:/usr/lib -Qy -lc -laio `cat $ORACLE_HOME/lib32/sysliblist` -R$ORACLE_HOME/lib32 -lm
ld: fatal: library -ljava: not found
ld: fatal: library -ljvm: not found
ld: fatal: File processing errors. No output written to $ORACLE_HOME/sysman/lib/libnmemso.so
*** Error code 1
make: Fatal error: Command failed for target `$ORACLE_HOME/sysman/lib/libnmemso.so'
Current working directory $ORACLE_HOME/sysman/lib32
*** Error code 1
make: Fatal error: Command failed for target `libnmemso'

I found metalink note 418557.1 which describes similar issue:

Cause

As the error indicates:

'/usr/bin/ld: cannot find -ljava
collect2: ld returned 1 exit status
Opatch is not able to find libjava.so file in the variable JRE_LIB_DIR in file env_sysman.mk and giving error

'/usr/bin/ld: cannot find -ljava'


Solution

To implement the solution, execute the following steps:

Continue with the patch installation when you receive thie error and complete it.

Search for file libjava.so on the system
For example you find it under the following directory

$ORACLE_HOME/jdk/jre/lib/i386

Backup and edit $ORACLE_HOME/sysman/lib/env_sysman.mk

Change : JRE_LIB_DIR=%JRE_HOME%/bin
To : JRE_LIB_DIR=$(ORACLE_HOME)/jdk/jre/lib/i386

Issue the failed relink command:

% make -f ins_sysman.mk relink_sharedobj SHAREDOBJ=libnmemso

However we did not have i386 directory inside $ORACLE_HOME/jdk/jre/lib, as we are on Solaris. Instead we had sparc and sparcv9 directories. The sparc directory had 32 bit libjava.so and sparcv9 had 64-bit libjava.so. Since our DB is 64-bit, sparcv9 would be the directory.

When we opened the $ORACLE_HOME/sysman/lib/env_sysman.mk file we found that the line was pointing to a directory in gold instance from which the current instance was cloned:

JRE_LIB_DIR=/gold11i/jdk/jre/lib/sparcv9

We changed this to the correct development path

JRE_LIB_DIR=/dev11i/jdk/jre/lib/sparcv9

and relinked again. It worked fine this time without issues.

APP-FND-01630: Cannot open file $APPLTMP/OFAAA5uaODo.t for reading

Recently MSCPLD (ASCP Loader) failed with these errors:

Advanced Supply Chain Planning: Version : 11.5.0

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.

MSCPLD module: Loader Worker With Direct Load Option
+---------------------------------------------------------------------------+

Current system time is 06-APR-2009 20:34:10

+---------------------------------------------------------------------------+


===================================================================
Debug Mode : Disabled
Output to Terminal : No
Argument Method : Database Fetch
Trace Mode : Disabled

===================================================================
Argument 1 (CTRL_FILE) = $APPLCSF/out/data01/mbpoutput/MSLD_RESERVATIONS.ctl
Argument 2 (DATA_FILE) = $APPLCSF/out/data01/mbpoutput/MSLD_RESERVATIONS.dat
Argument 3 (DISCARD_FILE) = $APPLCSF/out/data01/mbpoutput/MSLD_RESERVATIONS.dis
Argument 4 (BAD_FILE) = $APPLCSF/out/data01/mbpoutput/MSLD_RESERVATIONS.bad
Argument 5 (LOG_FILE) =
Argument 6 (NUM_OF_ERRORS) = 0

===================================================================
Creating dummy log file ...
Parent Program Name: MSCMON
This IS part of a Plan run.
usdsop: exec failed during spawnsqlldr: No such file or directory
Program exited with status 1
APP-FND-01630: Cannot open file $APPLTMP/OFAAA5uaODo.t for reading

Cause: USDINS encountered an error when attempting to open file $APPLTMP/OFAAA5uaODo.t for reading.

Action: Verify that the filename is correct and that the environment variables controlling that filename are correct.

Action: If the file is opened in read mode, check that the file exists. Check that you have privileges to read the file in the file directory. Contact your system administrator to obtain read privileges.

Action: If the file is opened in write or append mode, check that you have privileges to create and write files in the file directory. Contact your system administrator to obtain create and write privileges.

We checked permissions, utl_file_dir, profile option utl_file_log and various other settings. Everything was configured correctly.

Then we checked FND_EXECUTABLES for what is being called by MSCPLD

SQL> utables
Name Null? Type
----------------------------------------- -------- ----------------------------
APPLICATION_ID NOT NULL NUMBER(15)
EXECUTABLE_ID NOT NULL NUMBER(15)
EXECUTABLE_NAME NOT NULL VARCHAR2(30)
LAST_UPDATE_DATE NOT NULL DATE
LAST_UPDATED_BY NOT NULL NUMBER(15)
CREATION_DATE NOT NULL DATE
CREATED_BY NOT NULL NUMBER(15)
LAST_UPDATE_LOGIN NOT NULL NUMBER(15)
EXECUTION_METHOD_CODE NOT NULL VARCHAR2(1)
EXECUTION_FILE_NAME VARCHAR2(61)
SUBROUTINE_NAME VARCHAR2(30)
ICON_NAME VARCHAR2(30)
EXECUTION_FILE_PATH VARCHAR2(510)

SQL> select EXECUTABLE_NAME,EXECUTION_FILE_NAME
2 from fnd_executables
3 where EXECUTABLE_NAME='MSCPLD';

EXECUTABLE_NAME
------------------------------
EXECUTION_FILE_NAME
-------------------------------------------------------------
MSCPLD
MSCPLD.sh

When we checked the contents of MSCPLD.sh, it was having these lines:

#!/bin/ksh
#
# $Header: MSCPLD.sh 115.0 2004/02/26 21:05:53 shhan noship $
#
# Shell script to support direct path option in
# sql*loader this sets the oracle_home to server oracle home
#
# make sure following environment variables are setup correct
# and point to the server oracle home
# ORACLE_HOME
# LD_LIBRARY_PATH
# PATH

# please uncomment after setting

export ORACLE_HOME=/prod11i/oracle/10.2.0
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH

$MSC_TOP/bin/MSCSLD $*
~

The ORACLE_HOME was pointing to Production path of 10g DB home.

After we corrected this to /dev11i/oracle/10.2.0, it worked fine.

Forms doesn't launch from self service

Srinivas contacted me about this issue today.  After converting an instance into shared application file system aka shared APPL_TOP, forms were not launching from self service.

I opened a putty session in one window and tailed the apache access log.  On another window I opened Internet Explorer and arranged both side by side.  I logged in successfully. The OA framework page showing responsibilities came up.  I chose System Administrator and clicked on Concurrent Requests.  Internet Explorer stopped there progress bar froze at midway.  Apache access log showed HTTP-403:

192.168.122.10 - - [02/Apr/2009:18:12:03 -0400] "GET /pls/apps11i/fnd_icx_launch.launch?resp_app=SYSADMIN&resp_key=SYSTEM_ADMINISTRATOR&secgrp_key=STANDARD&start_func=FND_FNDCPQCR_SYS&other_params= HTTP/1.1" 403 244 0

I checked the $IAS_CONFIG_HOME/Apache/Apache/conf/httpd_pls.conf file and found these lines:

Allow from apps11i.justanexample.dev.com

This was incorrect.  It should have been:

Allow from apps11i.dev.justanexample.com

Once I corrected this and bounced Apache, no more HTTP-403 errors appeared and forms launched correctly