java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

Two days back, this error was reported to me. This was for an instance which has 4 application tier nodes which means 4 physical servers which run Apache/Forms. There is a BigIP load balancer configured which manages the load between the 4. Some users were getting this error on their browsers when they were trying to access the AppsLogin page which has username and password fields.

This appeared to be a problem in one of the application server's jserv. To isolate this is what I did:

BigIP URL: http://bigip.justanexample.com/oa_servlets/AppsLogin

I changed this first to: http://server1.justanexample.com/oa_servlets/AppsLogin in the browser.
Page appeared.
Then, I changed it to http://server2.justanexample.com/oa_servlets/AppsLogin
Page appeared
Then, I changed it to http://server3.justanexample.com/oa_servlets/AppsLogin
Error appeared: java.lang.NoClassDefFoundError.........
Then, I changed it to http://server4.justanexample.com/oa_servlets/AppsLogin
Page appeared

We checked mod_jserv.log of Server3:

mod_jserv.log:[10/06/2008 20:08:18:616] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles: Could not initialize class oracle.apps.fnd.profiles.Profiles


It was clear that the problem was with Jserv of server3.

So I got Apache bounced on server3.

Even after bounce it didn't work.

We checked for any orphan Apache / Jserv processes and found this one still running after stopping Apache:

applmgr 27925 1 0 May 11 ? 4:20 /apps11i/erp/comn/util/java/1.4/j2sdk1.4.2_04/bin/java -ms128m -mx256m -cla

We killed this process:

kill -9 27925

Started Apache.

Server 3 URL http://server3.justanexample.com/oa_servlets/AppsLogin started working.

User was requested to retest for the issue. They confirmed that the problem was resolved.

Metalink Note 461872.1 describes this problem:

Symptoms

After any configuration change, while trying to login using http://machine.domain:port/OA_HTML/AppsLocalLogin.jsp


ERROR
--------
"Exception: java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles"in mod_jserv.log

Cause

Bug 4466442: APPSLOCALLOGIN FAILS WITH NOCLASSDEFFO
UNDERROR: ORACLE/APPS/FND/PROFILES/PROFILE.
This bug outlines the issue that you encounter: java.lang.NoClassDefFoundError:

Java Cache port is busy in that machine.

Solution

To implement the solution, please execute the following steps:

1. If you have multiple application tiers, check the value of s_java_object_cache_port to be same in all.
2. Check if you any dummy process is occupying port specified by s_java_object_cache_port in Context XML file.
Try to kill the process
(or )

3. Change the avlue of s_java_object_cache_port in Context XML file to unused port number
4. Run autoconfig to reflect the changes in the system

We had followed this note but the s_java_object_cache_port was not occupied by any process. However we resolved the problem by making sure no Apache / Jserv processes were running after Apache shutdown and then restarting Apache through adapcctl.sh start