oracle.apps.fnd.common.PoolException

Sandhya reported this error to me.  The clone was done through OEM Grid Applications Pack.  AppsLocalLogin.jsp page showed this error:

oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException

A check through jserv.log and mod_jserv.log showed these errors:

[19/03/2009 11:10:23:091 EDT] oracle.apps.mwa = $MWA_TOP
Exception in static block of jtf.cache.appsimpl.AppsCacheLogger. Stack trace is: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException:
Exception creating new Poolable object.
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1509)
at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:362)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(ExtendedProfileStore.java:210)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfile(ExtendedProfileStore.java:169)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getProfile(ExtendedProfileStore.java:148)
at oracle.apps.fnd.common.logging.DebugEventManager.configureUsingDatabaseValues(DebugEventManager.java:1201)
at oracle.apps.fnd.common.logging.DebugEventManager.configureLogging(DebugEventManager.java:1044)
at oracle.apps.fnd.common.logging.DebugEventManager.internalReinit(DebugEventManager.java:1013)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:980)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:967)
at oracle.apps.fnd.common.AppsLog.reInitialize(AppsLog.java:570)
at oracle.apps.fnd.common.AppsContext.initLog(AppsContext.java:987)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:964)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.fnd.common.WebAppsContext.(WebAppsContext.java:1002)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContextWithDBCFile(ArchitectureWrapper.java:143)
at oracle.apps.jtf.cache.ArchitectureWrapper.createDefaultAppsContext(ArchitectureWrapper.java:107)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContext(ArchitectureWrapper.java:96)
at oracle.apps.jtf.cache.appsimpl.AppsCacheLogger.(AppsCacheLogger.java:43)
at oracle.apps.jtf.cache.appsimpl.AppsCacheEnvironment.getCacheLogger(AppsCacheEnvironment.java:67)
at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:711)
at oracle.apps.jtf.cache.CacheManager.(CacheManager.java:378)
at oracle.apps.fnd.cache.Cache.setCacheFullName(Cache.java:228)
at oracle.apps.fnd.cache.Cache.initCache(Cache.java:114)
at oracle.apps.fnd.cache.Cache.(Cache.java:89)
at oracle.apps.fnd.cache.AppsCache.(AppsCache.java:86)
at oracle.apps.fnd.cache.AolCaches.getCache(AolCaches.java:155)
at oracle.apps.fnd.profiles.Profiles.(Profiles.java:241)
at oracle.apps.fnd.profiles.ExtendedProfileStore.init(ExtendedProfileStore.java:498)
at oracle.apps.fnd.profiles.ExtendedProfileStore.(ExtendedProfileStore.java:119)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at oracle.apps.fnd.common.AppsContext.instantiateProfileStore(AppsContext.java:4312)
at oracle.apps.fnd.common.AppsContext.makeProfileStore(AppsContext.java:1171)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:970)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:952)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:958)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.java:175)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.(ApplicationsObjectLibrary.java:79)
at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:339)
at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)
at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)
at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)
at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)
at org.apache.jserv.JServ.start(JServ.java:625)
at org.apache.jserv.JServ.main(JServ.java:234)
Caused by: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
at oracle.apps.fnd.common.Pool.createObject(Pool.java:1321)
at oracle.apps.fnd.common.Pool.borrowObject(Pool.java:1062)
at oracle.apps.fnd.security.DBConnObjPool.borrowObject(DBConnObjPool.java:752)
at oracle.apps.fnd.security.AppsConnectionManager.borrowConnection(AppsConnectionManager.java:256)
at oracle.apps.fnd.common.Context.borrowConnection(Context.java:1951)
at oracle.apps.fnd.common.AppsContext.getPrivateConnectionFinal(AppsContext.java:2617)
at oracle.apps.fnd.common.AppsContext.getPrivateConnection(AppsContext.java:2550)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2408)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2212)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2126)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2139)
at oracle.apps.fnd.common.Context.getJDBCConnection(Context.java:1682)
at oracle.apps.fnd.cache.GenericCacheLoader.load(GenericCacheLoader.java:168)
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1500)
... 51 more
Caused by: oracle.apps.fnd.common.AppsException: java.sql.SQLException: ORA-01017: invalid username/password; logon denied

I checked the Guest user password in dbc file inside $FND_SECURE. It was correct.
I checked the value of profile option GUEST_USER_PASSWORD. It was correct.
I checked whether apps was able to decrypt the guest user password by using the apps password. This failed.

Incidentally all the ad startup and shutdown were also erroring out with these errors:

Cannot complete applications logon. You may have entered an invalid applications password.

I checked whether apps was able to decrypt the guest user password by using the Production apps password. That worked.

So on a hunch I changed the passwords of APPS and APPLSYS back to Production password.

alter user apps identified by prodpass;
alter user applsys identified by prodpass;

It worked. No more errors.

Then I shutdown all services and did FNDCPASS to change APPS password.

I got a lot of errors like this one:

FNDCPASS was not able to decrypt password for SYSADMIN during applsys password change.
FNDCPASS was not able to decrypt password for GUEST during applsys password change.
FNDCPASS was not able to decrypt password for JOHNDOE during applsys password change.

I checked the encryption again and now it was working with new apps password. So I changed the APPS and APPLSYS password to new password manually:

alter user apps identified by clonepass;
alter user applsys identified by clonepass;

Again it worked. No errors. All services are up and no errors were found in sanity checks. I have advised the DBAs to try FNDCPASS once again in future just before they reclone it after 1 month. Metalink Notes 459601.1 and 454299.1 have some pointers for this FNDCPASS error.