TEMP tablespace grows continuously

Recently we had the problem of TEMP tablespace growing uncontrollably in a Production instance:

Following WARNING was appearing in DB alert log:

WARNING: Detected too many memory locking problems.
WARNING: Performance degradation may occur.
Tue Mar 24 23:19:49 2009
Incremental checkpoint up to RBA [0x3b2.800e.0], current log tail at RBA [0x3b2.872c.0]
Tue Mar 24 23:34:35 2009
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
Tue Mar 24 23:39:53 2009
Incremental checkpoint up to RBA [0x3b2.9185.0], current log tail at RBA [0x3b2.9511.0]
Tue Mar 24 23:59:58 2009
Incremental checkpoint up to RBA [0x3b2.a21f.0], current log tail at RBA [0x3b2.a57f.0]
Wed Mar 25 00:20:02 2009
Incremental checkpoint up to RBA [0x3b2.c6fc.0], current log tail at RBA [0x3b2.ce49.0]

This is very similar to the issue mentioned in Metalink Note 357765.1

Cause is Incorrect statistics on FIXED objects

Bad execution plan needed much sort-space.
Multiple waitevents on 'direct path write temp'.

Fix: As per Metalink Note 357765.1

SQL> exec dbms_stats.gather_fixed_objects_stats;

You need to run the above command, anytime you change init.ora parameters.

Advanced options like duplex printing or tray selections

I always wondered how to use the advanced options in printers like duplex printing, tray selection etc. from E-Business Suite.

Metalink Note 394130.1 How To Enable Printer Options Such As Duplex Printing Or Tray Selections For Bitmap Concurrent Programs.

Pasta and XML Publisher does not have the ability to enable printer options such as duplex printing or printer tray selection for a bitmap concurrent program.

Oracle has advised contacting Adobe or the printer vendor for a PPD (Postscript Printer Definition) file that supports the desired function or features associated with the make and model of printer being used.

Refer to metalink note 394130.1 for more details.

oracle.apps.fnd.common.PoolException

Sandhya reported this error to me.  The clone was done through OEM Grid Applications Pack.  AppsLocalLogin.jsp page showed this error:

oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException

A check through jserv.log and mod_jserv.log showed these errors:

[19/03/2009 11:10:23:091 EDT] oracle.apps.mwa = $MWA_TOP
Exception in static block of jtf.cache.appsimpl.AppsCacheLogger. Stack trace is: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException:
Exception creating new Poolable object.
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1509)
at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:362)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(ExtendedProfileStore.java:210)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfile(ExtendedProfileStore.java:169)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getProfile(ExtendedProfileStore.java:148)
at oracle.apps.fnd.common.logging.DebugEventManager.configureUsingDatabaseValues(DebugEventManager.java:1201)
at oracle.apps.fnd.common.logging.DebugEventManager.configureLogging(DebugEventManager.java:1044)
at oracle.apps.fnd.common.logging.DebugEventManager.internalReinit(DebugEventManager.java:1013)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:980)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:967)
at oracle.apps.fnd.common.AppsLog.reInitialize(AppsLog.java:570)
at oracle.apps.fnd.common.AppsContext.initLog(AppsContext.java:987)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:964)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.fnd.common.WebAppsContext.(WebAppsContext.java:1002)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContextWithDBCFile(ArchitectureWrapper.java:143)
at oracle.apps.jtf.cache.ArchitectureWrapper.createDefaultAppsContext(ArchitectureWrapper.java:107)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContext(ArchitectureWrapper.java:96)
at oracle.apps.jtf.cache.appsimpl.AppsCacheLogger.(AppsCacheLogger.java:43)
at oracle.apps.jtf.cache.appsimpl.AppsCacheEnvironment.getCacheLogger(AppsCacheEnvironment.java:67)
at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:711)
at oracle.apps.jtf.cache.CacheManager.(CacheManager.java:378)
at oracle.apps.fnd.cache.Cache.setCacheFullName(Cache.java:228)
at oracle.apps.fnd.cache.Cache.initCache(Cache.java:114)
at oracle.apps.fnd.cache.Cache.(Cache.java:89)
at oracle.apps.fnd.cache.AppsCache.(AppsCache.java:86)
at oracle.apps.fnd.cache.AolCaches.getCache(AolCaches.java:155)
at oracle.apps.fnd.profiles.Profiles.(Profiles.java:241)
at oracle.apps.fnd.profiles.ExtendedProfileStore.init(ExtendedProfileStore.java:498)
at oracle.apps.fnd.profiles.ExtendedProfileStore.(ExtendedProfileStore.java:119)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at oracle.apps.fnd.common.AppsContext.instantiateProfileStore(AppsContext.java:4312)
at oracle.apps.fnd.common.AppsContext.makeProfileStore(AppsContext.java:1171)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:970)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:952)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:958)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.java:175)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.(ApplicationsObjectLibrary.java:79)
at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:339)
at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)
at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)
at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)
at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)
at org.apache.jserv.JServ.start(JServ.java:625)
at org.apache.jserv.JServ.main(JServ.java:234)
Caused by: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
at oracle.apps.fnd.common.Pool.createObject(Pool.java:1321)
at oracle.apps.fnd.common.Pool.borrowObject(Pool.java:1062)
at oracle.apps.fnd.security.DBConnObjPool.borrowObject(DBConnObjPool.java:752)
at oracle.apps.fnd.security.AppsConnectionManager.borrowConnection(AppsConnectionManager.java:256)
at oracle.apps.fnd.common.Context.borrowConnection(Context.java:1951)
at oracle.apps.fnd.common.AppsContext.getPrivateConnectionFinal(AppsContext.java:2617)
at oracle.apps.fnd.common.AppsContext.getPrivateConnection(AppsContext.java:2550)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2408)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2212)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2126)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2139)
at oracle.apps.fnd.common.Context.getJDBCConnection(Context.java:1682)
at oracle.apps.fnd.cache.GenericCacheLoader.load(GenericCacheLoader.java:168)
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1500)
... 51 more
Caused by: oracle.apps.fnd.common.AppsException: java.sql.SQLException: ORA-01017: invalid username/password; logon denied

I checked the Guest user password in dbc file inside $FND_SECURE. It was correct.
I checked the value of profile option GUEST_USER_PASSWORD. It was correct.
I checked whether apps was able to decrypt the guest user password by using the apps password. This failed.

Incidentally all the ad startup and shutdown were also erroring out with these errors:

Cannot complete applications logon. You may have entered an invalid applications password.

I checked whether apps was able to decrypt the guest user password by using the Production apps password. That worked.

So on a hunch I changed the passwords of APPS and APPLSYS back to Production password.

alter user apps identified by prodpass;
alter user applsys identified by prodpass;

It worked. No more errors.

Then I shutdown all services and did FNDCPASS to change APPS password.

I got a lot of errors like this one:

FNDCPASS was not able to decrypt password for SYSADMIN during applsys password change.
FNDCPASS was not able to decrypt password for GUEST during applsys password change.
FNDCPASS was not able to decrypt password for JOHNDOE during applsys password change.

I checked the encryption again and now it was working with new apps password. So I changed the APPS and APPLSYS password to new password manually:

alter user apps identified by clonepass;
alter user applsys identified by clonepass;

Again it worked. No errors. All services are up and no errors were found in sanity checks. I have advised the DBAs to try FNDCPASS once again in future just before they reclone it after 1 month. Metalink Notes 459601.1 and 454299.1 have some pointers for this FNDCPASS error.

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

This error was reported after clone:

JSP Error:

---

Request URI:/OA_HTML/AppsLocalLogin.jsp

Exception:

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

---

jserv.log showed these errors:

[19/03/2009 11:29:13:045 EDT] oracle.apps.mwa = $MWA_TOP

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. The

Distributed Caching System failed to initialize on port: 12442. The list of hosts in the distributed caching system is: 11ioltp.justanexample.com . The port 12442 should be free on each host running the JVMs. The default port 12442 can be overridden using -Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=<

port number>

Metalink Note 461872.1 describes this problem.  This occurs when your java object cache port can not be contacted.  s_java_object_cache_port is the context file variable which needs to be changed.  If you would like to avoid running autoconfig, you can change the value of this variable jserv.properties and jserv_restrict.properties.  We changed it from 12442 to 12443:

jserv.properties:wrapper.bin.parameters=-Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=12443

jserv_restrict.properties:wrapper.bin.parameters=-Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=12443

Change the port number.

Bounce Apache

Retry.

It worked for us.

Solaris 10 pre-requisites for Oracle Database 11g

Metalink Note 743042.1 has the Requirements for Installing Oracle 11gR1 RDBMS on Solaris 10 SPARC 64-bit.

The notable among these is kernel patch 137111-04.

There is a post on Sun blogs which describes Solaris 10 kernel patch id sequence.  If you follow the obsolescence path of this patch, the curret patch which is not yet obsolete is 137137-09.  So you should apply 137137-09 or higher to your Solaris 10 OS, before you can consider moving to 11g.

VCS Agent Exit Codes

VCS QuickStart uses processes called Agents to start, stop, and monitor resources. Agentsmonitor all resources continually to validate both online and offline resource status.VCS QS Custom Application agents use the following programs to manage an application

Start Program – a script or program which starts the application

Stop Program – a script or program which gracefully stops the application

Monitor Program – a script or program which monitors the application

Clean Program – a script or program which kills or forcibly stops the application

The VCS QS Monitor Program uses the following exit values to communicate status:

100 – OFFLINE

110 – ONLINE

99 – UNKNOWN