oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

Yesterday, the users of an EBS R12.2 instance got this error when they logged in:

Error Page
You have Encountered an unexpected error.  Please contact the System Administrator for assistance.

On checking the $EBS_DOMAIN_HOME/servers/oacore_server1/logs/oacore_server1.out, we found this error:

oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full.
       at oracle.ias.cache.CacheHandle.findObject(CacheHandle.java:1680)
       at oracle.ias.cache.CacheHandle.locateObject(CacheHandle.java:1118)
       at oracle.ias.cache.CacheAccess.get(CacheAccess.java:877)
       at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:771)
       at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802)
       at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624)
       at oracle.apps.fnd.cache.AppsCache.get(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Grant.getGrantArray(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenusForGrantee(Authorization.java:829)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenus(Authorization.java:744)
       at oracle.apps.fnd.functionSecurity.Authorization.getFuncSecGrants(Authorization.java:251)
       at oracle.apps.fnd.functionSecurity.Authorization.testMenuTreeFunction(Authorization.java:499)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:254)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:279)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:160)

We tried bouncing services and deleting $EBS_DOMAIN_HOME/servers/oacore_server1/cache.  None of those actions helped.  Things got back to normal only after the Xmx, Xms,and permsize startup parameters for the oacore JVM were changed in weblogic console on Gary's suggestion:

-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

I also changed it in the context file:

Old: s_oacore_jvm_start_options">-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m

New: s_oacore_jvm_start_options">-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

The oacore_server1 and oacore_server2 were bounced after this.  We haven't seen that error ever since.

There is a support.oracle.com article: Receive Intermittent Error You Have Encountered An Unexpected Error. Please Contact Your System Administrator (Doc ID 1519012.1)

Cause

There are user accounts having extremely high numbers of FUN_ADHOC_RECI_XXXXXXX / FUN_ADHOC_INIT_XXXXXXX assigned.

Users have an extremely high number of (ADHOC) ROLES assigned to them, so when these attempt to login this fills the JOC ( Java object cache ) and causes it to run into it's limits resulting in the errors reported. Once bounce is done all is working fine until such an user logs in again.

While working in AGIS and creating and progressing batches, in the workflow there are several ad hoc roles created which remain on the system and do not get end dated or deleted. This can cause performance issues.
Ad Hoc Roles in WF_LOCAL_ROLES starting with FUN_ADHOC_RECI_XXXXXXX ; FUN_ADHOC_INIT_XXXXXXX with no expiration_date.

A. Run the following SQL to verify if there are accounts having extreme numbers of roles assigned

SQL> SELECT user_name, count(*) FROM wf_user_roles WHERE role_name <> user_name GROUP BY user_name ORDER BY 2;


B. Run following for particular user

SQL> SELECT distinct role_name FROM wf_user_roles
WHERE user_name = cp_user_name
or (user_name = (SELECT name FROM wf_local_roles wlr, fnd_user fusr
WHERE fusr.person_party_id = wlr.orig_system_id
AND wlr.orig_system = 'HZ_PARTY'
AND fusr.user_name = cp_user_name
AND rownum < 2))
AND role_name <> user_name;

Note: Replace cp_user_name with name of user having high number of ADHOC roles

Solution

To implement the solution, please execute the following steps:

1. Ensure that you have taken a backup of your system before applying the recommended solution.

2. Follow the steps given in document to purge the WF_LOCAL_ROLES for the AGIS transactions in 'COMPLETE' status.

AGIS: HOW TO DELETE AD HOC ROLES CREATED IN WORKFLOW (Doc ID 1446561.1)

3. If you are satisfied that the issue is resolved, migrate the solution as appropriate to other environments.

We had also logged SR with Oracle where they pointed us to the very same article and also asked us to do the following:

Action Plan
===========

1. How to find out the existing adhoc roles?

select name, start_date, start_date, expiration_date
from wf_local_roles
where orig_system = 'WF_LOCAL_ROLES'
order by name;

2. Define an expiration date for the ad hoc role:

exec WF_DIRECTORY.SetAdHocRoleExpiration (role_name=> >,expiration_date=>sysdate-1);


3. Periodically, purge expired users and roles in order to improve performance.

exec WF_PURGE.Directory(end_date);

This purges all users and roles in the WF_LOCAL_ROLES,WF_LOCAL_USER_ROLES, and WF_USER_ROLE_ASSIGNMENTS tables whose expiration date is less than or
equal to the specified end date and that are not referenced in any notification.

Parameter: end_date Date to purge to.

For more information, please refer to Oracle Workflow API Reference on page 2 – 128.
Use the workflow API's to purge the ad hoc roles:

NOTE:
After end dating the adhoc roles, the expired adhoc roles can also be purged by running the Purge Obsolete Workflow Runtime Data concurrent program. Make sure the "Core Workflow Only" parameter set to N.

Oracle also shared 3 open bugs (unpublished, can be read only Oracle employees) for this issue:

Bug 19025537 : ORACLE.IAS.CACHE.CACHEFULLEXCEPTION: J2EE JOC-017 THE CACHE IS FULL.
Bug 11772304 : JOC INVESTIGATION WITH 12.2
Bug 19582421 : R12.2 THE CACHE IS FULL; EXCEPTION IN OACORE_SERVER1 LOG.

Oracle finally shared the contents of Bug 19582421:

Action Plan
==========

Please review the following from Bug 19582421 : R12.2 THE CACHE IS FULL; EXCEPTION IN OACORE_SERVER1 LOG.


Workaround Steps:

1. Extract CacheDefaultConfig.xml from cache.jar


cd $FMW_HOME/oracle_common/modules/oracle.javacache_11.1.1
jar -xf cache.jar CacheDefaultConfig.xml

2. Edit CacheDefaultConfig.xml.
- diskCache size was the parameter that fixed it.
- changed the max-objects as well as per the bug.

Original:

xmlns="http://www.oracle.com/oracle/ias/cache/configuration11"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" max-objects="5000"
max-size="10" private="false" cache-dump-path="jocdump" system="false"
clean-interval="60" version="11.1.1.2.0" internal-version="110000">

init-retry-delay="2000" enable-ssl="false" auto-recover="false">
dedicated-coordinator="false" outOfProc="false">

default-level="SEVERE"/>




Modified:

xmlns="http://www.oracle.com/oracle/ias/cache/configuration11"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" max-objects="100000"
max-size="50" private="false" cache-dump-pa
th="jocdump" system="false" clean-interval="60" version="11.1.1.2.0"
internal-version="110000">

init-retry-delay="2000" enable-ssl="false" auto-recover="false">
dedicated-coordinator="false" outOfProc="false">





default-level="SEVERE"/>




3. Upload the changed file to the jar file

jar uf cache.jar CacheDefaultConfig.xml

4. Modify javacache.xml and make the same changes. This file is probably not
getting used. But made the changes anyway to keep the values in sync.
cd $EBS_DOMAIN_HOME/config/fmwconfig/servers/oacore_server1

5. Bounce apache and oacore.

These 5 steps resolved the issue.