Oracle Apps Technology

Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Saturday, September 27, 2014

oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

Yesterday, the users of an EBS R12.2 instance got this error when they logged in:

Error Page
You have Encountered an unexpected error.  Please contact the System Administrator for assistance.

On checking the $EBS_DOMAIN_HOME/servers/oacore_server1/logs/oacore_server1.out, we found this error:



oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full.
       at oracle.ias.cache.CacheHandle.findObject(CacheHandle.java:1680)
       at oracle.ias.cache.CacheHandle.locateObject(CacheHandle.java:1118)
       at oracle.ias.cache.CacheAccess.get(CacheAccess.java:877)
       at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:771)
       at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802)
       at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624)
       at oracle.apps.fnd.cache.AppsCache.get(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Grant.getGrantArray(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenusForGrantee(Authorization.java:829)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenus(Authorization.java:744)
       at oracle.apps.fnd.functionSecurity.Authorization.getFuncSecGrants(Authorization.java:251)
       at oracle.apps.fnd.functionSecurity.Authorization.testMenuTreeFunction(Authorization.java:499)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:254)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:279)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:160)

We tried bouncing services and deleting $EBS_DOMAIN_HOME/servers/oacore_server1/cache.  None of those actions helped.  Things got back to normal only after the Xmx, Xms,and permsize startup parameters for the oacore JVM were changed in weblogic console on Gary's suggestion:

-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

I also changed it in the context file:

Old: s_oacore_jvm_start_options">-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m
New: s_oacore_jvm_start_options">-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

The oacore_server1 and oacore_server2 were bounced after this.  We haven't seen that error ever since.


There is a support.oracle.com article: Receive Intermittent Error You Have Encountered An Unexpected Error. Please Contact Your System Administrator (Doc ID 1519012.1)

Cause

There are user accounts having extremely high numbers of FUN_ADHOC_RECI_XXXXXXX / FUN_ADHOC_INIT_XXXXXXX assigned.

Users have an extremely high number of (ADHOC) ROLES assigned to them, so when these attempt to login this fills the JOC ( Java object cache ) and causes it to run into it's limits resulting in the errors reported. Once bounce is done all is working fine until such an user logs in again.

While working in AGIS and creating and progressing batches, in the workflow there are several ad hoc roles created which remain on the system and do not get end dated or deleted. This can cause performance issues.
Ad Hoc Roles in WF_LOCAL_ROLES starting with FUN_ADHOC_RECI_XXXXXXX ; FUN_ADHOC_INIT_XXXXXXX with no expiration_date.

A. Run the following SQL to verify if there are accounts having extreme numbers of roles assigned

SQL> SELECT user_name, count(*) FROM wf_user_roles WHERE role_name <> user_name GROUP BY user_name ORDER BY 2;


B. Run following for particular user

SQL> SELECT distinct role_name FROM wf_user_roles
WHERE user_name = cp_user_name
or (user_name = (SELECT name FROM wf_local_roles wlr, fnd_user fusr
WHERE fusr.person_party_id = wlr.orig_system_id
AND wlr.orig_system = 'HZ_PARTY'
AND fusr.user_name = cp_user_name
AND rownum < 2))
AND role_name <> user_name;

Note: Replace cp_user_name with name of user having high number of ADHOC roles


Solution

To implement the solution, please execute the following steps:

1. Ensure that you have taken a backup of your system before applying the recommended solution.

2. Follow the steps given in document to purge the WF_LOCAL_ROLES for the AGIS transactions in 'COMPLETE' status.

AGIS: HOW TO DELETE AD HOC ROLES CREATED IN WORKFLOW (Doc ID 1446561.1)

3. If you are satisfied that the issue is resolved, migrate the solution as appropriate to other environments.



We had also logged SR with Oracle where they pointed us to the very same article and also asked us to do the following:


Action Plan
===========

1. How to find out the existing adhoc roles?

select name, start_date, start_date, expiration_date
from wf_local_roles
where orig_system = 'WF_LOCAL_ROLES'
order by name;

2. Define an expiration date for the ad hoc role:

exec WF_DIRECTORY.SetAdHocRoleExpiration (role_name=> >,expiration_date=>sysdate-1);


3. Periodically, purge expired users and roles in order to improve performance.

exec WF_PURGE.Directory(end_date);

This purges all users and roles in the WF_LOCAL_ROLES,WF_LOCAL_USER_ROLES, and WF_USER_ROLE_ASSIGNMENTS tables whose expiration date is less than or
equal to the specified end date and that are not referenced in any notification.

Parameter: end_date Date to purge to.

For more information, please refer to Oracle Workflow API Reference on page 2 – 128.
Use the workflow API's to purge the ad hoc roles:

NOTE:
After end dating the adhoc roles, the expired adhoc roles can also be purged by running the Purge Obsolete Workflow Runtime Data concurrent program. Make sure the "Core Workflow Only" parameter set to N.

Oracle also shared 3 open bugs (unpublished, can be read only Oracle employees) for this issue:

Bug 19025537 : ORACLE.IAS.CACHE.CACHEFULLEXCEPTION: J2EE JOC-017 THE CACHE IS FULL.
Bug 11772304 : JOC INVESTIGATION WITH 12.2
Bug 19582421 : R12.2 THE CACHE IS FULL; EXCEPTION IN OACORE_SERVER1 LOG.


Thursday, August 28, 2014

adopreports utility in R12.2

I discovered a utility in R12.2 when I was looking for the directory of adop :

which adop
$NE_BASE/EBSapps/appl/ad/bin/adop

cd $NE_BASE/EBSapps/appl/ad/bin/
$ ls
adop  adopreports

Curious I executed adopreports
$ adopreports

Enter the APPS username: apps
Enter the APPS Password:



    Online Patching Diagnostic Reports Main Menu
    --------------------------------------------

    1.  Run edition reports
    2.  Patch edition reports
    3.  Other generic reports
    4.  Exit

    Enter your choice [4]: 3




    Other Generic Reports Sub Menu
    ------------------------------

    1.  Editions summary
    2.  Editioned objects summary
    3.  Free space in important tablespaces
    4.  Status of critical AD_ZD objects
    5.  Actual objects in current edition
    6.  Objects dependencies
    7.  Objects dependency tree
    8.  Editioning views column mappings
    9.  Index details for a table
    10.  Inherited objects in the current edition
    11.  All log messages
    12.  Materialized view details
    13.  Database sessions by edition
    14.  Table details (Synonyms, EV, etc.)
    15.  Count and status of DDL execution by phase
    16.  Back to main menu

This is a great utility for R12.2

Monday, August 4, 2014

R12.2 URL Validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists,

Sravan came across this error while using SSO enabled URL for R12.2


URL validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists, Please contact system administrator.


This occurs for any user that is newly created. The user_guid column is not populated. Here is the Solution:

ssoreg_r12_linux erppgai1 DEV appspassword cleanup
ssoreg_r12_linux erppgai1 DEV appspassword dereg

./oamreg_linux erppgai1 DEV appspassword undeployag

./oamreg_linux erppgai1 DEV appspassword deployag

./oamreg_linux erppgai1 DEV appspassword reg

Tuesday, March 11, 2014

R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade



This problem occurs during rapidwiz where it fails to create apache instance and the error is logged in deploytech.log:

from deploytech.log:

ERROR: Failed to create the EBS domain.
Failed to deploy the E-Business Suite domain

Oracle would usually advise you to re-install. However I found a way to do it by tracking the rapidwiz log files to see what all scripts it calls to create it manually:   

Create missing apache instance, not created by rapidwiz

Reference :

. ./EBSapps.env run
cd $FMW_HOME/opmnctl/bin
./opmnctl createinstance -oracleInstance $FMW_HOME/webtier/instances/EBS_web_ebs122_OHS1
-adminhost admin.justanexample.com -adminPort 7002 -adminRegistration OFF

Creating Oracle Instance directories...Done
Recording OPMN ports reservations...Done
Bootstrapping OPMN configuration files...Done
Instantiating opmnctl for direct usage...Done
Skipping instance registration
Command succeeded.

./opmnctl createcomponent -componentType OHS -componentName EBS_web_ebs122
-oracleInstance $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1

Creating empty component directories...Done
Provisioning OHS files for EBS_web_ebs122
  Copying OHS files from ORACLE_HOME to ORACLE_INSTANCE locations
  Customizing httpd.conf
  Adding component's process control to OPMN...Done
Skipping EBS_web_ebs122 component registration.
Command succeeded.


cd $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/bin
./opmnctl registerinstance –adminhost admin.justanexample.com -adminPort 7002

Command requires login to weblogic admin server (alpcmfapq11.comfin.ge.com):
  Username: weblogic
  Password:

Deploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl admin.justanexample.com:7002 -username weblogic -name NonJ2EEManagement -source $FMW_HOME/webtier/opmn/applications/NonJ2EEManagement.ear -nostage -deploy -upload -noexit
Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: deploy completed on Server AdminServer

Done
Registering instance
Command succeeded.

Copy these *.conf files to $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/config/OHS/EBS_web_ebs122 obtained from Vision instance :


admin.conf
apps.conf
custom.conf
httpd.conf
mod_wl_ohs.conf
oracle_apache.conf
oracle_apache_ssl.conf
security2.conf
security2_dmz.conf
ssl.conf
ssl_terminator.conf
trusted.conf
url_fw.conf
url_fw_ws.conf
webgate.conf

Global replace the server name and port in the files above.

Run adconfig to regenerate these files

Instantiate the Oracle HTTP Server configuration files admin.conf, ssl.conf, mod_wl_ohs.conf on Run File System.
Note: Since the templates are being instantiated, all customizations will be lost and have to be restored explicitly.
Perform the instantiation with the following commands (the backslash at the end of each line is being used to divide each command across several screen lines for readability). When prompted for the APPS password, press .

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/ssl_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/mod_wl_ohs_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/admin_conf_FMW.tmp

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-outfile=$INST_TOP/logs/appl/rgf/TXK/test.log

apps.conf generated successfully

mod_wl_ohs.conf generated successfully
 
 

Monday, March 10, 2014

R12.2 Dictionary Corrupted error during adop phase=prepare

Do not follow the document 1576086.1.  I followed it and increased my invalid counts from 1000 to 100,000 and wasted 10 hours to get back to where I started.  Instead follow these instructions:


Reference : Ora-04023 Reported while Accessing Valid Objects (Doc ID 1610514.1)
Dictionary Corrupted:
Details are as follows:
OWBSYS          HAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          WB_RTI_OBJECT_CATALOG                          OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

PUBLIC          WB_OLAP_AW_PRECOMPUTE                          OWBSYS          WB_OLAP_AW_PRECOMPUTE                          TS mismatch: 23-SEP-10 05:17:34 26-NOV-12 18:48:25

OWBSYS          ALL_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          RAB_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19


please contact Oracle Support and request them to open a bug against Oracle Application Install(166), Component Online Patching(OP)
        [ERROR]     Prepare phase completed with errors/warnings. Please check logfiles
        Log file: /erpodpa1/erpapp/fs_ne/EBSapps/log/adop/3/adop_20140306_005850.log

select owner,object_type,object_name
from dba_objects
where object_name in
(
'HAB_RT_EXEC_PROC_RUN_METRICS',
'WB_RTI_OBJECT_CATALOG',
'UAB_RT_EXEC_PROCESS',
'UAB_RT_EXEC_PROCESS_RUNS',
'UAB_RT_EXEC_PROC_RUN_COUNTS',
'UAB_RT_EXEC_PROC_RUN_METRICS',
'WB_OLAP_AW_PRECOMPUTE',
'ALL_RT_INSTALLATIONS',
'RAB_RT_INSTALLATIONS',
'HAB_RT_EXEC_PROCESS',
'HAB_RT_EXEC_PROCESS_RUNS',
'HAB_RT_EXEC_PROC_RUN_COUNTS'
);


Select against a view  fails with following error.

SQL> Select * from . ;

ORA-04023: could not be validated or authorized

This is issue is due to a timestamp inconsistency in the data dictionary. This timestamp inconsistency will lead to the problem when we  populate the relevant cached objects and later try to select from the affected views.

Run following script to find out the objects which are having timestamp discrepencies.

Connect as / as sysdba

SQL> @?/rdbms/admin/utldtchk.sql

This script is available from 11g onwards.

Recompile all the objects manually returned by this script.
Conn owbsys/owbsys
alter view ALL_RT_INSTALLATIONS compile;
alter view RAB_RT_INSTALLATIONS compile;
alter view HAB_RT_EXEC_PROCESS compile;
alter view HAB_RT_EXEC_PROCESS_RUNS compile;
alter view HAB_RT_EXEC_PROC_RUN_COUNTS compile;
alter view HAB_RT_EXEC_PROC_RUN_METRICS compile;
alter view UAB_RT_EXEC_PROCESS compile;
alter view UAB_RT_EXEC_PROCESS_RUNS compile;
alter view UAB_RT_EXEC_PROC_RUN_COUNTS compile;
alter view UAB_RT_EXEC_PROC_RUN_METRICS compile;
alter function WB_OLAP_AW_PRECOMPUTE compile;
drop public synonym WB_OLAP_AW_PRECOMPUTE;
CREATE PUBLIC SYNONYM WB_OLAP_AW_PRECOMPUTE
FOR OWBSYS.WB_OLAP_AW_PRECOMPUTE;

It will remove the timestamp inconsistency in data dictionary and resolve this issue.

Verify by re-running the script, it should return zero rows.

SQL> @?/rdbms/admin/utldtchk.sql

If you still see occurances, bounce the database and rerun the above script.