java.lang.NoClassDefFoundError: javax/servlet/jsp/el/ELException

Recently James gave an rm command on the erpapp directory which has appl, comn, 8.0.6 and iAS directories, thinking that it was a blank directory.  He realised after about 30 seconds and cancelled the command.  However many files had been deleted.  He told me, and based on the errors which were appearing in the logs, I checked $OA_JAVA and found that $OA_JAVA/oracle/apps/fnd directory was missing.  I copied it from another instance on the same techstack level.  However this error was still coming in jvm logs:

java.lang.NoClassDefFoundError: javax/servlet/jsp/el/ELException

        at oracle.cabo.ui.xml.UINodeUtils.createDefaultManager(Unknown Source)

        at oracle.adf.mds.internal.parse.ParserUtils.createDefaultManager(ParserUtils.java:218)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:279)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)

        at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)

        at oracle.adf.mds.internal.MetadataManagerBase.findElement(MetadataManagerBase.java:1244)

        at oracle.adf.mds.MElement.findElement(MElement.java:97)

        at oracle.apps.fnd.framework.webui.JRAD2AKMapper.getRootMElement(JRAD2AKMapper.java:503)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getWebBeanTypeDataFromJRAD(OAWebBeanFactoryImpl.java:3719)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getRootApplicationModuleClass(OAWebBeanFactoryImpl.java:3452)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:937)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:508)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:429)

        at _oa__html._OA._jspService(_OA.java:84)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

        at org.apache.jserv.JServConnection.run(JServConnection.java:294)

        at java.lang.Thread.run(Thread.java:534)

After copying a few more class files which were present in $OA_JAVA, the error changed:

java.lang.NoClassDefFoundError: org/apache/commons/el/ExpressionEvaluatorImpl

        at java.lang.ClassLoader.defineClass0(Native Method)

        at java.lang.ClassLoader.defineClass(ClassLoader.java:539)

        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:123)

        at java.net.URLClassLoader.defineClass(URLClassLoader.java:251)

        at java.net.URLClassLoader.access$100(URLClassLoader.java:55)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:194)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:187)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:289)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:235)

        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302)

        at oracle.cabo.ui.xml.UINodeUtils.createDefaultManager(Unknown Source)

        at oracle.adf.mds.internal.parse.ParserUtils.createDefaultManager(ParserUtils.java:218)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:279)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)

        at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)

        at oracle.adf.mds.internal.MetadataManagerBase.findElement(MetadataManagerBase.java:1244)

        at oracle.adf.mds.MElement.findElement(MElement.java:97)

        at oracle.apps.fnd.framework.webui.JRAD2AKMapper.getRootMElement(JRAD2AKMapper.java:503)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getWebBeanTypeDataFromJRAD(OAWebBeanFactoryImpl.java:3719)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getRootApplicationModuleClass(OAWebBeanFactoryImpl.java:3452)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:938)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:509)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:430)

        at _oa__html._OA._jspService(_OA.java:84)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

        at org.apache.jserv.JServConnection.run(JServConnection.java:294)

        at java.lang.Thread.run(Thread.java:534)

Finally I copied the whole $OA_JAVA directory from another instance.  Once that was done, the errors stopped appearing.  You may face a similar issue if your clone omits some files in $OA_JAVA.

How to create a test .ps postscript file on Solaris

Rich asked me how to create a postscript file.

To create a test postscript file on Solaris you can use 

# /usr/bin/mp
# /usr/openwin/bin/mp
# /usr/lib/lp/postscript/postprint

These can provide a test PostScript file with which you can test out the printer/queue configuration.

vi test.txt
This is a test.
:wq

/usr/bin/mp test.txt >test.ps

You can then print this .ps file to test your printer.

A good article on basics of printing in Solaris is available here.

Concurrent Processing Architecture

The current Concurrent Processing architecture with Global Service Management consists of the following processes and communication model, where each process is responsible for performing a specific set of routines and communicating with parent and dependent processes.

Internal Concurrent Manager (FNDLIBR process) - Communicates with the Service
Manager.

The Internal Concurrent Manager (ICM) starts, sets the number of active processes, monitors, and terminates all other concurrent processes through requests made to the Service Manager, including restarting any failed processes. The ICM also starts and stops, and restarts the Service Manager for each node. The ICM will perform process migration during an instance or node failure. The ICM will be active on a single node. This is also true in a PCP environment, where the ICM will be active on at least one node at all times.

Service Manager (FNDSM process) - Communicates with the Internal Concurrent Manager,
Concurrent Manager, and non-Manager Service processes.

The Service Manager (SM) spawns, and terminates manager and service processes (these could be Forms, or Apache Listeners, Metrics or Reports Server, and any other process controlled through Generic Service Management). When the ICM terminates the SM that resides on the same node with the ICM will also terminate. The SM is ‘chained’ to the ICM. The SM will only reinitialize after termination when there is a function it needs to perform (start, or stop a process), so there may be periods of time when the SM is not active, and this would be normal. All processes initialized by the SM inherit the same environment as the SM. The SM’s environment is set by APPSORA.env file, and the gsmstart.sh script. The TWO_TASK used by the SM to connect to a RAC instance must match the instance_name from GV$INSTANCE. The apps_ listener must be active on each CP node to support the SM connection to the local instance. There should be a Service Manager active on each node where a Concurrent or non-Manager service process will reside.

Internal Monitor (FNDIMON process) - Communicates with the Internal Concurrent Manager.

The Internal Monitor (IM) monitors the Internal Concurrent Manager, and restarts any failed ICM on the local node. During a node failure in a PCP environment the IM will restart the ICM on a surviving node (multiple ICM's may be started on multiple nodes, but only the first ICM started will eventually remain active, all others will gracefully terminate). There should be an Internal Monitor defined on each node where the ICM may migrate.

Standard Manager (FNDLIBR process) - Communicates with the Service Manager and any
client application process.

The Standard Manager is a worker process, that initiates, and executes client requests on behalf of Applications batch, and OLTP clients.

Transaction Manager - Communicates with the Service Manager, and any user process initiated on behalf of a Forms, or Standard Manager request. See Note 240818.1 regarding Transaction Manager communication and setup requirements for RAC.

Parallel Concurrent Processing (PCP) is activated along with Generic Service Management(GSM); it can not be activated independent of GSM. With parallel concurrent processing implemented with GSM, the Internal Concurrent Manager (ICM) tries to
assign valid nodes for concurrent managers and other service instances. Primary and secondary nodes need not be explicitly assigned. However, you can assign primary and secondary nodes for directed load and failover capabilities.

Note: In previous releases, you must have assigned a primary and secondary node to each concurrent manager.

Internal Concurrent Manager:
The Internal Concurrent Manager can run on any node, and can activate an deactivate concurrent managers on all nodes. Since the Internal Concurrent Manager must be
active at all times, it needs high fault tolerance. To provide this fault tolerance, parallel concurrent processing uses Internal Monitor Processes

Internal Monitor Processes:
The sole job of an Internal Monitor Process is to monitor the Internal Concurrent Manager and to restart that manager should it fail. The first Internal Monitor Process to detect that the Internal Concurrent Manager has failed restarts that manager on its own node.

Only one Internal Monitor Process can be active on a single node. You decide which nodes have an Internal Monitor Process when you configure your system. You can also assign each Internal Monitor Process a primary and a secondary node to ensure failover protection.

Internal Monitor Processes, like concurrent managers, have assigned work shifts, and are activated and deactivated by the Internal Concurrent Manager.

However, automatic activation of PCP does not additionally require that primary nodes be assigned for all concurrent managers and other GSM-managed services. If no primary node is assigned for a service instance, the Internal Concurrent Manager(ICM) assigns a valid concurrent processing server node as the target node. In general, this node will be the same node where the Internal Concurrent Manager is running. In the case where the ICM is not on a concurrent processing server node, the ICM chooses an active concurrent processing server node in the system. If no concurrent processing server node is available, no target node will be assigned. Note that if a concurrent manager does have an assigned primary node, it will only try to start up on that node; if the primary node is down, it will look for its assigned secondary node, if one exists. If both the primary and secondary nodes are unavailable, the concurrent manager will not start (the ICM will not look for another node on which to start the concurrent manager). This strategy prevents overloading any node in the case of failover.

The concurrent managers are aware of many aspects of the system state when they start
up. When an ICM successfully starts up it checks the TNS listeners and database instances on all remote nodes and if an instance is down, the affected managers and services switch to their secondary nodes. Processes managed under GSM will only start on nodes that are in Online mode. If a node is changed from Online to Offline, the processes on that node will be shut down and switch to a secondary node if possible.

Concurrent processing provides database instance-sensitive failover capabilities. When an instance is down, all managers connecting to it switch to a secondary middle-tier node. However, if you prefer to handle instance failover separately from such middle-tier failover (for example, using TNS connection-time failover mechanism instead), use the profile option Concurrent:PCP Instance Check. When this profile option is set to OFF, Parallel Concurrent Processing will not provide database instance failover support; however, it will continue to provide middle-tier node failover support when a node goes down.

For the Internal Concurrent Manager you assign the primary node only.


Node Names:
The Concurrent Managers start up and call to the UNIX executable to determine the node name of the machine. This call is 'uname -a' and returns information about the operating system and the name of the node.

Each and every request that runs, takes this node name, time, absolute path to the output file and other information and puts an entry into the FND_CONCURRENT_REQUESTS table with the request ID.

This node name can be an alias but it should be something that the domain name server will resolve.

This node name value can be changed by running the UNIX command setuname or the hostname command. An example of this would be: setuname -n newnodename

Once you change the node name, be sure to bring down the concurrent managers (if they are not already) and bring them back up. Go to the Concurrent Manager Administer screen to see the new node name on the row with the Internal Manager information.

Metalink Notes: 241370.1, 602899.1

INFO: Exception thrown from action: copyGroupFromJar

Bimal reported this error in runInstaller, while upgrading to 10.2.0.4:

INFO: 2/24/09 11:52:42 PM EST: Starting install Install Phase 1 of component Oracle Notification Service Patch
INFO: Calling Action fileActions10.2.0.3.0 copyGroupFromJar
selectedNodes = null
copyGroup = filegroup6
permissions = null
owner = null
group = null
copyAsText = null
JarLoc = /tmp/OraInstall2009-02-24_11-44-27PM/temp831
gpEntries = [[ons.jar ->%s_patchprodLocation%/lib/ons.jar 48492 plats=1=>[453] langs=1=>[en,fr,ar,bn,pt_BR,bg,fr_CA,ca,hr,cs,da,nl,ar_EG,en_GB,et,fi,de,
el,iw,hu,is,in,it,ja,ko,es,lv,lt,ms,es_MX,no,pl,pt,ro,ru,zh_CN,sk,sl,es_ES,sv,th,zh_TW,tr,uk,vi]]
]
DllGroup = false

INFO: Exception thrown from action: copyGroupFromJar
Exception Name: IOException2
Exception String: Error in writing to file '/production/oracle/10.2.0/opmn/lib/ons.jar'. [/production/oracle/10.2.0/opmn/lib/ons.jar (No such file or directory)]
Exception Severity: 2

A search of oraInventory/Contents revealed :

cd $ORACLE_HOME/oraInventory/Contents
strings * |grep production

#/production/oracle/10.2.0/oracle_prodsq
/production/oracle/10.2.0sq
0/production/oracle/10.2.0/oracle_javavm_server_cmpsq
//production/oracle/10.2.0/oracle_javavm_companionsq
,/production/oracle/10.2.0/oracle.javavm.clientsq
+/production/oracle/10.2.0/oracle_ldap_rsf_cmpsq
/production/oracle/10.2.0sq
/production/oracle/10.2.0sq
%/production/oracle/10.2.0/oracle.has.dbsq
#/production/oracle/10.2.0/oracle_prodsq


This should be actually showing

/development/oracle/10.2.0

I checked the adcfgclone.pl logs in $ORACLE_HOME/appsutil/log/$CONTEXT_NAME directory but did not find any errors.

I have asked the DBAs to re-run adcfgclone.pl. I'll update this post as I learn more.

Unix Error Codes

Error Codes are PORT specific, meaning that it is specific to each unix flavor.

The header file /usr/include/sys/errno.h has the error codes on any given machine

EPERM          1 No permission match
ENOENT       2 No such file or directory
ESRCH           3 No such process
EINTR           4 Interrupted system call
EIO                5 I/O error
ENXIO          6 No such device or address
E2BIG           7 Arg list too long
ENOEXEC    8 Exec format error
EBADF          9 Bad file number
ECHILD      10 No child processes
ENOMEM   12 Not enough space
EACCES      13 Permission denied
EFAULT     14 Bad address
EEXIST       17 File exists
EXDEV        18 Cross-device link
ENODEV     19 No such device
ENOTDIR   20 Not a directory
EISDIR        21 Is a directory
EINVAL      22 Invalid argument
EMFILE      24 Too many open files in a process
ETXTBSY   26 Text file busy
EFBIG         27 File too large
ESPIPE        29 Illegal seek
EMLINK      31 Too many links
EPIPE          32 Broken pipe

-- From here on Error vary WIDELY between machines

EDOM                33 Argument out of domain
ERANGE            34 Result too large
EIDRM               36 Identifier removed
EL2NSYNC        38 Level 2 not synchronized
EL3HLT             39 Level 3 halted
EL3RST             40 Level 3 reset
EL2HLT             44 Level 2 halted
EXFULL            52 Message tables full
ENOANO           53 Anode table overflow
EBADRQC         54 Bad request code
EBADSLT          55 Invalid slot
EDEADLOCK    56 File locking deadlock
EBFONT            57 Bad font file format
ENOSTR            60 Not a stream device
ENODATA         61 No data available
ETIME               62 Timer expired
ENOPKG           65 Package not installed
EREMOTE        66 Object is remote
EADV                 68 Advertise error
ESRMNT           69 Srmount error
EPROTO            71 Protocol error
EMULTIHOP    74 Multihop attempted
EBADMSG         77 Not a data message
ELIBACC           83 Can not access a needed shared lib.
ELIBBAD           84 Accessing a corrupted shared lib.
ELIBEXEC         87 Can not exec a shared library directly

AppsLocalLogin.jsp page takes forever to appear, browser shows hourglass

After upgrading to JDK 1.6.0_11, on both of our Extranet application tiers, we could not reach the AppsLocalLogin.jsp page. On invoking /oa_servlets/AppsLogin or /OA_HTML/AppsLocalLogin.jsp, the browser would show an hourglass with the status bar message "Waiting for extranet.justanexample.com". There were no errors in apache or jserv logs. The browser was showing hourglass or was waiting forever. Load balancer or network issues were ruled out as we were able to reach the RapidInstall page by commenting out url_fw.conf in httpd.conf.

After trying a lot of things, I recalled a similar problem in 2005, when 11.5.10 was newly released. In jserv.properties, we set the value of wrapper.bin.parameters=-DLONG_RUNNING_JVM=false. This resolved the issue.

The context file variable s_long_running_jvm controls the value of -DLONG_RUNNING_JVM in jserv.properties. If you want this to be permanent, then change the value of s_long_running_jvm in your context file.

I dug up the TAR from my old notes and got this:

Q1) What is the affect of setting

a)wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
           OR
b)wrapper.bin.parameters=-DCACHEMODE=LOCAL in jserv.properties?

Q2) Why did this resolve NoClassDefFound OR Internal Server Error ?  Is this the permanent solution?
Solution
A1) Distributed Caching was introduced in Framework 11.5.10, refer

From Note 275879.1 Oracle Applications Java Caching Framework Developer's Guide Release 11i (11.5.10)

The setting of the Distributed Mode option is optional. This is the default configuration of the caching framework. The LONG_RUNNING_JVM=true is set and it ensures that caching framework runs in distributed mode. For backward compatibility the same can be ensured by setting -DCACHEMODE=DISTRIBUTED. The above settings determines whether the updates and invalidation's to the objects in the Component Cache are distributed across other JVMs. This allows Component Caches where the data updates need to be seen instantaneously across the JVMs. If the flag is not checked, then updates to the data in the same JVM are seen right away, but updates are not be seen in other JVMs until the "Time to Live" or "Idle Time" expires, if the other JVMs happen to cache the same object. Distributed mode has network overhead and there are some additional steps required to enable this mode for the JVM.

Although turning off Distributed Caching in 11.5.10 will defeat Cache Invalidation, there have been known issues when trying to implement Distributed Caching in E-Business Suite. Until properly patched, Distributed Caching has caused problems with performance and/or rampant NoClassDefFound errors and/or other strange behaviors.

In summary

Cache Invalidation is a feature where some of the middle tier Java Caches for critical reference data are kept in sync when the reference information is changed. This feature was developed for Function/Data Security and Profile Option values. In 11.5.10 the Distributed Caching feature is used to propagate the changes to all "long running" JVMs. If one turns off Distributed Caching, then reference data changes will not be propagated out and one will be forced to bounce the middle tiers to pick up any changes, as done previously.
A2) The caching was out of sync between the web-server that were being load balanced, and with the additional network overhead, started to see apj12 ( Apache/Jserv connection protocol) errors. Once the first server started hanging, this then propagated to the next server, and the next server, faster and faster until all the servers eventually hung. By forcing the cache to reside locally, the network bandwidth was reduced, as well as the issue of trying to keep all the caches in sync with one another.

This is not the solution but a Test to confirm if the issue is occurring because of Java Object Caching.
Distributed Caching issues tend to be very sporadic across the different customers, as it tends to propagate itself based on different product patches that try and take advantage of the distributed caching, or implement new long running queries.

If this is an issue with JOC then best suggestion is to make sure the latest JOC patches ( or their super-seeded patch ) has been applied:
Patch 5639951 REHOST ORACLE JAVA OBJECT CACHE (FORMERLY OCS4J) FOR ORACLE APPLICATIONS 11i (Present in Foxtrot)
Patch 5455628 CACHE DIAGNOSTIC ARU ON TOP OF 11.5.10.3RUP (Not present in Foxtrot)
Patch 6047864 REHOST JOC FIXES (BASED ON JOC 10.1.2.2) FOR APPS 11i  (Not present in Foxtrot)
Also make sure that profile option Self Service personal Home Page mode set to Framework Only As this was written to be used with the new Framework Code and not with the old OSSWA code that is being phased out.
Note: Disabling Distributed Java Caching

1.  Because of the ATG_PF.H Rollup 3 (RUP 3) Patch 4334965, turning off the Java Object Cache (JOC) by setting
    "-DLONG_RUNNING_JVM=false" still calls underlying Apps JOC code.

To disable Java Cache completely follow these steps:

Edit the IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

a) Set  wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

b) Add the line  wrapper.bin.parameters=-DCACHEMODE=LOCAL

c) Restart Apache for these changes to take effect

If the customer problem stop reproducing after these steps, then the issue is certainly related to JOC 

2. Alternatively, if on an ATG RUP4 Patch 4676589 environment to confirm whether its a JOC related issue or not:

Completely REMOVE the system properties -DLONG_RUNNING_JVM and -DCACHEMODE from jserv.properties (either comment out these two system properties or remove them completely but do NOT set them to false and LOCAL respectively) This allows the cache to run in JOC local mode (not in JOC distributed mode) If the problem goes away then it is a JOC distributed caching issue