java.lang.NoSuchMethodError: oracle.jdbc.dbaccess.DBAccess.lobWrite(Loracle/sql/CLOB;J[C)J

Following errors were appearing while accessing the OA Framework home page after keying in username and password

Request URI:/OA_HTML/OA.jsp

Exception:

java.lang.NoSuchMethodError: oracle.jdbc.dbaccess.DBAccess.lobWrite(Loracle/sql/CLOB;J[C)J

at oracle.sql.LobDBAccessImpl.putChars(LobDBAccessImpl.java:221)

at oracle.sql.CLOB.putChars(CLOB.java:339)

at oracle.sql.CLOB.putString(CLOB.java:359)

at oracle.jdbc.driver.OracleClobWriter.flushBuffer(OracleClobWriter.java:261)

at oracle.jdbc.driver.OracleClobWriter.flush(OracleClobWriter.java:204)

at oracle.apps.fnd.framework.webui.OAServerDelegate.handleError(OAServerDelegate.java:1941)

at oracle.apps.fnd.framework.webui.OAPageErrorHandler.prepareException(OAPageErrorHandler.java:1246)

at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:1987)

at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:509)

at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:430)

at _oa__html._OA._jspService(_OA.java:82)

at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:385)

at oracle.jsp.JspServlet.doDispatch(JspServlet.java:259)

at oracle.jsp.JspServlet.internalService(JspServlet.java:178)

at oracle.jsp.JspServlet.service(JspServlet.java:148)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:175)

at _oa__html._OA._jspService(_OA.java:92)

at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:385)

at oracle.jsp.JspServlet.doDispatch(JspServlet.java:259)

at oracle.jsp.JspServlet.internalService(JspServlet.java:178)

at oracle.jsp.JspServlet.service(JspServlet.java:148)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:175)

at _oa__html._OA._jspService(_OA.java:92)

at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:385)

at oracle.jsp.JspServlet.doDispatch(JspServlet.java:259)

at oracle.jsp.JspServlet.internalService(JspServlet.java:178)

at oracle.jsp.JspServlet.service(JspServlet.java:148)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

at org.apache.jserv.JServConnection.run(JServConnection.java:294)

at java.lang.Thread.run(Thread.java:619)

We also got this error:

Request URI:/OA_HTML/AppsLocalLogin.jsp

Exception:

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.common.Guest

at oracle.apps.fnd.functionSecurity.Authorization.testFunction(Authorization.java:317)

at oracle.apps.fnd.functionSecurity.FunctionSecurity.testFunction(FunctionSecurity.java:376)

at oracle.apps.fnd.functionSecurity.FunctionSecurity.testFunction(FunctionSecurity.java:356)

at oracle.apps.fnd.umx.password.PasswordUtil.generateForgotPwdLinkPVT(PasswordUtil.java:135)

at oracle.apps.fnd.umx.password.PasswordUtil.generateForgotPwdUrl(PasswordUtil.java:221)

at _oa__html._AppsLocalLogin._jspService(_AppsLocalLogin.java:864)

at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:385)

at oracle.jsp.JspServlet.doDispatch(JspServlet.java:259)

at oracle.jsp.JspServlet.internalService(JspServlet.java:178)

at oracle.jsp.JspServlet.service(JspServlet.java:148)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

at org.apache.jserv.JServConnection.run(JServConnection.java:294)

at java.lang.Thread.run(Thread.java:619)

Issue was due to the fact that Operating System was unable to get enough contiguous swap space.  After adding swap file of 10GB, it worked fine.

Mouse form focus issue after Dev6i patchset 19

After going live with Dev6i patchset 19, we faced issues in many forms where the flexfield would not pop up if mouse was clicked.  Tab worked fine, but mouse click didn't work.

As per metalink note 753704.1 and 753702.1, it’s a known issue and we need to apply forms patch 7482264 and force regenerate jar files to fix it. We were able to reproduced the issue in a test instance. We applied the patch to the test instance and it resolved the issue.

We are now doing regression testing to ensure that this one off patch has not created new problems as indicated in some other bugs.

java.lang.NoClassDefFoundError: javax/servlet/jsp/el/ELException

Recently James gave an rm command on the erpapp directory which has appl, comn, 8.0.6 and iAS directories, thinking that it was a blank directory.  He realised after about 30 seconds and cancelled the command.  However many files had been deleted.  He told me, and based on the errors which were appearing in the logs, I checked $OA_JAVA and found that $OA_JAVA/oracle/apps/fnd directory was missing.  I copied it from another instance on the same techstack level.  However this error was still coming in jvm logs:

java.lang.NoClassDefFoundError: javax/servlet/jsp/el/ELException

        at oracle.cabo.ui.xml.UINodeUtils.createDefaultManager(Unknown Source)

        at oracle.adf.mds.internal.parse.ParserUtils.createDefaultManager(ParserUtils.java:218)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:279)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)

        at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)

        at oracle.adf.mds.internal.MetadataManagerBase.findElement(MetadataManagerBase.java:1244)

        at oracle.adf.mds.MElement.findElement(MElement.java:97)

        at oracle.apps.fnd.framework.webui.JRAD2AKMapper.getRootMElement(JRAD2AKMapper.java:503)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getWebBeanTypeDataFromJRAD(OAWebBeanFactoryImpl.java:3719)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getRootApplicationModuleClass(OAWebBeanFactoryImpl.java:3452)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:937)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:508)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:429)

        at _oa__html._OA._jspService(_OA.java:84)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

        at org.apache.jserv.JServConnection.run(JServConnection.java:294)

        at java.lang.Thread.run(Thread.java:534)

After copying a few more class files which were present in $OA_JAVA, the error changed:

java.lang.NoClassDefFoundError: org/apache/commons/el/ExpressionEvaluatorImpl

        at java.lang.ClassLoader.defineClass0(Native Method)

        at java.lang.ClassLoader.defineClass(ClassLoader.java:539)

        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:123)

        at java.net.URLClassLoader.defineClass(URLClassLoader.java:251)

        at java.net.URLClassLoader.access$100(URLClassLoader.java:55)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:194)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:187)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:289)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:235)

        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302)

        at oracle.cabo.ui.xml.UINodeUtils.createDefaultManager(Unknown Source)

        at oracle.adf.mds.internal.parse.ParserUtils.createDefaultManager(ParserUtils.java:218)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:279)

        at oracle.adf.mds.internal.parse.ParserUtils.createNode(ParserUtils.java:115)

        at oracle.adf.mds.adapters.DBAdapter.getElementData(DBAdapter.java:324)

        at oracle.adf.mds.internal.MetadataManagerBase.findElement(MetadataManagerBase.java:1244)

        at oracle.adf.mds.MElement.findElement(MElement.java:97)

        at oracle.apps.fnd.framework.webui.JRAD2AKMapper.getRootMElement(JRAD2AKMapper.java:503)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getWebBeanTypeDataFromJRAD(OAWebBeanFactoryImpl.java:3719)

        at oracle.apps.fnd.framework.webui.OAWebBeanFactoryImpl.getRootApplicationModuleClass(OAWebBeanFactoryImpl.java:3452)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:938)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:509)

        at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:430)

        at _oa__html._OA._jspService(_OA.java:84)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at oracle.jsp.provider.Jsp20RequestDispatcher.forward(Jsp20RequestDispatcher.java:162)

        at oracle.jsp.runtime.OraclePageContext.forward(OraclePageContext.java:187)

        at _oa__html._OA._jspService(_OA.java:94)

        at oracle.jsp.runtime.HttpJsp.service(HttpJsp.java:119)

        at oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java:417)

        at oracle.jsp.JspServlet.doDispatch(JspServlet.java:267)

        at oracle.jsp.JspServlet.internalService(JspServlet.java:186)

        at oracle.jsp.JspServlet.service(JspServlet.java:156)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:588)

        at org.apache.jserv.JServConnection.processRequest(JServConnection.java:456)

        at org.apache.jserv.JServConnection.run(JServConnection.java:294)

        at java.lang.Thread.run(Thread.java:534)

Finally I copied the whole $OA_JAVA directory from another instance.  Once that was done, the errors stopped appearing.  You may face a similar issue if your clone omits some files in $OA_JAVA.

How to create a test .ps postscript file on Solaris

Rich asked me how to create a postscript file.

To create a test postscript file on Solaris you can use 

# /usr/bin/mp
# /usr/openwin/bin/mp
# /usr/lib/lp/postscript/postprint

These can provide a test PostScript file with which you can test out the printer/queue configuration.

vi test.txt
This is a test.
:wq

/usr/bin/mp test.txt >test.ps

You can then print this .ps file to test your printer.

A good article on basics of printing in Solaris is available here.

Concurrent Processing Architecture

The current Concurrent Processing architecture with Global Service Management consists of the following processes and communication model, where each process is responsible for performing a specific set of routines and communicating with parent and dependent processes.

Internal Concurrent Manager (FNDLIBR process) - Communicates with the Service
Manager.

The Internal Concurrent Manager (ICM) starts, sets the number of active processes, monitors, and terminates all other concurrent processes through requests made to the Service Manager, including restarting any failed processes. The ICM also starts and stops, and restarts the Service Manager for each node. The ICM will perform process migration during an instance or node failure. The ICM will be active on a single node. This is also true in a PCP environment, where the ICM will be active on at least one node at all times.

Service Manager (FNDSM process) - Communicates with the Internal Concurrent Manager,
Concurrent Manager, and non-Manager Service processes.

The Service Manager (SM) spawns, and terminates manager and service processes (these could be Forms, or Apache Listeners, Metrics or Reports Server, and any other process controlled through Generic Service Management). When the ICM terminates the SM that resides on the same node with the ICM will also terminate. The SM is ‘chained’ to the ICM. The SM will only reinitialize after termination when there is a function it needs to perform (start, or stop a process), so there may be periods of time when the SM is not active, and this would be normal. All processes initialized by the SM inherit the same environment as the SM. The SM’s environment is set by APPSORA.env file, and the gsmstart.sh script. The TWO_TASK used by the SM to connect to a RAC instance must match the instance_name from GV$INSTANCE. The apps_ listener must be active on each CP node to support the SM connection to the local instance. There should be a Service Manager active on each node where a Concurrent or non-Manager service process will reside.

Internal Monitor (FNDIMON process) - Communicates with the Internal Concurrent Manager.

The Internal Monitor (IM) monitors the Internal Concurrent Manager, and restarts any failed ICM on the local node. During a node failure in a PCP environment the IM will restart the ICM on a surviving node (multiple ICM's may be started on multiple nodes, but only the first ICM started will eventually remain active, all others will gracefully terminate). There should be an Internal Monitor defined on each node where the ICM may migrate.

Standard Manager (FNDLIBR process) - Communicates with the Service Manager and any
client application process.

The Standard Manager is a worker process, that initiates, and executes client requests on behalf of Applications batch, and OLTP clients.

Transaction Manager - Communicates with the Service Manager, and any user process initiated on behalf of a Forms, or Standard Manager request. See Note 240818.1 regarding Transaction Manager communication and setup requirements for RAC.

Parallel Concurrent Processing (PCP) is activated along with Generic Service Management(GSM); it can not be activated independent of GSM. With parallel concurrent processing implemented with GSM, the Internal Concurrent Manager (ICM) tries to
assign valid nodes for concurrent managers and other service instances. Primary and secondary nodes need not be explicitly assigned. However, you can assign primary and secondary nodes for directed load and failover capabilities.

Note: In previous releases, you must have assigned a primary and secondary node to each concurrent manager.

Internal Concurrent Manager:
The Internal Concurrent Manager can run on any node, and can activate an deactivate concurrent managers on all nodes. Since the Internal Concurrent Manager must be
active at all times, it needs high fault tolerance. To provide this fault tolerance, parallel concurrent processing uses Internal Monitor Processes

Internal Monitor Processes:
The sole job of an Internal Monitor Process is to monitor the Internal Concurrent Manager and to restart that manager should it fail. The first Internal Monitor Process to detect that the Internal Concurrent Manager has failed restarts that manager on its own node.

Only one Internal Monitor Process can be active on a single node. You decide which nodes have an Internal Monitor Process when you configure your system. You can also assign each Internal Monitor Process a primary and a secondary node to ensure failover protection.

Internal Monitor Processes, like concurrent managers, have assigned work shifts, and are activated and deactivated by the Internal Concurrent Manager.

However, automatic activation of PCP does not additionally require that primary nodes be assigned for all concurrent managers and other GSM-managed services. If no primary node is assigned for a service instance, the Internal Concurrent Manager(ICM) assigns a valid concurrent processing server node as the target node. In general, this node will be the same node where the Internal Concurrent Manager is running. In the case where the ICM is not on a concurrent processing server node, the ICM chooses an active concurrent processing server node in the system. If no concurrent processing server node is available, no target node will be assigned. Note that if a concurrent manager does have an assigned primary node, it will only try to start up on that node; if the primary node is down, it will look for its assigned secondary node, if one exists. If both the primary and secondary nodes are unavailable, the concurrent manager will not start (the ICM will not look for another node on which to start the concurrent manager). This strategy prevents overloading any node in the case of failover.

The concurrent managers are aware of many aspects of the system state when they start
up. When an ICM successfully starts up it checks the TNS listeners and database instances on all remote nodes and if an instance is down, the affected managers and services switch to their secondary nodes. Processes managed under GSM will only start on nodes that are in Online mode. If a node is changed from Online to Offline, the processes on that node will be shut down and switch to a secondary node if possible.

Concurrent processing provides database instance-sensitive failover capabilities. When an instance is down, all managers connecting to it switch to a secondary middle-tier node. However, if you prefer to handle instance failover separately from such middle-tier failover (for example, using TNS connection-time failover mechanism instead), use the profile option Concurrent:PCP Instance Check. When this profile option is set to OFF, Parallel Concurrent Processing will not provide database instance failover support; however, it will continue to provide middle-tier node failover support when a node goes down.

For the Internal Concurrent Manager you assign the primary node only.


Node Names:
The Concurrent Managers start up and call to the UNIX executable to determine the node name of the machine. This call is 'uname -a' and returns information about the operating system and the name of the node.

Each and every request that runs, takes this node name, time, absolute path to the output file and other information and puts an entry into the FND_CONCURRENT_REQUESTS table with the request ID.

This node name can be an alias but it should be something that the domain name server will resolve.

This node name value can be changed by running the UNIX command setuname or the hostname command. An example of this would be: setuname -n newnodename

Once you change the node name, be sure to bring down the concurrent managers (if they are not already) and bring them back up. Go to the Concurrent Manager Administer screen to see the new node name on the row with the Internal Manager information.

Metalink Notes: 241370.1, 602899.1

INFO: Exception thrown from action: copyGroupFromJar

Bimal reported this error in runInstaller, while upgrading to 10.2.0.4:

INFO: 2/24/09 11:52:42 PM EST: Starting install Install Phase 1 of component Oracle Notification Service Patch
INFO: Calling Action fileActions10.2.0.3.0 copyGroupFromJar
selectedNodes = null
copyGroup = filegroup6
permissions = null
owner = null
group = null
copyAsText = null
JarLoc = /tmp/OraInstall2009-02-24_11-44-27PM/temp831
gpEntries = [[ons.jar ->%s_patchprodLocation%/lib/ons.jar 48492 plats=1=>[453] langs=1=>[en,fr,ar,bn,pt_BR,bg,fr_CA,ca,hr,cs,da,nl,ar_EG,en_GB,et,fi,de,
el,iw,hu,is,in,it,ja,ko,es,lv,lt,ms,es_MX,no,pl,pt,ro,ru,zh_CN,sk,sl,es_ES,sv,th,zh_TW,tr,uk,vi]]
]
DllGroup = false

INFO: Exception thrown from action: copyGroupFromJar
Exception Name: IOException2
Exception String: Error in writing to file '/production/oracle/10.2.0/opmn/lib/ons.jar'. [/production/oracle/10.2.0/opmn/lib/ons.jar (No such file or directory)]
Exception Severity: 2

A search of oraInventory/Contents revealed :

cd $ORACLE_HOME/oraInventory/Contents
strings * |grep production

#/production/oracle/10.2.0/oracle_prodsq
/production/oracle/10.2.0sq
0/production/oracle/10.2.0/oracle_javavm_server_cmpsq
//production/oracle/10.2.0/oracle_javavm_companionsq
,/production/oracle/10.2.0/oracle.javavm.clientsq
+/production/oracle/10.2.0/oracle_ldap_rsf_cmpsq
/production/oracle/10.2.0sq
/production/oracle/10.2.0sq
%/production/oracle/10.2.0/oracle.has.dbsq
#/production/oracle/10.2.0/oracle_prodsq


This should be actually showing

/development/oracle/10.2.0

I checked the adcfgclone.pl logs in $ORACLE_HOME/appsutil/log/$CONTEXT_NAME directory but did not find any errors.

I have asked the DBAs to re-run adcfgclone.pl. I'll update this post as I learn more.

Unix Error Codes

Error Codes are PORT specific, meaning that it is specific to each unix flavor.

The header file /usr/include/sys/errno.h has the error codes on any given machine

EPERM          1 No permission match
ENOENT       2 No such file or directory
ESRCH           3 No such process
EINTR           4 Interrupted system call
EIO                5 I/O error
ENXIO          6 No such device or address
E2BIG           7 Arg list too long
ENOEXEC    8 Exec format error
EBADF          9 Bad file number
ECHILD      10 No child processes
ENOMEM   12 Not enough space
EACCES      13 Permission denied
EFAULT     14 Bad address
EEXIST       17 File exists
EXDEV        18 Cross-device link
ENODEV     19 No such device
ENOTDIR   20 Not a directory
EISDIR        21 Is a directory
EINVAL      22 Invalid argument
EMFILE      24 Too many open files in a process
ETXTBSY   26 Text file busy
EFBIG         27 File too large
ESPIPE        29 Illegal seek
EMLINK      31 Too many links
EPIPE          32 Broken pipe

-- From here on Error vary WIDELY between machines

EDOM                33 Argument out of domain
ERANGE            34 Result too large
EIDRM               36 Identifier removed
EL2NSYNC        38 Level 2 not synchronized
EL3HLT             39 Level 3 halted
EL3RST             40 Level 3 reset
EL2HLT             44 Level 2 halted
EXFULL            52 Message tables full
ENOANO           53 Anode table overflow
EBADRQC         54 Bad request code
EBADSLT          55 Invalid slot
EDEADLOCK    56 File locking deadlock
EBFONT            57 Bad font file format
ENOSTR            60 Not a stream device
ENODATA         61 No data available
ETIME               62 Timer expired
ENOPKG           65 Package not installed
EREMOTE        66 Object is remote
EADV                 68 Advertise error
ESRMNT           69 Srmount error
EPROTO            71 Protocol error
EMULTIHOP    74 Multihop attempted
EBADMSG         77 Not a data message
ELIBACC           83 Can not access a needed shared lib.
ELIBBAD           84 Accessing a corrupted shared lib.
ELIBEXEC         87 Can not exec a shared library directly

AppsLocalLogin.jsp page takes forever to appear, browser shows hourglass

After upgrading to JDK 1.6.0_11, on both of our Extranet application tiers, we could not reach the AppsLocalLogin.jsp page. On invoking /oa_servlets/AppsLogin or /OA_HTML/AppsLocalLogin.jsp, the browser would show an hourglass with the status bar message "Waiting for extranet.justanexample.com". There were no errors in apache or jserv logs. The browser was showing hourglass or was waiting forever. Load balancer or network issues were ruled out as we were able to reach the RapidInstall page by commenting out url_fw.conf in httpd.conf.

After trying a lot of things, I recalled a similar problem in 2005, when 11.5.10 was newly released. In jserv.properties, we set the value of wrapper.bin.parameters=-DLONG_RUNNING_JVM=false. This resolved the issue.

The context file variable s_long_running_jvm controls the value of -DLONG_RUNNING_JVM in jserv.properties. If you want this to be permanent, then change the value of s_long_running_jvm in your context file.

I dug up the TAR from my old notes and got this:

Q1) What is the affect of setting

a)wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
           OR
b)wrapper.bin.parameters=-DCACHEMODE=LOCAL in jserv.properties?

Q2) Why did this resolve NoClassDefFound OR Internal Server Error ?  Is this the permanent solution?
Solution
A1) Distributed Caching was introduced in Framework 11.5.10, refer

From Note 275879.1 Oracle Applications Java Caching Framework Developer's Guide Release 11i (11.5.10)

The setting of the Distributed Mode option is optional. This is the default configuration of the caching framework. The LONG_RUNNING_JVM=true is set and it ensures that caching framework runs in distributed mode. For backward compatibility the same can be ensured by setting -DCACHEMODE=DISTRIBUTED. The above settings determines whether the updates and invalidation's to the objects in the Component Cache are distributed across other JVMs. This allows Component Caches where the data updates need to be seen instantaneously across the JVMs. If the flag is not checked, then updates to the data in the same JVM are seen right away, but updates are not be seen in other JVMs until the "Time to Live" or "Idle Time" expires, if the other JVMs happen to cache the same object. Distributed mode has network overhead and there are some additional steps required to enable this mode for the JVM.

Although turning off Distributed Caching in 11.5.10 will defeat Cache Invalidation, there have been known issues when trying to implement Distributed Caching in E-Business Suite. Until properly patched, Distributed Caching has caused problems with performance and/or rampant NoClassDefFound errors and/or other strange behaviors.

In summary

Cache Invalidation is a feature where some of the middle tier Java Caches for critical reference data are kept in sync when the reference information is changed. This feature was developed for Function/Data Security and Profile Option values. In 11.5.10 the Distributed Caching feature is used to propagate the changes to all "long running" JVMs. If one turns off Distributed Caching, then reference data changes will not be propagated out and one will be forced to bounce the middle tiers to pick up any changes, as done previously.
A2) The caching was out of sync between the web-server that were being load balanced, and with the additional network overhead, started to see apj12 ( Apache/Jserv connection protocol) errors. Once the first server started hanging, this then propagated to the next server, and the next server, faster and faster until all the servers eventually hung. By forcing the cache to reside locally, the network bandwidth was reduced, as well as the issue of trying to keep all the caches in sync with one another.

This is not the solution but a Test to confirm if the issue is occurring because of Java Object Caching.
Distributed Caching issues tend to be very sporadic across the different customers, as it tends to propagate itself based on different product patches that try and take advantage of the distributed caching, or implement new long running queries.

If this is an issue with JOC then best suggestion is to make sure the latest JOC patches ( or their super-seeded patch ) has been applied:
Patch 5639951 REHOST ORACLE JAVA OBJECT CACHE (FORMERLY OCS4J) FOR ORACLE APPLICATIONS 11i (Present in Foxtrot)
Patch 5455628 CACHE DIAGNOSTIC ARU ON TOP OF 11.5.10.3RUP (Not present in Foxtrot)
Patch 6047864 REHOST JOC FIXES (BASED ON JOC 10.1.2.2) FOR APPS 11i  (Not present in Foxtrot)
Also make sure that profile option Self Service personal Home Page mode set to Framework Only As this was written to be used with the new Framework Code and not with the old OSSWA code that is being phased out.
Note: Disabling Distributed Java Caching

1.  Because of the ATG_PF.H Rollup 3 (RUP 3) Patch 4334965, turning off the Java Object Cache (JOC) by setting
    "-DLONG_RUNNING_JVM=false" still calls underlying Apps JOC code.

To disable Java Cache completely follow these steps:

Edit the IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

a) Set  wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

b) Add the line  wrapper.bin.parameters=-DCACHEMODE=LOCAL

c) Restart Apache for these changes to take effect

If the customer problem stop reproducing after these steps, then the issue is certainly related to JOC 

2. Alternatively, if on an ATG RUP4 Patch 4676589 environment to confirm whether its a JOC related issue or not:

Completely REMOVE the system properties -DLONG_RUNNING_JVM and -DCACHEMODE from jserv.properties (either comment out these two system properties or remove them completely but do NOT set them to false and LOCAL respectively) This allows the cache to run in JOC local mode (not in JOC distributed mode) If the problem goes away then it is a JOC distributed caching issue

Create New Timesheet shows blank screen

Samir mailed today. After enabling Oracle Time and Labor in OAM as a newly licensed product, when they access the responsibility Timesheets > Create New Timesheet, the screen goes blank.

Apache logs looked fine. Finally I did a view source on the page and found this at the end:

top.fInitUnexpectedError("The system profile option setup indicates that your system is not licensed to run Self-Service Time. Please contact your system administrator and verify the profile option settings.")

I spoke to Vivek who suggested that we set these profile options to Yes:

PA: Licensed to use Project Time and Expenses
PA: Licensed to use Self-Service Time

After setting these to Yes, it proceeded ahead and showed a page with this message:

Some of your employee information is missing or invalid. Please notify your system administrator.

This was coming because there is no HR record for SYSADMIN account.

The Create Timesheet functionality works fine from Self Service Time responsibility.

I'll update this post, once I learn more.

Windows Command Line shortcuts

If you go to c:\windows\system32 and execute dir *.msc you'll find quite a few:

certmgr.msc        Certificates
ciadv.msc          Indexing Service
compmgmt.msc       Computer Management
devmgmt.msc        Device Management
dfrg.msc           Defragmenter
diskmgmt.msc       Disk Manager
eventvwr.msc       Event Viewer
fsmgmt.msc         File share management
gpedit.msc         Group Policy Editor
lusrmgr.msc        Local Users and Groups
ntmsmgr.msc        Removable Storage
ntmsoprq.msc       Removable Storage Operator Requests
perfmon.msc        Performance Monitor
rsop.msc           Resultant Set of Policy
secpol.msc         Security Policy
services.msc       Services
wmimgmt.msc        Windows Management Infrastructure

If you give a dir *.cpl you'll find more shortcuts for Control Panel applets:

access.cpl         Accessibility
appwiz.cpl         Add Remove Programs
bthprops.cpl       Bluetooth Properties 
desk.cpl           Display properties
firewall.cpl       Firewall
hdwwiz.cpl         Add new hardware wizard
inetcpl.cpl        Internet Properties
infocardcpl.cpl    Windows Cardspace
intl.cpl           Regional and Language options
irprops.cpl        Wireless Link properties
javacpl.cpl        Sun Java Control Panel
joy.cpl            Game Controllers
main.cpl           Mouse Properties
mmsys.cpl          Sound and audio device properties
ncpa.cpl           Network Connections Properties
netsetup.cpl       Network Setup Wizard
nusrmgr.cpl        User Accounts
nwc.cpl            Gateway Services for Netware properties
odbccp32.cpl       ODBC Administrator
plugincpl13125.cpl Oracle Jinitiator 1.3.1.25 control panel
plugincpl13128.cpl Oracle Jinitiator 1.3.1.28 control panel
plugincpl13129.cpl Oracle Jinitiator 1.3.1.29 control panel
powercfg.cpl       Power Options properties
sapi.cpl           Speech properties

sysdm.cpl          System Properties
telephon.cpl       Phone and Modem Options
timedate.cpl       Date and Time properties
wscui.cpl          Windows Security Center
wuaucpl.cpl        Windows Automatic Updates

You can call any of the above *.cpl or *.msc files from command prompt or Start > Run.

ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV]

We got this error in one of our Production instances whenever we ran Gather Schema Statistics program:

Errors in Database alert log:
$ORACLE_HOME/admin/$CONTEXT_NAME/udump/$TWO_TASK_ora_13052.trc:
ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address
not mapped to object] [0x000000000] [] []

Trace file:
ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address not mapped to object]
[0x000000000] [] []
Current SQL statement for this session:
begin :retval := "CTXSYS"."TEXTOPTSTATS".ODCIStatsCollect(:idx_info,:stat_opt,:stats,:env);
end;
----- PL/SQL Call Stack -----
object line object
handle number name
3b1247878 6 CTXSYS.TEXTOPTSTATS
3b1688f20 1 anonymous block
3b144c860 3379 package body SYS.DBMS_STATS
3b144c860 3639 package body SYS.DBMS_STATS
3b144c860 10584 package body SYS.DBMS_STATS
3b144c860 13153 package body SYS.DBMS_STATS
3b144c860 13202 package body SYS.DBMS_STATS
3a7244100 1 anonymous block
3b11b3e38 676 package body APPS.FND_STATS
3b11b3e38 1487 package body APPS.FND_STATS
3b11b3e38 1004 package body APPS.FND_STATS
3b11b3e38 782 package body APPS.FND_STATS
3c197dab0 1 anonymous block

We had an SR logged with Oracle for this. They noticed that we were not passing estimate percent which defaults to 10%. They asked us to explicitly give estimate percent as 30% and run GSS. The issue was resolved after this. From the SR:

ISSUE CLARIFICATION
====================

Gather Schema Statistics concurrent program issue for ' All ' schemas : it completes with error.


ISSUE VERIFICATION
===================
Verified the issue by the alert log file and trace file , which displays :

ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address not mapped to object] [0x000000000]" errors are encountered with core dumps.


CAUSE DETERMINATION
====================
Parameters to run Gather Shema Statistics are incorrect.

CAUSE JUSTIFICATION
====================
The following parameters are set as fillow :

GATHER OPTIONS: GATHER AUTO
Estimate Percent: 10


.
PROPOSED SOLUTION(S)
======================
GATHER OPTIONS: GATHER
Estimate Percent: 30


PROPOSED SOLUTION JUSTIFICATION(S)
====================================
GATHER AUTO : If underlying tables are not in monitoring mode , then "Gather Auto" option will try to change it. Since this is a DDL statement, it requires an exclusive lock on the table. If
table is under heavy load and frequently accessed , then this might take a while.
Note : 331017.1


Estimate Percent : Note.141532.1

Percentage of rows to estimate. If left empty it will default to 10%. The valid range is 0-99. A higher percentage will be more accurate, but take longer to run. If the object(s) that you
are gathering statistics for do not change often or the object(s) has data entered that is very similiar you may choose a lower number. However, if the data changes frequently a larger number entered for this parameter would be recommended to provide a more accurate representation of your data.

SOLUTION / ACTION PLAN
=======================
To implement the solution, please execute the following steps:

1. Schema Name: ALL
Estimate Percent: 30
Backup Flag : NOBACKUP
HISTORY MODE: LAST RUN
GATHER OPTIONS: GATHER
Invalidate Dependent Cursors: Y (Yes)


2. References :
Note. 141532.1 ' How To Gather Schema Statistics for Oracle Applications 11i '
Note. 331017.1 ' Gather Schema Statistics program hangs or fails with ORA-54 error '

LsInventorySession failed: Unable to create patchObject

Srinivas reported this error today:

opatch lsinventory showed this error at the end:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject

OPatch failed with error code 73

Metalink note 751152.1 suggested a few things but we didn't want to reclone the Oracle home. So we enabled debug mode:

export OPATCH_DEBUG=TRUE

And found that:

PatchObject::PatchObject looks for the actions file $ORACLE_HOME/inventory/oneoffs/5587976/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file $ORACLE_HOME/inventory/oneoffs/5587976/etc/config/inventory.xml
parserXMLFile : start
parserXMLFile : start
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file $ORACLE_HOME/inventory/oneoffs/5648872/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file $ORACLE_HOME/inventory/oneoffs/5648872/etc/config/inventory.xml
PatchObject::PatchObject now looks for the actions file $ORACLE_HOME/inventory/oneoffs/5648872/etc/config/actions
PatchObject::PatchObject now looks for the inventory file $ORACLE_HOME/inventory/oneoffs/5648872/etc/config/inventory
Loading the information from inventory/filemap.


Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
LsInventorySession failed: Unable to create patchObject
Cleaning up the directory : "$ORACLE_HOME/.patch_storage/patch_unzip"...

OPatch failed with error code 73

From the above output, it was clear that opatch was unable to find patch 5648872 in $ORACLE_HOME/inventory/oneoffs directory. So I compared the contents of $ORACLE_HOME/inventory/ContentsXML/comps.xml file in this instance and production:

In this instance:
$ cd $ORACLE_HOME/inventory/ContentsXML
$ grep 5648872 *
$ grep 5648872 comps.xml

5648872

In Production:
$ cd $ORACLE_HOME/inventory/ContentsXML
$ grep 5648872 *
comps.xml: 5648872

The lines



were extra in this instance.

Instead of manually editing the lines, I ftped the $ORACLE_HOME/inventory/ContentsXML/comps.xml file from Production and globally replaced the instance name in the file. Replacing the comps.xml file with this changed file from Production, resolved the issue.

We got this issue again after two months in a different instance. We checked metalink note 793600.1 which references Bug 7624037 A ROLLING PATCH UPGRADE ON TOP OF 10.2.0.4 CORRUPTS INVENTORY ON REMOTE NODES.

Even though our instance is not a RAC instance, we followed the advice and copied the $ORACLE_HOME/inventory/oneoffs directory from Production instance along with $ORACLE_HOME/inventory/ContentsXML/comps.xml file.

In comps.xml, we did a global replace of ORACLE_HOME path to point to development path instead of production path.

After this opatch succeeded.

Hope they fix the bug soon. Currently the patch suggested for the bug 7364519 (86 MB size) is password protected which means Oracle hasn't regression tested it yet.

Will update this post as I learn more.

Applications File Server could not Create a Temporary File

During migration to shared APPL_TOP, we changed the APPLTMP and APPLPTMP and mount point to a dedicated mount for temporary files. The DBAs not wanting to run autoconfig, just changed the environment variable and bounced the services. While viewing the log or output file of any concurrent request this error occurred:

“The Applications File Server could not create a temporary file.”

Metalink Note 433104.1 describes this issue:

Cause

Incorrect setup of the apps.conf configuration file

As per the customer's last update, "we found there were some source instance entries in the $IAS_ORACLE_HOME/Apache/Apache/conf/apps.conf with some customization for temp files."

As per Note 276695.1, "FNDWRR does use APPLTMP as its temp directory, but remember FNDWRR is a CGI program so you must set APPLTMP in your WEBSERVER environment."
Solution

To implement the solution, please execute the following steps:

1. Ensure the “apps.conf” file is configured correctly, particularly temporary directory paths like "SetEnv APPLTMP"

2. Relink Applications programs and Generate product JAR files via adadmin.

There was no need to relink programs or generate JAR files. We just changed the SetEnv APPLTMP line to point to the correct directory. This resolved the issue.

FRM-92500 Failed to connect to the server

After migrating to shared appl_top on new servers, we got the following error while accessing forms:

ERROR
FRM-92500 Failed to connect to the server: myhost.oracle.com:9000

Details:
Java Exception:
Java.security.accessControlException: access denied (java.net.socketPermission myhost.oracle.com resolve)
at java.security.AccessControlContext.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkConnect(Unknown Source)
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at java.net.InetAddress.getByName(Unknown Source)
at java.net.InetSocketAddress.(Unknown Source)

Metalink Note 753152.1, describes this problem.

We had already run Regenerate jar files, so we ran Regenerate jar files with force option this time.

The problem was resolved after the jar files were regenerated with force option.

LRM-00123: invalid character 0 found in the input file

When starting the Oracle database with Veritas Cluster Server, we got the following error:

LRM-00123: invalid character 0 found in the input file
ORA 1078 : Text: failure in processing initialization parameters

Oracle reads the PFILE/SPFILE information from our internal character set identified with PL/SQL and not the database created character set. While technically not hard-coded as US7ASCII, this is what the character set used is for SSTDLANG.

The problem is caused by ORACORE's enforcement of compiler character set when reading the INIT.ORA file. SSTDLANG is the limitation.

What we noticed is that by mistake we had given the name of the spfile as the pfile name. Veritas Oracle agent needs a plain text ASCII file like init.ora. So we create a file called vcs_init.ora which has a single line
spfile=path_to_spfile

Once we corrected this VCS was able to bring up the database without issues.

ORA-01403 ORA-06512 at "APPS.AD_FILE_SYS_SNAPSHOTS_PKG", line 2472

Recently while doing shared APPL_TOP migration, we got this error while trying to update current snapshot:

About to attempt instantiating the current-view snapshot: Fri Feb 13 2009 22:24:14

Attempting to instantiate the current-view snapshot...
ORA-01403: no data found
ORA-06512: at "APPS.AD_FILE_SYS_SNAPSHOTS_PKG", line 2472
ORA-06512: at line 2

AD Administration error:
Error while executing statement (curr-vw inst)


AD Administration error:
Error calling adusnapInstCurrVwSnapshot

AD Administration error:
Could not get SnapshotId

Failed to store APPL_TOP Snapshot to tables
adusnapMaintainSnapshot() : Error calling adusnapUpdateCurrentView()
Error calling adusnapMaintainSnapshot

Metalink Note 284759.1 gave a clue as it described this error on the clone of Production. However we were doing this on Production and had not faced this error in any of the iterations before. I checked the name of the appl_top in ad_appl_tops table and found that it was incorrect. The name of the appl_top refereed to the old server name on which the application was running 6 months ago. 6 months ago we had migrated to the new server but never changed the appl_top name. I went through the code of AD_FILE_SYS_SNAPSHOTS_PKG and found that it was calling ad_snapshots. The appl_top id in ad_appl_tops and the appl_top_id in ad_snapshots was not matching. This was because we had cleaned up ad_appl_tops and run autoconfig on all tiers. I searched in dba_tab_columns and found that there were 16 tables which had appl_top_id column. Updating all these tables with the new appl_top_id would have been a pain. So instead I suggested the DBAs to

update ad_appl_tops
set appl_top_id = 'appl_top_id of the old server'
where appl_top_id = 'newly generated appl_top_id for new server';

Once the DBAs did this, the update current snapshot process completed without errors.

Portal 3.0.9 first page appears on accessing /oa_servlets/AppsLogin

While logging on to a test instance, I saw this error:

Forbidden

You don't have permission to access /pls/$TWO_TASK_portal30/portal30.home on this server.

apache access_log shows

192.168.11.23 - - [13/Feb/2009:10:41:14 -0500] "GET /pls/$TWO_TASK_portal30/portal30.home HTTP/1.1" 403 245 0

access_log_pls shows

error_log_pls shows

[Fri Feb 13 10:40:56 2009] [error] mod_plsql: /pls/$TWO_TASK_portal30/portal30.home HTTP-403 ORA-1017 Proxy log On failed.
Please verify that you have specified correct connectivity information i.e. username, password & connect-string in the DAD

[Fri Feb 13 10:41:14 2009] [error] mod_plsql: /pls/$TWO_TASK_portal30/portal30.home HTTP-403 ORA-1017 Proxy log On failed.
Please verify that you have specified correct connectivity information i.e. username, password & connect-string in the DAD

I checked the wdbsvr.app file and found this in the portal30 DAD:

[DAD_apps11i_portal30]

connect_string  =  apps11i

password        =  secretseven

username        =  portal30

default_page    =  portal30.home

document_table  =  portal30.wwdoc_document

document_path   =  docs

document_proc   =  portal30.wwdoc_process.process_download

upload_as_long_raw =

upload_as_blob  = *

reuse           =  Yes

connmax         = 10

enablesso       = Yes

pathalias       = url

pathaliasproc   = portal30.wwpth_api_alias.process_download

cgi_env_list=SERVER_NAME=apps11i.justanexample.com,

REQUEST_PROTOCOL=http,SERVER_PORT=8002,HOST=apps11i.justanexample.com:8000

I checked in the db

SQL> conn portal30/secretseven

ERROR:

ORA-01017: invalid username/password; logon denied

Since this was a test instance, I decided to see what happens if we change the password of portal30 user to secretseven.

So I changed the password of portal30 to secretseven

conn / as sysdba

alter user portal30 identified by secretseven

Accessing the /oa_servlets/AppsLogin page took me to:

http://apps11i.justanexample.com:8000/servlet/page?_pageid=51&_dad=apps11i_portal30&_schema=PORTAL30

which showed the Oracle Portal 3.0.9 first page:

I tried accessing it through: 

http://apps11i.justanexample.com:8000/OA_HTML/AppsLocalLogin.jsp

That showed me the expected username and password page:

Will update as I learn more.

Solaris 10 zone won't plumb up virtual IPs

Recently unix team rebooted Solaris 10 zones and the zones wouldn't come up. This was tracked down to /etc/hostname* files :

$ ls -ltr /etc/hostname*
-rw-r--r-- 1 root root 9 Oct 15 10:41 /etc/hostname.nxge3
-rw-r--r-- 1 root root 9 Jan 19 10:44 /etc/hostname.nxge3:1
-rw-r--r-- 1 root root 9 Jan 20 16:21 /etc/hostname.nxge3:2

$ cat /etc/hostname.nxge3
server1

$ cat /etc/hostname.nxge3
vip1

$ cat /etc/hostname.nxge3
vip2

The first file without a colon is the physical name of the zone. The :1 and :2 are virtual names for the server. These files contain only the hostname without domainname. Unless you have corresponding entry for these virtual hostnames in /etc/hosts the zone will not plumb up the virtual IPs. As per our example above these entries need to be present in the /etc/hosts file:

192.168.7.1 server1.justanexample.com
192.168.7.1 vip1.justanexample.com
192.168.7.1 vip2.justanexample.com

ORA-12154 during adconfig

sqlplus /nolog @$COMMON_TOP/admin/install/$CONTEXT_NAME/txkJavaMailerCfg.sql -nopromptmsg $TWO_TASK http 8020

AutoConfig Profile Phase
Running Profile Process 9 of 11 for AD_TOP
Executing script in InstantiateFile:
$COMMON_TOP/admin/install/erppgjh1_erppgjh1/txkJavaMailerCfg.sh

script returned:
****************************************************

txkJavaMailerCfg.sh started at Wed Feb 11 11:06:24 EST 2009


The environment settings are as follows ...

ORACLE_HOME : /apps11i/erpapp/8.0.6
ORACLE_SID :
TWO_TASK : apps11i
PATH : /apps11i/erpapp/iAS/Apache/perl/bin:/apps11i/erpapp/8.0.6/bin:/apps11i/erpapp/appl/fnd/11.5.0/bin:/apps11i/erpapp/appl/ad/11.5.0/bin:$COMMON_TOP/util/java/1.4/j2sdk1.4.2_04/bin:$COMMON_TOP/util/unzip/unzip::/apps11i/erpapp/8.0.6/bin:/usr/bin:/usr/ccs/bin:/usr/sbin:/apps11i/erpapp/iAS/Apache/perl/bin:/apps11i/erpapp/AventX/bin:/apps11i/erpapp/iAS/Apache/perl/bin:/apps11i/erpapp/8.0.6/bin:$COMMON_TOP/util/java/1.4/j2sdk1.4.2_04/bin:$COMMON_TOP/util/unzip/unzip:/apps11i/erpapp/8.0.6/bin:/usr/bin:/usr/ccs/bin:/usr/sbin:/usr/local/bin:/usr/bin:/usr/openwin/bin:/usr/dt/bin:/sbin:/usr/sbin:/usr/platform/sun4u/sbin:/usr/ucb:/usr/ccs/bin:/usr/java/bin
LD_LIBRARY_PATH : /apps11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads:/apps11i/erpapp/appl/cz/11.5.0/bin:/apps11i/erpapp/8.0.6/lib:/usr/dt/lib:/usr/openwin/lib


Executable : /erppgjh1/erpapp/8.0.6/bin/sqlplus


The arguments passed to txkJavaMailerCfg.sql are as follows ...

USERNAME : -nopromptmsg
PASSWORD :
TWO_TASK : erppgjh1
URL_PROTO_GENERAL : http
PORT_GENERAL : 8020

-nopromptmsg erppgjh1 http 8020

SQL*Plus: Release 8.0.6.0.0 - Production on Wed Feb 11 11:06:25 2009

(c) Copyright 1999 Oracle Corporation. All rights reserved.

ERROR:
ORA-12154: TNS:could not resolve service name


ERRORCODE = 1 ERRORCODE_END
.end std out.

.end err out.
****************************************************

We have a TAR open with Oracle for this. They have advised us to apply TXK Rollup S patch 6372396:

ACTION PLAN
============

Please apply patch ='6372396' and retest issue this issue is addressed in
Bug 6443745 - CLNADMPRF.SH FAIL WITH ORA-12154 WHILE RUNNING AUTOCONFIG

and the fix is patch 6372396.

Applying the TXK Rollup S patch 6372396 resolved this problem.

We also faced this issue in instances which are already on TXK Rollup S.  As a quick-fix we re-applied TXK Rollup S to correct this issue.  I am still investigating this issue for a more logical fix.

Where is the workflow notification mailer smtp and imap hostname stored in DB

This query tells you where the Workflow notification mailer is running:

select target_node
from apps.fnd_concurrent_queues
where concurrent_queue_name like 'WFMLRSVC%';

This query tells you the IMAP inbound server name, SMTP outbound server name, value of reply to email address:

SELECT b.component_name,
c.parameter_name,
a.parameter_value
FROM apps.fnd_svc_comp_param_vals a,
apps.fnd_svc_components b,
apps.fnd_svc_comp_params_b c
WHERE b.component_id = a.component_id
AND b.component_type = c.component_type
AND c.parameter_id = a.parameter_id
AND c.encrypted_flag = 'N'
AND b.component_name like '%Mailer%'
AND c.parameter_name in ('INBOUND_SERVER','OUTBOUND_SERVER', 'REPLYTO')
ORDER BY c.parameter_name;

What's inside FND_LOBS ?

FND_LOBS table is one of the largest objects in an E-Business Suite instance. In our largest instance the size of this table is 160 GB.

FND_LOBS table contains all the attachments which were attached by users in all the modules of E-Business Suite, since the instance was created.

To know which Forms provide Attachment feature
----------------------------------------------------------------------

SQL>select *

from fnd_attachment_functions

where function_name like '%FND_%';

You can see the contents of fnd_lobs through this query:

select file_name, file_id,
to_char(upload_date,'dd-mon-rr hh24:mi:ss'),
to_char(expiration_date ,'dd-mon-rr hh24:mi:ss'),
file_content_type,
dbms_lob.getlength(file_data) size_byte
from applsys.fnd_lobs
--where file_id= ;

Metalink Note 338651.1 has more information about FND_LOBS:

FND_LOBS stores information about all LOBs managed by the Generic File Manager (GFM).
Each row includes the file identifier, name, content-type, and actual data. Each row also includes the dates the file was uploaded and will expire, the associated program name and tag, and the language and Oracle characterset.

The file data, which is a binary LOB, is stored exactly as it is uploaded from a client browser, which means that no translation work is required during a download to make it HTTP compliant.

Therefore uploads from non-browser sources will have to prepare the contents
appropriately (for instance, separating lines with CRLF).

The program_name and program_tag may be used by clients of the GFM for any purpose,
such as striping, partitioning, or purging the table if the program is de-installed.
They are otherwise strictly informative.

These columns and the expiration date are properly set when the
procedure FND_GFM.CONFIRM_UPLOAD is called. If not called, the column
expiration_date remains set, and will eventually be purged by the procedure
FND_GFM.PURGE_EXPIRED.

FND_DOCUMENTS_LONG_RAW stores images and OLE Objects, such as Word Documents and Excel spreadsheets, in the database. If the user elects to link an OLE Object to the document, this table stores the information necessary for Oracle Forms to activate the OLE server, and it saves a bit-mapped image of the OLE server's contents.
If the user does not elect to link an OLE Object, the entire document will be stored in this table.

FND_DOCUMENTS_LONG_TEXT stores information about long text documents.

FND_DOCUMENTS_SHORT_TEXT stores information about short text documents.

Relaying denied 550 5.7.1

Relaying denied error was received during this test:

Login to appserver.justanexample.com
telnet pluto.justanexample.com 25
MAIL FROM:pink.panther@justanexample.com
250 2.1.0 pink.panther@justanexample.com... Sender ok
RCPT TO:anthony.gonsalves@justanexample.com
550 5.7.1 anthony.gonsalves@justanexample.com... Relaying denied

In sendmail log in /var/log/sylog, the corresponding log entry was:

Feb 10 09:11:12 pluto sendmail[13783]: [ID 801593 mail.info] n1AEB3Ea013783: Authentication-Warning: pluto.justanexample.com: physicalpluto.justanexample.com [192.168.9.22] didn't use HELO protocol
Feb 10 09:11:21 pluto sendmail[13783]: [ID 801593 mail.notice] n1AEB3Ea013783: ruleset=check_rcpt, arg1=anthony.gonsalves@justanexample.com, relay=appserver.justanexample.com[192.168.4.1], reject=550 5.7.1 anthony.gonsalves@justanexample.com Relaying denied

I googled on ruleset=check_rcpt, and found this from sendmail.org:
check_rcpt
The address supplied through the RCPT command can be checked against the check_rcpt ruleset. On first look, this ruleset doesn't make much sense. Why check the recipient? sendmail does this anyway when trying to deliver, esp. for local recipients. However, this ruleset can be used to check whether your system is (mis)used as a relay. The check_compat ruleset, which seems to be better suited for this purpose, since it gets both addresses (sender and recipient) as parameters, is called too late. To reject a misuse at the earliest moment (and save your bandwidth etc), you can refer to the address of the sending system, which is available in the macro ${client_addr}. However, to use it in a rule, you have to refer to it as: $(dequote "" $&{client_addr} $) so sendmail defers evaluation and tokenizes it. The old solution is based on a proposal from Chin Huang: But since there is a problem with these rules, here is a new solution. First, we check whether it is a local client: it can do whatever it want. Next, we remove the local part, maybe repeatedly. If it still has routing information in it, it seems to be a relay attempt. (There are also a more detailed explanation of this ruleset and some additional thoughts.)
So list in the class

F{LocalIP} /etc/mail/LocalIP

the IP addresses of the local clients you will allow to relay through your mail server, for example

134.245 127.0.0.1

A client which connects from one of these IP numbers can send mail through your relay anywhere.

Scheck_rcpt # first: get client addr R$+ $: $(dequote "" $&{client_addr} $) $| $1 R0 $| $* $@ ok no client addr: directly invoked R$={LocalIP}$* $| $* $@ ok from here # not local, check rcpt R$* $| $* $: $>3 $2 # remove local part, maybe repeatedly R$+ $:$>removelocal $1 # still something left? R$*<@$*>$* $#error $@ 5.7.1 $: 550 we do not relay Sremovelocal # remove RelayTo part (maybe repeatedly) # R$*<@$*$={RelayTo}.>$* $>3 $1 $4 R$*<@$=w.>$* $: $>removelocal $>3 $1 $3 R$*<@$*>$* $@ $1<@$2>$3 # dequote local part R$+ $: $>3 $(dequote $1 $) R$*<@$*>$* $: $>removelocal $1<@$2>$3

The trailing $* after $={LocalIP} matches incompletely specified IP addresses on octet boundaries, as can be seen by 134.245 which matches a whole class B subnet.

If you relay mail for other systems (e.g., the secondary MX of a system points to your mailhost or your server is the primary MX, but you forward the mail to another system), use also:

F{RelayTo} /etc/mail/RelayTo

to list all hosts you relay mail to or accept mail for. For example, we put

uni-kiel.de

in RelayTo. Then enable the rule

R$*<@$*$={RelayTo}.>$* $>3 $1 $4

The leading $* will match subdomains of those domains in RelayTo too. You can also use a map instead of a class, if you slightly change the rules.

If you encounter a problem with this ruleset, try to find out which options you're missing or debug it directly.

However adding /etc/mail/LocalIP didn't help as our sendmail version was 8.13.8 and most probably this feature is available in Sendmail 9. I searched inside sendmail configuration file sendmail.cf for filenames which are checked:

# grep /etc/mail /etc/mail/sendmail.cf
Fw/etc/mail/local-host-names
FR-o /etc/mail/relay-domains
O AliasFile=/etc/mail/aliases
#O ErrorHeader=/etc/mail/error-header
O HelpFile=/etc/mail/helpfile
O StatusFile=/etc/mail/statistics
#O UserDatabaseSpec=/etc/mail/userdb
#O ServiceSwitchFile=/etc/mail/service.switch
#O DefaultAuthInfo=/etc/mail/default-auth-info
Ft/etc/mail/trusted-users

/etc/mail/local-host-names stood out. So I seached on that. and found this comment on http://www.ozzu.com/unix-linux-forum/how-setup-relay-host-sendmail-t29690.html

/etc/mail/relay-domains contains a list of hosts which are allowed to relay mail through your mail server. This list may consist of either specific hosts or whole domains.

/etc/mail/sendmail.cw (after sendmail version 8.10, this file is local-host-names) contains a list of domains for which your mail server will accept mail. This list is usually the domains hosted by your machine.

NOTE: put a CR/LF at the end of the last domain. Otherwise it won't work.

So I added appserver.justanexample.com in /etc/mail/local-host-names and bounced sendmail:

svcadm restart sendmail

The relay denied error stopped coming after this change.

dms0/Spy dms1/Spy oprocmgr-service http-403 http-404 in apache access log

Recently we saw these HTTP-404 and HTTP-403 errors in Apache Logs:

/oprocmgr-service?cmd=Getprocs HTTP/1.1" 404 222 0

192.168.9.2 - - [09/Feb/2009:13:54:46 -0500] "GET /dms1/Spy?recurse=all&format=xml&operation=get&value=false&units=true&description=true&name=/&noop=/ HTTP/1.1" 403 218 0
192.168.2.33 - - [09/Feb/2009:13:54:46 -0500] "GET /dms0/Spy?recurse=all&format=xml&operation=get&value=false&units=true&description=true&name=/&noop=/ HTTP/1.1" 403 218 0


Metalink Note 579106.1 tells us the cause and resolution:

Cause

These messages are generated when you use the OEM Grid Control to monitor the HTTP server or other application server components from it .

The reason is the Grid control is not certified to monitor iAS 1.0.2.2.

Solution

To implement the solution , please execute the following steps to avoid generation of these messages :

1. Stop the OEM Grid agent.

2. Remove the monitoring for the iAS 1.0.2.2 used by E-Business Suite

3. Bounce Apache server.

RDA standalone

RDA or Remote Diagnostic Agent is a set of diagnostic scripts provided by Oracle support which provides them with valuable information about your instance. Recent versions of RDA come bundled with Software Configuration Manager which runs as a cron job in your system and periodically collects configuration information about your instance and sends it to Oracle.

Metalink Note 330472.1 has a link for standalone RDA without Software Configuration Manager.

Metalink Note: 330363.1 (RDA4 FAQ)

Good queries on FND tables for Apps11i

1. TO FIND APPLICATION DETAILS
===============================

This will provide you application id alongwith shortname and base path of it:

SELECT *
FROM fnd_application;


2. TO FIND PRODUCT INSTALLATION DETAILS
========================================

SELECT application_id,
product_version,
status,
patch_level
FROM fnd_product_installations;

Note: Here application_id would be id as per query no. 1, product_version could
be '11.5.0', status could be 'I' / 'S' / 'N' and patch_level could be '11i.AD.I'

3. TO FIND THE TABLE DETAILS
=============================

SELECT application_id,
table_id,
table_name,
user_table_name,
description
FROM fnd_tables
WHERE table_name LIKE UPPER('%&table_name%');

Note: Enter the table_name for which you want to find out details. You can put
application_id to filter tables.


4. TO FIND TABLE COLUMNS
=========================

Make sure to enter the table name in CAPS only:

SELECT application_id,
table_id,
column_id,
column_name,
user_column_name,
column_sequence,
column_type,
width,
description
FROM fnd_columns
WHERE table_id =
(SELECT table_id
FROM fnd_tables
WHERE table_name LIKE UPPER('&table_name'));

Note: Enter the table_name for which you want to find out column details.


5. TO FIND VIEWS DETAILS
=========================
SET LONG 1000

SELECT application_id,
view_id,
view_name,
description,
text
FROM fnd_views
WHERE view_name LIKE UPPER('%&view_name%');


6. TO FIND VIEW COLUMNS
========================

SELECT application_id,
view_id,
column_sequence,
column_name
FROM fnd_view_columns
WHERE view_id =
(SELECT view_id
FROM fnd_views
WHERE view_name LIKE UPPER('&view_name'));

Note: Enter the view_name for which you want to find out view column details.

7. TO FIND CURRENCY DETAILS
===========================

SELECT currency_code,
symbol,
enabled_flag,
currency_flag,
description,
precision,
extended_precision,
minimum_accountable_unit,
start_date_active,
end_date_active
FROM fnd_currencies
WHERE currency_code LIKE '%¤cy_code%';

Note: Here currency_code could be 'USD', 'GBP' etc.

8. TO FIND THE EXECUTABLE DETAILS
==================================

SELECT application_id,
executable_id,
executable_name,
execution_file_name,
subroutine_name,
icon_name,
execution_file_path
FROM fnd_executables
WHERE application_id = &application_id AND executable_name LIKE '%&
executable_name%';

Note: Here application_id could be id as per query no. 1 and executable_name
could be 'APXPBFOR'

9. TO FIND INDEX DETAILS
========================
SELECT dba.status,
fnd.application_id,
fnd.table_id,
fnd.index_id,
fnd.index_name,
fnd.description
FROM fnd_indexes fnd,
dba_indexes dba
WHERE table_id =
(SELECT table_id
FROM fnd_tables
WHERE table_name LIKE UPPER('&table_name')) and fnd.index_name = dba.
index_name;

Note: Enter the table_name for which you want to find out index details.

10. TO FIND INDEX COLUMNS
==========================

SELECT application_id,
table_id,
index_id,
column_sequence,
column_id
FROM fnd_index_columns WHERE table_id =
(SELECT table_id
FROM fnd_tables
WHERE table_name = UPPER('&table_name'));

Note: Enter the table_name for which y.
ou want to find out index columns. If
you are aware of index_id from query no. 9 above then specify that in the where
clause instead of table_name.

11. TO KNOW PRIMARY KEY DETAILS
===============================

SELECT application_id,
table_id,
primary_key_id,
primary_key_name,
description,
enabled_flag
FROM fnd_primary_keys
WHERE table_id =
(SELECT table_id
FROM fnd_tables
WHERE table_name LIKE UPPER('&table_name'));

Note: Enter the table_name for which you want to find out primary key details.

12. TO FIND SEQUENCE DETAILS
============================

SELECT application_id,
sequence_id,
sequence_name,
start_value,
increment_by,
min_value,
max_value,
cache_size,
cycle_flag,
order_flag,
description
FROM fnd_sequences
WHERE sequence_name = UPPER('&sequence_name');


13. TO FIND PROFILE OPTION DETAILS
==================================

SELECT application_id,
profile_option_id,
profile_option_name,
site_enabled_flag,
resp_enabled_flag,
user_enabled_flag
FROM fnd_profile_options
WHERE profile_option_name LIKE UPPER('%&profile_option_name%');


14. TO FIND FOLDERS DETAILS
============================

Note: Here the NAME is completely case sensitive and hence needs to be passed
as defined in FOLDER:

SELECT folder_id,
object,
name,
public_flag,
autoquery_flag,
where_clause,
order_by
FROM fnd_folders
WHERE name like '%&name%';.

Version Conflicts utility failed

Mohan pinged me about one issue in R12 upgrade.  He had successfully installed R12 on DB node and primary node file system.  However he was facing issues on secondary node.

This was a shared application file system.  This was the error:

RW-50016: Error: - APPL_TOP environment file was not created:
File = /R12/appl/apps_st/R12appl/$CONTEXT_NAME.env
ADOVARS environment file passes instantiated variables test:
File = $APPL_TOP/admin/adovars.env

APPSCONFIG passes instantiated variables test:
File = $APPL_TOP/admin/adconfig.txt

in autoconfig log file is see this error

Starting AutoConfig at Thu Feb 5 13:22:01 2009
Using adconfig.pl version 120.16.12000000.8

Classpath : $OA_JAVA/lib/appsborg2.zip:$OA_JAVA/classes

ERROR: Version Conflicts utility failed.
Terminate.

A quick search on metalink yielded note 559806.1

Here are the steps for the workaround:
1. Do the installation using the setting that you require. This will fail due to the bug. (if you have a current failed installation then use that)
2. Copy the config file ($INST_TOP/conf_.txt) from the failed Installation file to a different location and modify the s_javatop variable with the correct value :
For example :
change from:
s_javatop=/u00/TEST/apps/apps_st/comn/java/classes
to:
s_javatop=/u00/TEST/maxcomn/java/classes

(Change other variables if they are incorrect if any)

3. Delete the failed install but preserve the modified conf_.txt . ie delete evrything but the altered config file.
4. Point the /etc/oraInst.loc to a new location to create a new oraInventory, or delete if you are using a single oraInventory for each installation. If this is the only installation on this Node then delete the current oraInventory.
5. Install using the modified config file as follows :
rapidwiz -silent -config .txt
For Example :
$ rapidwiz -silent -config /u00/TEST/conf_TEST.txt

The current s_javatop in config file was
/R12/appl/apps_st/apps/apps_st/comn/java/classes

This was changed to

/R12/appl/apps_st/R12comn/java/classes

Mohan said that it is really a bug.  He had checked config file for the other two nodes and s_javatop was correctly set on those,  but for 3rd node it is set differently

Mohan deleted the sid_host directory from inst_top and started rapidwiz

It ran fine and generated the env file.