^M character

Vinu asked me about ^M character in his scripts. ^M denotes carriage return. 

You can verify this by pressing the keys Ctrl and M together on your windows command prompt or unix prompt.

I told him that it is a well known issue. Whenever an ASCII file is transferred in binary mode to unix, ^M character is appended after every line. To convert it to correct unix format, you can use the dos2unix utility and the following command:

dos2unix original.file withoutM.file

In a loop (originally referenced here) you can fix all files:

$ for files in `find . -print`;
> do
> mv $files $files.old
> dos2unix -437 $files.old $files
> rm -rf $files.old
> done

Some of the other characters from Ctrl+A to Ctrl+Z:

Oct Dec Hex Name
000 0 0x00 NUL
001 1 0x01 SOH, Control-A
002 2 0x02 STX, Control-B
003 3 0x03 ETX, Control-C
004 4 0x04 EOT, Control-D
005 5 0x05 ENQ, Control-E
006 6 0x06 ACK, Control-F
007 7 0x07 BEL, Control-G
010 8 0x08 BS, backspace, Control-H
011 9 0x09 HT, tab, Control-I
012 10 0x0a LF, line feed, newline, Control-J
013 11 0x0b VT, Control-K
014 12 0x0c FF, form feed, NP, Control-L
015 13 0x0d CR, carriage return, Control-M
016 14 0x0e SO, Control-N
017 15 0x0f SI, Control-O
020 16 0x10 DLE, Control-P
021 17 0x11 DC1, XON, Control-Q
022 18 0x12 DC2, Control-R
023 19 0x13 DC3, XOFF, Control-S
024 20 0x14 DC4, Control-T
025 21 0x15 NAK, Control-U
026 22 0x16 SYN, Control-V
027 23 0x17 ETB, Control-W
030 24 0x18 CAN, Control-X
031 25 0x19 EM, Control-Y
032 26 0x1a SUB, Control-Z

FRM-92050 Failed to connect to the server

Srinivas reported this error after converting an instance to shared application file system:

Forms interface was not launching with this error:

FRM-92050: Failed to connect to the Server: apps11i.justanexample.com:8000

Details:
Java Exception:
java.io.EOFException
at java.io.DataInputStream.readInt(Unknown Source)
at oracle.forms.net.SocketConnection.connect(Unknown Source)
at oracle.forms.engine.Runform.initConnection(Unknown Source)
at oracle.forms.engine.Runform.startRunform(Unknown Source)
at oracle.forms.engine.Main.createRunform(Unknown Source)
at oracle.forms.engine.Main.start(Unknown Source)
at sun.applet.AppletPanel.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I tried doing a telnet apps11i.justanexample.com 8000. This is a good test to see if the service is running. The service was not running. It was started through adfrmctl.sh. When I tried again, it connected but on pressing enter the connection was broken, which should not happen.

I found a direct hit in Metalink Note: 438231.1

Forms Server is up (f60webmx process is shown as a result of "ps -ef | grep f60" command) but when starting the forms server with the following command instead of adfrmctl.sh:

f60srvm em=__QA port= log=forms_log.txt mode=socket exe=f60webmx 2>&1 | tee forms_out.txt &

It returns
Failed to exec runform f60webmx

Cause

The issue is caused by the following setup: $FND_TOP/bin/f60webmx is renamed (e.g. to $FND_TOP/bin/f60webmx.sav) or removed and thus is inaccessible.

A similar issue is described in Metalink Note 1079806.6: Failed to Exec Runform when Starting Forms Server.

To relink f60webmx, I executed:

adrelink.sh force=y ranlib=y "fnd f60webmx"

It errored out with:

make -f $APPL_TOP/admin/out/link_fnd_21892.mk $FND_TOP/bin/f60webmx

Starting link of fnd executable 'f60webmx' on Fri Mar 27 16:02:14 EDT 2009
make: Fatal error: Don't know how to make target `$FND_TOP/lib/xitiap.o'
Done with link of fnd executable 'f60webmx' on Fri Mar 27 16:02:14 EDT 2009

Metalink Note 178919.1 describes this problem and sites missing file as the cause.

When I checked for $FND_TOP/lib/xitiap.o, it was indeed missing.

So I compared the number of files in $FND_TOP/lib in this instance and another instance. Total of 4 files were missing in this instance:

afpwrr.o
fdrrun.o
xitfnd.o
xitiap.o

These 4 files were copied from a working instance, and Srini ran:

adrelink.sh force=y ranlib=y "fnd f60webmx"

f60webmx got created without any errors this time.

FRM-92050 error also disappeared as forms was able to locate f60webmx.

java.lang.SecurityException: cannot verify signature block file META-INF/CUST

Anand called me at 1:30 AM tonight. After applying Jan 2009 CPU, they were getting weird errors. Java console gave these errors:

Oracle JInitiator: Version 1.3.1.29
Using JRE version 1.3.1.29-internal Java HotSpot(TM) Client VM
User home directory = C:\Documents and Settings

Proxy Configuration: Browser Proxy Configuration

JAR cache enabled
Location: C:\Documents and Settings\Oracle Jar Cache
Maximum size: 50 MB
Compression level: 0
----------------------------------------------------
c: clear console window
f: finalize objects on finalization queue
g: garbage collect
h: display this help message
l: dump classloader list
m: print memory usage
q: hide console
s: dump system properties
t: dump thread list
x: clear classloader cache
0-5: set trace level to
----------------------------------------------------

Downloading http://apps11i.justanexample.com/OA_JAVA/oracle/apps/fnd/jar/fndforms.jar to JAR cache

java.lang.SecurityException: cannot verify signature block file META-INF/CUST
at sun.security.util.SignatureFileVerifier.process(Unknown Source)
at java.util.jar.JarVerifier.processEntry(Unknown Source)
at java.util.jar.JarVerifier.update(Unknown Source)
at java.util.jar.JarFile.initializeVerifier(Unknown Source)
at java.util.jar.JarFile.getInputStream(Unknown Source)
at oracle.jre.protocol.jar.JarCache$CachedJarLoader.authenticate(Unknown Source)
at oracle.jre.protocol.jar.JarCache$CachedJarLoader.access$4100(Unknown Source)
at oracle.jre.protocol.jar.JarCache$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.jre.protocol.jar.JarCache.privileged(Unknown Source)
at oracle.jre.protocol.jar.JarCache.access$2800(Unknown Source)
at oracle.jre.protocol.jar.JarCache$CachedJarLoader.download(Unknown Source)
at oracle.jre.protocol.jar.JarCache$CachedJarLoader.load(Unknown Source)
at oracle.jre.protocol.jar.JarCache.get(Unknown Source)
at oracle.jre.protocol.jar.CachedJarURLConnection.connect(Unknown Source)
at oracle.jre.protocol.jar.CachedJarURLConnection.getJarFile(Unknown Source)
at sun.misc.URLClassPath$JarLoader.getJarFile(Unknown Source)
at sun.misc.URLClassPath$JarLoader.(Unknown Source)
at sun.misc.URLClassPath$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(Unknown Source)
at sun.misc.URLClassPath.getLoader(Unknown Source)
at sun.misc.URLClassPath.getResource(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at sun.applet.AppletClassLoader.findClass(Unknown Source)
at sun.plugin.security.PluginClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.applet.AppletClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.applet.AppletClassLoader.loadCode(Unknown Source)
at sun.applet.AppletPanel.createApplet(Unknown Source)
at sun.plugin.AppletViewer.createApplet(Unknown Source)
at sun.applet.AppletPanel.runLoader(Unknown Source)
at sun.applet.AppletPanel.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


This error appeared in apache log:

[Fri Mar 27 01:53:10 2009] [error] [client 192.168.4.70] client denied by server configuration: $OA_JAVA/oracle/apps/fnd/util/ParameterList.class

Most probably the certificate was corrupt.

So I created a new certificate by giving the command:

adjkey -initialize

After that we ran adadmin and force regenerated all jar files.

Problem was resolved after this.

Here's something I found by googling:

The Signed JAR File

When jarsigner is used to sign a JAR file, the output signed JAR file is exactly the same as the input JAR file, except that it has two additional files placed in the META-INF directory:
a signature file, with a .SF extension, and
a signature block file, with a .DSA extension.

The base file names for these two files come from the value of the -sigFile option. For example, if the option appears as
-sigFile MKSIGN
the files are named "MKSIGN.SF" and "MKSIGN.DSA".

If no -sigfile option appears on the command line, the base file name for the .SF and .DSA files will be the first 8 characters of the alias name specified on the command line, all converted to upper case. If the alias name has fewer than 8 characters, the full alias name is used. If the alias name contains any characters that are not legal in a signature file name, each such character is converted to an underscore ("_") character in forming the file name.

The Signature (.SF) File
A signature file (the .SF file) looks similar to the manifest file that is always included in a JAR file when jarsigner is used to sign the file. That is, for each source file included in the JAR file, the .SF file has three lines, just as in the manifest file, listing the following:
the file name,
the name of the digest algorithm used (SHA), and
a SHA digest value.
In the manifest file, the SHA digest value for each source file is the digest (hash) of the binary data in the source file. In the .SF file, on the other hand, the digest value for a given source file is the hash of the three lines in the manifest file for the source file.
The signature file also, by default, includes a header containing a hash of the whole manifest file. The presence of the header enables verification optimization, as described in

JAR File Verification.
The Signature Block (.DSA) File
The .SF file is signed and the signature is placed in the .DSA file. The .DSA file also contains, encoded inside it, the certificate or certificate chain from the keystore which authenticates the public key corresponding to the private key used for signing.

JAR File Verification
A successful JAR file verification occurs if the signature(s) are valid, and none of the files that were in the JAR file when the signatures were generated have been changed since then. JAR file verification involves the following steps:
Verify the signature of the .SF file itself.
That is, the verification ensures that the signature stored in each signature block (.DSA) file was in fact generated using the private key corresponding to the public key whose certificate (or certificate chain) also appears in the .DSA file. It also ensures that the signature is a valid signature of the corresponding signature (.SF) file, and thus the .SF file has not been tampered with.
Verify the digest listed in each entry in the .SF file with each corresponding section in the manifest.

The .SF file by default includes a header containing a hash of the entire manifest file. When the header is present, then the verification can check to see whether or not the hash in the header indeed matches the hash of the manifest file. If that is the case, verification proceeds to the next step.
If that is not the case, a less optimized verification is required to ensure that the hash in each source file information section in the .SF file equals the hash of its corresponding section in the manifest file (seeThe Signature (.SF) File).
One reason the hash of the manifest file that is stored in the .SF file header may not equal the hash of the current manifest file would be because one or more files were added to the JAR file (using the jar tool) after the signature (and thus the .SF file) was generated. When the jar tool is used to add files, the manifest file is changed (sections are added to it for the new files), but the .SF file is not. A verification is still considered successful if none of the files that were in the JAR file when the signature was generated have been changed since then, which is the case if the hashes in the non-header sections of the .SF file equal the hashes of the corresponding sections in the manifest file.
Read each file in the JAR file that has an entry in the .SF file. While reading, compute the file's digest, and then compare the result with the digest for this file in the manifest section. The digests should be the same, or verification fails.
If any serious verification failures occur during the verification process, the process is stopped and a security exception is thrown. It is caught and displayed by jarsigner.

So its possible that we had an issue with the certificate which caused this problem.

How to get the value of environment variables in PL/SQL code

A well coded program avoids hard-coding of paths and uses variables. If you want to get the value of unix environment variables in your programs, you can use the DBMS_SYSTEM.GET_ENV procedure. For applying it in E-Business Suite, you would need to make sure that the variables are defined in RDBMS $ORACLE_HOME/custom$CONTEXT_NAME.env

For example, I defined environment variable CUSTOM_TOP in a new file $ORACLE_HOME/custom$CONTEXT_NAME.env. This file is called automatically by your database environment file $CONTEXT_NAME.env.

To test, whether the value appears simply do this in an sql session:

SQL> set autoprint on
SQL> var CUSTOM_TOP varchar2(255)
SQL> exec dbms_system.get_env('CUSTOM_TOP',:CUSTOM_TOP);

PL/SQL procedure successfully completed.


CUSTOM_TOP
--------------------------------------------------------------------------------
/custom/apps11i

Here's another code snippet:

1 DECLARE
2 v_file UTL_FILE.FILE_TYPE;
3 V_directory VARCHAR2(255);
4 BEGIN
5 dbms_system.get_env('CUSTOM_TOP', v_directory);
6 v_directory := v_directory||'/out';
7 dbms_output.put_line(v_directory);
8 v_file := UTL_FILE.FOPEN(v_directory,'TEST.DAT','W');
9 UTL_FILE.PUT_LINE(v_file,'This is a test file on CUSTOM_TOP');
10 UTL_FILE.FCLOSE(v_file);
11 EXCEPTION
12 WHEN OTHERS
13 THEN
14 DBMS_OUTPUT.PUT_LINE(SQLERRM);
15* END;
SQL> /
/custom/apps11i

PL/SQL procedure successfully completed.

SQL> host ls -ltr /customnp/apps11i/TEST.DAT
-rw-r--r-- 1 oracle dba 13 Mar 23 16:52 /custom/apps11i/TEST.DAT

SQL> host cat /custom/apps11i/TEST.DAT
This is a test file on CUSTOM_TOP

Forms compilation errors parameter.G_query_find

Recently all custom forms compilation failed with errors like these:

FRM-18108: Failed to load the following objects.

Source Module:APPSTAND
Source Object: STANDARD_PC_AND_VA
Source Module:APPSTAND
Source Object: STANDARD_TOOLBAR
Source Module:APPSTAND
Source Object: STANDARD_CALENDAR
Compiling package specification APP_CUSTOM...
No compilation errors.

or

Compiling KEY-NXTBLK trigger on QUERY_FIND data block...
Compilation error on KEY-NXTBLK trigger on QUERY_FIND data block:
PL/SQL ERROR 49 at line 1, column 1
bad bind variable 'parameter.G_query_find'
PL/SQL ERROR 49 at line 1, column 1
bad bind variable 'parameter.G_query_find'
PL/SQL ERROR 49 at line 3, column 1
bad bind variable 'parameter.G_query_find'

It turned out that FORMS60_PATH was not set correctly. Whenever forms are compiled FORMS60_PATH is set to:

FORMS60_PATH=$FORMS60_PATH:$AU_TOP/forms/US:$AU_TOP/resource

In context file it was set as:

$AU_TOP/resource:$AU_TOP/resource/stub

After correcting this and compiling manually it worked. To make this permanent change the s_f60path to include $AU_TOP/forms/US:$AU_TOP/resource

FNDCPSPN fails to relink

We got this error after applying TXK Rollup S:

Starting link of fnd executable 'FNDCPSPN' on Tue Mar 24 23:14:10 EDT 2009
ld -dy $ORACLE_HOME/lib/SC4.2/crti.o
$ORACLE_HOME/lib/SC4.2/crt1.o
$ORACLE_HOME/lib/SC4.2/crtn.o -s -o
$FND_TOP/bin/FNDCPSPN
$FND_TOP/lib/afcpspn.o \
$FND_TOP/lib/libfnd.a -lsql -lclntsh
$ORACLE_HOME/lib/nautab.o $ORACLE_HOME/lib/naeet.o
$ORACLE_HOME/lib/naect.o $ORACLE_HOME/lib/naedhs.o `cat
$ORACLE_HOME/lib/naldflgs` -lnetv2 -lnttcp -lnetwork -lncr -lnetv2
-lnttcp -lnetwork -lclient -lvsn -lcommon -lgeneric -lmm -lnlsrtl3 -lcore4
-lnlsrtl3 -lcore4 -lnlsrtl3 -lnetv2 -lnttcp -lnetwork -lncr -lnetv2 -lnttcp
-lnetwork -lclient -lvsn -lcommon -lgeneric -lepc -lnlsrtl3 -lcore4
-lnlsrtl3 -lcore4 -lnlsrtl3 -lclient -lvsn -lcommon -lgeneric -lnlsrtl3
-lcore4 -lnlsrtl3 -lcore4 -lnlsrtl3 `cat
$ORACLE_HOME/lib/sysliblist` -R
/opt/SUNWcluster/lib:$ORACLE_HOME/lib -Y
P,$ORACLE_HOME/lib::$ORACLE_HOME/network/jre11/lib/sparc/n
ative_threads:$CZ_TOP/bin:/usr/dt/lib:/usr/openwin/lib
:/opt/SUNWcluster/lib:/usr/ccs/lib:/usr/lib -Qy -lc -laio -lm
$ORACLE_HOME/rdbms/lib/defopt.o
$ORACLE_HOME/rdbms/lib/ssbbded.o
Undefined first referenced
symbol in file
FDCPSPN
$FND_TOP/lib/afcpspn.o
ld: fatal: Symbol referencing errors. No output written to
$FND_TOP/bin/FNDCPSPN
*** Error code 1
make: Fatal error: Command failed for target
`$FND_TOP/bin/FNDCPSPN'
Done with link of fnd executable 'FNDCPSPN' on Tue Mar 24 23:14:10 EDT 2009

Relink of module "FNDCPSPN" failed.
See error messages above (also recorded in log file) for possible
reasons for the failure. Also, please check that the Unix userid
running adrelink has read, write, and execute permissions
on the directory $FND_TOP/bin,
and that there is sufficient space remaining on the disk partition
containing your Oracle Applications installation.

Metalink Note: 417535.1 has the solution:

Cause

$FND_TOP/lib/libfnd.a does not include fdcpspn.lc


It sounds like a problem with the environment being used, not a problem with the patch. Patch 4905678 does not bring in afcpspn.lc, but somehow the file already exists on the system, and fdcpspn.lc does not exist in $FND_TOP/lib/libfnd.a. Any patch that tries to relink with afcpspn.lc and fdcpspn.lc on this system will fail.

A similare issue is described in INTERNAL, not visible to customers, Bug 4115814 (92) CERT 4030688 CERTIFICATION OF AFMDMSG.LCT FAILED ON RELINKING FNDCPSPN
You have to check $FND_TOP/lib/libfnd.a does not include the fdcpspn.o

Solution

To implement the solution, please execute the following steps:

1. Please take a backup of $FND_TOP/lib/ directory

2. adrelink.sh force=y ranlib=y "fnd FNDCPSPN"
and check if the adrelink is OK (exiting with status 0)

3. If failure at step 2:

4. cd $FND_TOP/lob
5. ar rvl libfnd.a fdcpspn.o
6. adrelink.sh force=y ranlib=y "fnd FNDCPSPN"

TEMP tablespace grows continuously

Recently we had the problem of TEMP tablespace growing uncontrollably in a Production instance:

Following WARNING was appearing in DB alert log:

WARNING: Detected too many memory locking problems.
WARNING: Performance degradation may occur.
Tue Mar 24 23:19:49 2009
Incremental checkpoint up to RBA [0x3b2.800e.0], current log tail at RBA [0x3b2.872c.0]
Tue Mar 24 23:34:35 2009
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
Tue Mar 24 23:39:53 2009
Incremental checkpoint up to RBA [0x3b2.9185.0], current log tail at RBA [0x3b2.9511.0]
Tue Mar 24 23:59:58 2009
Incremental checkpoint up to RBA [0x3b2.a21f.0], current log tail at RBA [0x3b2.a57f.0]
Wed Mar 25 00:20:02 2009
Incremental checkpoint up to RBA [0x3b2.c6fc.0], current log tail at RBA [0x3b2.ce49.0]

This is very similar to the issue mentioned in Metalink Note 357765.1

Cause is Incorrect statistics on FIXED objects

Bad execution plan needed much sort-space.
Multiple waitevents on 'direct path write temp'.

Fix: As per Metalink Note 357765.1

SQL> exec dbms_stats.gather_fixed_objects_stats;

You need to run the above command, anytime you change init.ora parameters.

Advanced options like duplex printing or tray selections

I always wondered how to use the advanced options in printers like duplex printing, tray selection etc. from E-Business Suite.

Metalink Note 394130.1 How To Enable Printer Options Such As Duplex Printing Or Tray Selections For Bitmap Concurrent Programs.

Pasta and XML Publisher does not have the ability to enable printer options such as duplex printing or printer tray selection for a bitmap concurrent program.

Oracle has advised contacting Adobe or the printer vendor for a PPD (Postscript Printer Definition) file that supports the desired function or features associated with the make and model of printer being used.

Refer to metalink note 394130.1 for more details.

oracle.apps.fnd.common.PoolException

Sandhya reported this error to me.  The clone was done through OEM Grid Applications Pack.  AppsLocalLogin.jsp page showed this error:

oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException

A check through jserv.log and mod_jserv.log showed these errors:

[19/03/2009 11:10:23:091 EDT] oracle.apps.mwa = $MWA_TOP
Exception in static block of jtf.cache.appsimpl.AppsCacheLogger. Stack trace is: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException:
Exception creating new Poolable object.
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1509)
at oracle.apps.fnd.profiles.Profiles.getProfile(Profiles.java:362)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfileFromDB(ExtendedProfileStore.java:210)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getSpecificProfile(ExtendedProfileStore.java:169)
at oracle.apps.fnd.profiles.ExtendedProfileStore.getProfile(ExtendedProfileStore.java:148)
at oracle.apps.fnd.common.logging.DebugEventManager.configureUsingDatabaseValues(DebugEventManager.java:1201)
at oracle.apps.fnd.common.logging.DebugEventManager.configureLogging(DebugEventManager.java:1044)
at oracle.apps.fnd.common.logging.DebugEventManager.internalReinit(DebugEventManager.java:1013)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:980)
at oracle.apps.fnd.common.logging.DebugEventManager.reInitialize(DebugEventManager.java:967)
at oracle.apps.fnd.common.AppsLog.reInitialize(AppsLog.java:570)
at oracle.apps.fnd.common.AppsContext.initLog(AppsContext.java:987)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:964)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.fnd.common.WebAppsContext.(WebAppsContext.java:1002)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContextWithDBCFile(ArchitectureWrapper.java:143)
at oracle.apps.jtf.cache.ArchitectureWrapper.createDefaultAppsContext(ArchitectureWrapper.java:107)
at oracle.apps.jtf.cache.ArchitectureWrapper.createAppsContext(ArchitectureWrapper.java:96)
at oracle.apps.jtf.cache.appsimpl.AppsCacheLogger.(AppsCacheLogger.java:43)
at oracle.apps.jtf.cache.appsimpl.AppsCacheEnvironment.getCacheLogger(AppsCacheEnvironment.java:67)
at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:711)
at oracle.apps.jtf.cache.CacheManager.(CacheManager.java:378)
at oracle.apps.fnd.cache.Cache.setCacheFullName(Cache.java:228)
at oracle.apps.fnd.cache.Cache.initCache(Cache.java:114)
at oracle.apps.fnd.cache.Cache.(Cache.java:89)
at oracle.apps.fnd.cache.AppsCache.(AppsCache.java:86)
at oracle.apps.fnd.cache.AolCaches.getCache(AolCaches.java:155)
at oracle.apps.fnd.profiles.Profiles.(Profiles.java:241)
at oracle.apps.fnd.profiles.ExtendedProfileStore.init(ExtendedProfileStore.java:498)
at oracle.apps.fnd.profiles.ExtendedProfileStore.(ExtendedProfileStore.java:119)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at oracle.apps.fnd.common.AppsContext.instantiateProfileStore(AppsContext.java:4312)
at oracle.apps.fnd.common.AppsContext.makeProfileStore(AppsContext.java:1171)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:970)
at oracle.apps.fnd.common.Context.setProfileStore(Context.java:952)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:958)
at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:918)
at oracle.apps.fnd.common.AppsContext.(AppsContext.java:752)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.java:175)
at oracle.apps.mwa.container.ApplicationsObjectLibrary.(ApplicationsObjectLibrary.java:79)
at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:339)
at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)
at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)
at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)
at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)
at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)
at org.apache.jserv.JServ.start(JServ.java:625)
at org.apache.jserv.JServ.main(JServ.java:234)
Caused by: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
at oracle.apps.fnd.common.Pool.createObject(Pool.java:1321)
at oracle.apps.fnd.common.Pool.borrowObject(Pool.java:1062)
at oracle.apps.fnd.security.DBConnObjPool.borrowObject(DBConnObjPool.java:752)
at oracle.apps.fnd.security.AppsConnectionManager.borrowConnection(AppsConnectionManager.java:256)
at oracle.apps.fnd.common.Context.borrowConnection(Context.java:1951)
at oracle.apps.fnd.common.AppsContext.getPrivateConnectionFinal(AppsContext.java:2617)
at oracle.apps.fnd.common.AppsContext.getPrivateConnection(AppsContext.java:2550)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2408)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2212)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2126)
at oracle.apps.fnd.common.AppsContext.getJDBCConnection(AppsContext.java:2139)
at oracle.apps.fnd.common.Context.getJDBCConnection(Context.java:1682)
at oracle.apps.fnd.cache.GenericCacheLoader.load(GenericCacheLoader.java:168)
at oracle.apps.fnd.profiles.Profiles.getProfileOption(Profiles.java:1500)
... 51 more
Caused by: oracle.apps.fnd.common.AppsException: java.sql.SQLException: ORA-01017: invalid username/password; logon denied

I checked the Guest user password in dbc file inside $FND_SECURE. It was correct.
I checked the value of profile option GUEST_USER_PASSWORD. It was correct.
I checked whether apps was able to decrypt the guest user password by using the apps password. This failed.

Incidentally all the ad startup and shutdown were also erroring out with these errors:

Cannot complete applications logon. You may have entered an invalid applications password.

I checked whether apps was able to decrypt the guest user password by using the Production apps password. That worked.

So on a hunch I changed the passwords of APPS and APPLSYS back to Production password.

alter user apps identified by prodpass;
alter user applsys identified by prodpass;

It worked. No more errors.

Then I shutdown all services and did FNDCPASS to change APPS password.

I got a lot of errors like this one:

FNDCPASS was not able to decrypt password for SYSADMIN during applsys password change.
FNDCPASS was not able to decrypt password for GUEST during applsys password change.
FNDCPASS was not able to decrypt password for JOHNDOE during applsys password change.

I checked the encryption again and now it was working with new apps password. So I changed the APPS and APPLSYS password to new password manually:

alter user apps identified by clonepass;
alter user applsys identified by clonepass;

Again it worked. No errors. All services are up and no errors were found in sanity checks. I have advised the DBAs to try FNDCPASS once again in future just before they reclone it after 1 month. Metalink Notes 459601.1 and 454299.1 have some pointers for this FNDCPASS error.

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

This error was reported after clone:

JSP Error:

---

Request URI:/OA_HTML/AppsLocalLogin.jsp

Exception:

java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

---

jserv.log showed these errors:

[19/03/2009 11:29:13:045 EDT] oracle.apps.mwa = $MWA_TOP

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed. The

Distributed Caching System failed to initialize on port: 12442. The list of hosts in the distributed caching system is: 11ioltp.justanexample.com . The port 12442 should be free on each host running the JVMs. The default port 12442 can be overridden using -Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=<

port number>

Metalink Note 461872.1 describes this problem.  This occurs when your java object cache port can not be contacted.  s_java_object_cache_port is the context file variable which needs to be changed.  If you would like to avoid running autoconfig, you can change the value of this variable jserv.properties and jserv_restrict.properties.  We changed it from 12442 to 12443:

jserv.properties:wrapper.bin.parameters=-Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=12443

jserv_restrict.properties:wrapper.bin.parameters=-Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=12443

Change the port number.

Bounce Apache

Retry.

It worked for us.

Solaris 10 pre-requisites for Oracle Database 11g

Metalink Note 743042.1 has the Requirements for Installing Oracle 11gR1 RDBMS on Solaris 10 SPARC 64-bit.

The notable among these is kernel patch 137111-04.

There is a post on Sun blogs which describes Solaris 10 kernel patch id sequence.  If you follow the obsolescence path of this patch, the curret patch which is not yet obsolete is 137137-09.  So you should apply 137137-09 or higher to your Solaris 10 OS, before you can consider moving to 11g.

VCS Agent Exit Codes

VCS QuickStart uses processes called Agents to start, stop, and monitor resources. Agentsmonitor all resources continually to validate both online and offline resource status.VCS QS Custom Application agents use the following programs to manage an application

Start Program – a script or program which starts the application

Stop Program – a script or program which gracefully stops the application

Monitor Program – a script or program which monitors the application

Clean Program – a script or program which kills or forcibly stops the application

The VCS QS Monitor Program uses the following exit values to communicate status:

100 – OFFLINE

110 – ONLINE

99 – UNKNOWN

afdbprf.sh fails with ORA-12899: value too large for column

Vickie faced this error while running adconfig on a newly upgraded 10.2.0.4 home:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/03122349/adconfig.log

In the autoconfig log $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/03122349/adconfig.log shows:

afdbprf.sh started at Thu Mar 12 23:49:43 EDT 2009
Executable : $ORACLE_HOME/bin/sqlplus


SQL*Plus: Release 10.2.0.4.0 - Production on Thu Mar 12 23:49:44 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Enter value for 1: Enter value for 2: Enter value for 3: Connected.
Updated profile option value - 1 row(s) updated
Application Id : 0
Profile Name : FND_DB_WALLET_DIR
Level Id : 10001
New Value : $ORACLE_HOME/appsutil/wallet
Old Value : $9.2.0_ORACLE_HOME/appsutil/wallet
Updated profile option value - 1 row(s) updated
Application Id : 174
Profile Name : ECX_UTL_XSLT_DIR
Level Id : 10001
New Value : /usr/tmp
Old Value : /usr/tmp
Updated profile option value - 1 row(s) updated
Application Id : 174
Profile Name : ECX_UTL_LOG_DIR
Level Id : 10001
New Value : /usr/tmp
Old Value : /usr/tmp
Updated profile option value - 1 row(s) updated
Application Id : 0
Profile Name : BIS_DEBUG_LOG_DIRECTORY
Level Id : 10001
New Value : /usr/tmp
Old Value : /usr/tmp
declare
*
ERROR at line 1:
ORA-12899: value too large for column
"APPLSYS"."FND_PROFILE_OPTION_VALUES"."PROFILE_OPTION_VALUE" (actual: 480,
maximum: 240)
ORA-06512: at line 44
ORA-06512: at line 139


Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64
bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ERRORCODE = 1 ERRORCODE_END
.end std out.

.end err out.
****************************************************
Hope this helps!



Metalink Note 458511.1 gives a solution:

Cause

Script afdbprf.sql called by afdbprf.sh is having the following included which caused the issue:

-- Set up UTL_FILE_LOG profile option
--
set_profile(1, 'UTL_FILE_LOG',
10001, 0,
'%s_db_util_filedir%',
NULL);

As we see in the above code the UTL_FILE_LOG profile option is filled with the value of the s_db_util_filedir

The value of utl_file_dir is over 240 characters , FND_PROFILE_OPTION_VALUES:PROFILE_OPTION_VALUE varchar2(240) is limited to 240 characters.

According to Bug 6404909 the FND_PROFILE_OPTION_VALUES:PROFILE_OPTION_VALUE varchar2(240) can not be (easily) changed because there are many, many locations within C-code user exits and other places in which a hardcoded value of 240 for this column exist.
Solution

To implement the solution, please execute the following steps:

1 - Change the value dbutilfiledir in the database context file ($ORACLE_HOME/appsutil/$CONTEXT_NAME.xml) to a value less the 240 characters

2 - run autoconfig again.

This solved the problem.

Results of dba_registry and utlu102s.sql differ

After a roller coaster upgrade from 9.2.0.8 to 10.2.0.4, I finally achieved this:

select comp_name,version,status
from dba_registry

SQL> /
Oracle Database Catalog Views
10.2.0.4.0 VALID

Oracle Database Packages and Types
10.2.0.4.0 VALID

JServer JAVA Virtual Machine
10.2.0.4.0 VALID

Oracle Database Java Packages
10.2.0.4.0 VALID

Oracle XDK
10.2.0.4.0 VALID

Oracle Text
10.2.0.4.0 VALID

Oracle Real Application Clusters
10.2.0.4.0 INVALID

Spatial
10.2.0.4.0 VALID

Oracle XML Database
10.2.0.4.0 VALID

Oracle interMedia
10.2.0.4.0 VALID


10 rows selected.

However utlu102s.sql still gives Oracle Database Server status as INVALID:

SQL> @utlu102s.sql
.
Oracle Database 10.2 Upgrade Status Utility 03-12-2009 18:21:06
.
Component Status Version HH:MM:SS
Oracle Database Server INVALID 10.2.0.4.0 01:01:55
JServer JAVA Virtual Machine VALID 10.2.0.4.0 00:15:19
Oracle XDK VALID 10.2.0.4.0 00:11:22
Oracle Database Java Packages VALID 10.2.0.4.0 00:00:47
Oracle Text VALID 10.2.0.4.0 00:02:42
Oracle XML Database VALID 10.2.0.4.0 00:03:17
Oracle Real Application Clusters INVALID 10.2.0.4.0 00:00:03
Oracle interMedia VALID 10.2.0.4.0 00:06:36
Spatial VALID 10.2.0.4.0 00:06:56
.
Total Upgrade Time: 02:16:51

PL/SQL procedure successfully completed.

Metalink Note 456845.1 describes this issue. However I went a little deeper and figured out a way to correct this:

utlu102s.sql reads results from dba_registry_log. A dbms_metadata query on dba_registry_log shows this:

SQL> select dbms_metadata.get_ddl('VIEW','DBA_REGISTRY_LOG','SYS') FROM DUAL;

DBMS_METADATA.GET_DDL('VIEW','DBA_REGISTRY_LOG','SYS')
--------------------------------------------------------------------------------
CREATE OR REPLACE FORCE VIEW "SYS"."DBA_REGISTRY_LOG" ("OPTIME", "NAMESPACE",


SQL> SET LONG2000
SQL> /

DBMS_METADATA.GET_DDL('VIEW','DBA_REGISTRY_LOG','SYS')
--------------------------------------------------------------------------------

CREATE OR REPLACE FORCE VIEW "SYS"."DBA_REGISTRY_LOG" ("OPTIME", "NAMESPACE",
"COMP_ID", "OPERATION", "MESSAGE") AS
SELECT optime,
namespace, cid,
DECODE(operation, 0, 'INVALID',
1, 'VALID',
2, 'LOADING',
3, 'LOADED',
4, 'UPGRADING',
5, 'UPGRADED',
6, 'DOWNGRADING',
7, 'DOWNGRADED',
8, 'REMOVING',
9, 'OPTION OFF',
10, 'NO SCRIPT',
99, 'REMOVED',
100, 'ERROR',
NULL),
errmsg
FROM registry$log

SQL>

So the view dba_registry_log is actually based on registry$log table.

SQL> desc registry$log
Name Null? Type
----------------------------------------- -------- ----------------------------
CID VARCHAR2(30)
NAMESPACE VARCHAR2(30)
OPERATION NOT NULL NUMBER
OPTIME TIMESTAMP(6)
ERRMSG VARCHAR2(1000)

SQL> select cid,operation from registry$log;
SQL> /
UPGRD_BGN -1
JAVAVM 1
CATPROC 1
RDBMS 1
XML 1
CATJAVA 1
CONTEXT 1
XDB 1
RAC 0
ORDIM 1
SDO 1
UPGRD_END -1
UTLRP_BGN -1
UTLRP_BGN -1
UTLRP_END -1
UTLRP_BGN -1
UTLRP_END -1
UTLRP_BGN -1
UTLRP_END -1
UTLRP_BGN -1
UTLRP_END -1
UTLRP_BGN -1
UTLRP_END -1

23 rows selected.

It is clear from above that 1 is the code for VALID and 0 is the code for INVALID.

So I passed this update statement.

SQL> update registry$log
2 set operation=1
3 where cid in ('CATPROC','RDBMS');

2 rows updated.

SQL> commit;

Commit complete.

SQL> @?/rdbms/admin/utlu102s.sql
.
Oracle Database 10.2 Upgrade Status Utility 03-12-2009 18:29:19
.
Component Status Version HH:MM:SS
Oracle Database Server VALID 10.2.0.4.0 01:01:55
JServer JAVA Virtual Machine VALID 10.2.0.4.0 00:15:19
Oracle XDK VALID 10.2.0.4.0 00:11:22
Oracle Database Java Packages VALID 10.2.0.4.0 00:00:47
Oracle Text VALID 10.2.0.4.0 00:02:42
Oracle XML Database VALID 10.2.0.4.0 00:03:17
Oracle Real Application Clusters INVALID 10.2.0.4.0 00:00:03
Oracle interMedia VALID 10.2.0.4.0 00:06:36
Spatial VALID 10.2.0.4.0 00:06:56
.
Total Upgrade Time: 02:16:51

PL/SQL procedure successfully completed.

SQL>

Metalink down but classic metalink available

New Metalink is down today.  Following message comes:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, you@your.address and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

However we can access the classic metalink at https://metalink2.oracle.com

Classic Metalink is going to be retired by the end of this year.

Couldn't set locale correctly

Srinivas pinged me about this error which comes when you login as applmgr on a server:

couldn't set locale correctly

I checked the $HOME/.profile file of the user and found an environment variable LANG:

LANG=/usr/lib/locale/en_US.UTF-8
export LANG

On manually setting this variable, we get the same error.

$ LANG=/usr/lib/locale/en_US.UTF-8
couldn't set locale correctly

AventX support had asked the DBAs to set this environment variable, as they were unable to register the license key.

The error was coming because the file en_US.UTF-8 didn't exist:

$ ls -ld /usr/lib/locale/en_US.UTF-8
/usr/lib/locale/en_US.UTF-8: No such file or directory

$ cd /usr/lib/locale
$ ls -ltr
total 52
-rw-r--r-- 1 root bin 1848 Dec 8 2004 lcttab
-rw-r--r-- 1 root bin 270 Dec 8 2004 geo
drwxr-xr-x 8 root bin 512 Mar 26 2008 C
lrwxrwxrwx 1 root root 3 Mar 26 2008 POSIX -> ./C
drwxr-xr-x 3 root bin 512 Mar 26 2008 iso_8859_15
drwxr-xr-x 4 root bin 512 Mar 26 2008 iso_8859_1
drwxr-xr-x 5 root bin 512 Mar 26 2008 common
drwxr-xr-x 3 root bin 512 Mar 26 2008 en.UTF-8
drwxr-xr-x 3 root bin 512 Mar 26 2008 en_CA
drwxr-xr-x 9 root bin 512 Mar 26 2008 en_CA.ISO8859-1
drwxr-xr-x 10 root bin 512 Mar 26 2008 en_CA.UTF-8
drwxr-xr-x 3 root bin 512 Mar 26 2008 en_US
drwxr-xr-x 9 root bin 512 Mar 26 2008 en_US.ISO8859-1
drwxr-xr-x 9 root bin 512 Mar 26 2008 en_US.ISO8859-15
drwxr-xr-x 3 root bin 512 Mar 26 2008 en_US.ISO8859-15@euro
drwxr-xr-x 3 root bin 512 Mar 26 2008 es
drwxr-xr-x 3 root root 512 Mar 26 2008 es.UTF-8
drwxr-xr-x 3 root bin 512 Mar 26 2008 es_MX
drwxr-xr-x 8 root bin 512 Mar 26 2008 es_MX.ISO8859-1
drwxr-xr-x 9 root bin 512 Mar 26 2008 es_MX.UTF-8
drwxr-xr-x 3 root bin 512 Mar 26 2008 fr
drwxr-xr-x 3 root bin 512 Mar 26 2008 fr.UTF-8
drwxr-xr-x 3 root bin 512 Mar 26 2008 fr_CA
drwxr-xr-x 8 root bin 512 Mar 26 2008 fr_CA.ISO8859-1
drwxr-xr-x 9 root bin 512 Mar 26 2008 fr_CA.UTF-8

I'll update this post once I learn more.

Relaying denied revisited

Recently I was called about complaints of workflow mailer failing to send outgoing mails, after the database server failed over. We use the sendmail daemon of the database server as the mail server. On checking the /etc/hosts file of the server, I found that it did not have the entries for the application tier servers. Sendmail always checks if a host which is trying to send mail is present in the mail server's /etc/hosts file. After adding the servers, relaying denied was resloved.

As discussed in a previous post, Sendmail does these checks:

1. Checks whether the host trying to send mail is in /etc/hosts
2. Does a reverse DNS lookup on the IP of the host trying to send mail, to see if the name is same as that reported by the host. For example if the host reports its name as appserver.justanexample.com (based on /etc/hosts of mail server), but a reverse DNS lookup shows that the name of the server is appserver.dev.justanexample.com, then Sendmail will reject it with Relay Denied error

CLASSPATH and AF_CLASSPATH after JDK 6

If you are on JDK 5 or 6, to avoid some weird problems in Oracle Apps, ensure that your CLASSPATH and AF_CLASSPATH includes the necessary JDK 6 libraries:

[JDK60_TOP]/lib/dt.jar,
[JDK60_TOP]/lib/tools.jar,
[JDK60_TOP]/jre/lib/rt.jar, and
[JDK60_TOP]/jre/lib/charsets.jar

where JDK60_TOP is $COMMON_TOP/util/jdk1.6.0_xx

Metalink Note 184714.1 has some pointers for troubleshooting CLASSPATH issues.