Apps binaries write core files in /var/crash/cores

Ever since we migrated to Solaris 10, we have had various concurrent program binaries creating core dumps. Some of them get resolved after we relink the binaries. But some continue core dumps even after relinking them. ARGLTP or the AR to GL Transfer Program is one of the binaries which does this. I have had an SR logged with Oracle for some time. The strangest thing is that the concurrent requests calling ARGLTP complete normal. But for every run of ARGLTP a coredump with signal 11 is created. Here's a stack trace:

adb /erp11i/erp/appl/ar/11.5.0/bin/ARGLTP
1208446806-ARGLTP-15767-62984-54011-tsgsd1009-sun4u
core file = 1208446806-ARGLTP-15767-62984-54011-tsgsd1009-sun4u -- program
``/erp11i/erp/appl/ar/11.5.0/bin/ARGLTP
'' on platform SUNW,Netra-T12
SIGSEGV: Segmentation Fault
$c
libc.so.1`strcmp+0x170(1739c8, 73657364, 0, 0, 1d0634, 1d0614)
afppre+0x1cc(2a81e0, 261b88, 50, 50, ff1ea2b4, ff1f29f8)
fdpcls+0x3e4(4e, 1c2b28, 1c2b18, 1c2800, 1c2800, 1)
main+0x76e0(14f730, 14f774, ffbf9fbc, a, 2a4760, 7f80)
_start+0xdc(0, 0, 0, 0, 0, 0)
$q

A common thing in all the coring binaries is the OS library libc.so.1. Oracle has asked us to check with Sun if this is a known issue and if a new version of libc.so.1 is available. I have an SR logged with Sun also for this.

We finally got a solution. Patch 6815663 is a post ATG RUP6 one off patch which fixes this issue. Since it is not fully regression tested, it is protected by a password. You have to log an SR to get the password. After applying this patch, we have seen that none of the binaries like ARGLTP, PARGDR, PAVDVC, PASGLT etc. are coring anymore.

Apps binaries coring after cloning instance on Solaris 8 to Solaris 10

After cloning an instance which is on Solaris 8 to an instance which is on Solaris 10, various binaries are coredumping and creating huge core files in /var/crash/cores:

# ls -lh
total 22662940
-rw------- 1 root root 11M Feb 29 17:53 1204325626-ARGLTP-10853-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 17:54 1204325682-ALECDC-14421-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 18:16 1204326978-ALECDC-22443-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 18:37 1204328275-ALECDC-1648-66128-54055-devserver-sun4u
-rw------- 1 root root 11M Feb 29 18:53 1204329219-ARGLTP-24665-66128-54055-devserver-sun4u
-rw------- 1 root root 19M Feb 29 18:55 1204329356-PAVDVC-2528-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 18:59 1204329554-ALECDC-16012-66128-54055-devserver-sun4u
-rw------- 1 root root 14M Feb 29 19:04 1204329851-PARGDR-11673-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 19:20 1204330841-ALECDC-28963-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 19:42 1204332128-ALECDC-11910-66128-54055-devserver-sun4u
-rw------- 1 root root 11M Feb 29 19:53 1204332829-ARGLTP-20966-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 20:03 1204333423-ALECDC-24081-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 20:24 1204334694-ALECDC-4385-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 20:46 1204335982-ALECDC-13323-66128-54055-devserver-sun4u
-rw------- 1 root root 11M Feb 29 20:54 1204336441-ARGLTP-6842-66128-54055-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 21:07 1204337250-ALECDC-20612-66128-54055-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:21 1204338103-MSCSDW-10817-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:22 1204338132-MSCSDW-10747-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:22 1204338177-MSCSLD.new-10840-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:24 1204338281-MSCSLD.new-21677-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:24 1204338285-MSCSLD.new-22151-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:27 1204338427-MSCSLD.new-27500-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:27 1204338428-MSCSLD.new-28456-63904-63988-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:28 1204338491-MSCSDW-10741-63904-63988-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 21:29 1204338539-ALECDC-4860-66128-54055-devserver-sun4u
-rw------- 1 root root 35M Feb 29 21:31 1204338662-MSCSDW-10748-63904-63988-devserver-sun4u
-rw------- 1 root root 9.9M Feb 29 21:50 1204339826-ALECDC-19716-66128-54055-devserver-sun4u
-rw------- 1 root root 11M Feb 29 21:53 1204340029-ARGLTP-236-66128-54055-devserver-sun4u
-rw------- 1 root root 35M Feb 29 22:00 1204340414-MSCSLD.new-22508-63904-63988-devserver-sun4u

I tried executing these binaries from command line to see if they crashed, but they worked fine:

# PASGLT

Usage: Example:
PASGLT PASGLT
applsys/fnd
0 0
Y Y
07-JUN-91
10001
7
18:56:00
12-AUG-91
Y
[] Y

Note:
Batch name if blank ("") means all elegible batches.
All reschedule information can be blank.

ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on.

The SQL statement being executed at the time of the error was: and was executed from the file .


# $ALR_TOP/bin/ALECDC
You have not provided the required arguments for this program.
Usage: ALECDC 0 Y
appst117@tsgsd1007 # $AR_TOP/bin/ARGLTP
Usage: /ebiz11i/erpapp/appl/ar/11.5.0/bin/ARGLTP oracleid/password 0 Y
start_date(Canonical) post_thru_date(Canonical) gl_posted_date(Canonical) Canonical=YYYY/MM/DD HH24:MI:SS
report_only(Y|N) summary(Y|N) run_je_imp(Y|N) day_interval(POSITIVE INTEGER)
posting_control_id [debug_flag(Y|N|I)]
ARGLTP: Too few arguments.
Program exited with error.
ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on.

The SQL statement being executed at the time of the error was: and was executed from the file .
appst117@tsgsd1007 # $PA_TOP/bin/PAVDVC
Wrong number of parameters (0) received.

Usage: Example:
applsys/fnd
0 0
Y Y
4323
12-MAR-91
7
18:56:00
12-AUG-91
Y
[] Y

pavini() returned error
/ebiz11i/erpapp/appl/pa/11.5.0/bin/PAVDVC /ebiz11i/erpapp/appl/pa/11.5.0/bin/PAVDVC
Program exited with error.
ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on.

The SQL statement being executed at the time of the error was: and was executed from the file .

# $PA_TOP/bin/PARGDR
Failed to get arguments
Program exited with error.
ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on.

The SQL statement being executed at the time of the error was: and was executed from the file .

All the coredumping binaries belong to a single instance. I have suggested to the DBAs to relink all binaries through adadmin and see if the cores stop getting created. 

Metalink Note 456008.1 has information about relinkning ALECDC:

adrelink.sh force=y ranlib=y link_debug='Y' "alr ALECDC"

Metalink Note 467886.1 has information about relinking XDPCTRLS:

adrelink.sh force=y ranlib=y "xdp XDPCTRLS"