MSCNPS64 Program was terminated by signal 11 etc

Maran reported this error in ASCP after a fresh clone:

Memory Based Snapshot 64 bit Sun –program errors out in apps11i.

Error Details:

$MSC_TOP/bin/MSCNPS64
Program was terminated by signal 11

Based on previous experience, I checked the values of these environment variables:

APS64_ORACLE_HOME
APS64_ORA_NLS33
ORA_NLS33

All of them were sourced in.

However Oracle recommends that only APS64_ORACLE_HOME and APS64_ORA_NLS33 needs to be set. So I removed ORA_NLS33 and bounced the Concurrent Manager with only these set:

APS64_ORACLE_HOME
APS64_ORA_NLS33

The ASCP Plan crossed the stage where MSCNPS64 is called. However it failed during Loader Worker with this error:

APP-MRP-22130: Cannot connect to database

Cause: The current routine cannot connect to the database.

Action: Contact your system administrator or customer support representative.

Concurrent program returned no reason for failure.
ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on
.

The SQL statement being executed at the time of the error was: and was executed from the file .
$MSC_TOP/bin/MSCPLD.sh
Program exited with status 1

So this time, I removed APS64_ORA_NLS33 and set ORA_NLS33, which is how it runs in Production:

APS64_ORACLE_HOME
ORA_NLS33

The plan crossed the Loader worker stage but failed during Memory Based Planner MSONWS64:

Memory Based Planner 64-bit Sun
Program exited with status 1.

We have faced this error before when we ran out of RAM and swap. However we were on a new server with 62GB free memory. So lack of memory could not be the issue.

I checked the value of ulimit for open file descriptors and found:

ulimit -n
65536

On Production this value was very low 256.

655536 is the maximum limit. However I have seen issues when it is set to maximum, so I reduced it by 2.

ulimit -n 65534

When Plan was relaunched, it ran fine.

Maran ran it again to make sure the fix really worked.

64-bit binaries of ASCP

All E-Business Suite Application binaries are 32-bit. The only exception are ASCP Memory Based Snapshot programs: MSCNWS64.exe and MSONWS64.exe. These are 64-bit binaries. They have been made 64-bit because of their memory intensive nature.

I noticed that whenever we run adadmin to relink binaries of MSC and MSO, these two are never relinked. If you try to relink them manually through adrelink.sh force=y "MSC MSCNWS64.exe" it simply deletes the file and doesn't relink. I asked Oracle about this in an SR and this was their answer:

QUESTION
=========
How to relink MSCNWS64.exe and MSONWS64.exe manually and through adadmin ? We have seen that these files are not relinked if we relink all MSC and MSO binaries through adadmin

ANSWER
=======
Hi Vikram,

Recompiling the 64 bit Memory based planner executables that you are asking about is not required and doesn't work as the code lines are all self contained within that executable and it doesn't use other files in other directories like other executables for recompiling. So basically if an install or patch is applied this executable is fully self contained in similar fashion as if you were running a Netscape browser which is also an executable netscape.exe - so these executables do not need to be compiled/recompiled.