MSCNPS64 Program was terminated by signal 11 etc

Maran reported this error in ASCP after a fresh clone:

Memory Based Snapshot 64 bit Sun –program errors out in apps11i.

Error Details:

$MSC_TOP/bin/MSCNPS64
Program was terminated by signal 11

Based on previous experience, I checked the values of these environment variables:

APS64_ORACLE_HOME
APS64_ORA_NLS33
ORA_NLS33

All of them were sourced in.

However Oracle recommends that only APS64_ORACLE_HOME and APS64_ORA_NLS33 needs to be set. So I removed ORA_NLS33 and bounced the Concurrent Manager with only these set:

APS64_ORACLE_HOME
APS64_ORA_NLS33

The ASCP Plan crossed the stage where MSCNPS64 is called. However it failed during Loader Worker with this error:

APP-MRP-22130: Cannot connect to database

Cause: The current routine cannot connect to the database.

Action: Contact your system administrator or customer support representative.

Concurrent program returned no reason for failure.
ORACLE error 1012 in FDPCLS

Cause: FDPCLS failed due to ORA-01012: not logged on
.

The SQL statement being executed at the time of the error was: and was executed from the file .
$MSC_TOP/bin/MSCPLD.sh
Program exited with status 1

So this time, I removed APS64_ORA_NLS33 and set ORA_NLS33, which is how it runs in Production:

APS64_ORACLE_HOME
ORA_NLS33

The plan crossed the Loader worker stage but failed during Memory Based Planner MSONWS64:

Memory Based Planner 64-bit Sun
Program exited with status 1.

We have faced this error before when we ran out of RAM and swap. However we were on a new server with 62GB free memory. So lack of memory could not be the issue.

I checked the value of ulimit for open file descriptors and found:

ulimit -n
65536

On Production this value was very low 256.

655536 is the maximum limit. However I have seen issues when it is set to maximum, so I reduced it by 2.

ulimit -n 65534

When Plan was relaunched, it ran fine.

Maran ran it again to make sure the fix really worked.