MSONWS64.exe Signal 11 error in ASCP

After applying Jan - Oct 2008 critical patch update, ASCP plan failed with Signal 11 error on MSONWS64.exe.

In past experience, we have encountered this error when APS64_ORACLE_HOME, APS64_ORA_NLS33 and ORA_NLS33 are not set or set incorrectly.

We checked for these variables and found them missing.  The reason why they were missing is that they were present in $CONTEXT_NAME.env file.  When you run autoconfig, these files are regenerated and any manual changes to these files are lost.  All such variables should be set in the customization section of $APPL_TOP/admin/adovars.env to prevent them from disappearing after every autoconfig.

Anand Reddy set them in adovars.env, sourced the environment again and restarted services.  However the signal 11 error came again when plan was launched.  I decided to investigate this and had a hunch that it was due to those environment variables.

env |grep APS64 did not return anything

but echo $APS64_ORACLE_HOME and echo $APS64_ORA_NLS33 returned values.

That was strange.  If you execute $MSO_TOP/bin/MSONWS64.exe on unix prompt this is the expected output:

$ cd $MSO_TOP/bin

$ ./MSONWS64.exe

Oracle Home Set to ORACLE_HOME=$MSO_TOP/bin/nls8174

return value 0

Oracle NLS33 Set to ORA_NLS33=$MSO_TOP/bin/nls8174/ocommon/nls/admin/data

return value 0

Segmentation Fault(coredump)

Yes coredump is the expected output as you are not passing any values to this executable.

However in our instance which was giving signal 11 error the output was:

$ ./MSONWS64.exe

Segmentation Fault(coredump)

I took up an instance on which the expected output was appearing and copied the MSONWS64.exe of that instance to our instance which was erroring out.  Strangely the output remained:

$ ./MSONWS64.exe

Segmentation Fault(coredump)

On the working instance I did this:

unset APS64_ORACLE_HOME

The output became:

$ ./MSONWS64.exe

Oracle NLS33 Set to ORA_NLS33=$MSO_TOP/bin/nls8174/ocommon/nls/admin/data

return value 0

Segmentation Fault(coredump)

Then I did 

unset APS64_ORACLE_NLS33

The output matched our instance's output:

$ ./MSONWS64.exe

Segmentation Fault(coredump)

So my hunch had proved right, somehow these two environment variables APS64_ORACLE_HOME and APS64_ORA_NLS33 were not getting set.

I decided to check the adovars.env file:

# Begin customizations

#

#   Add any custom variables or additional variable settings in this section.

# For example, you may wish to update your PATH to ensure it lists the

# directory containing the "jre" executable, typically $OA_JRE_TOP/bin.

#

# Make sure PATH does not include other versions of the JRE.

# Make sure PATH does not include the JDK.

APS64_USE_EXPORT=NO

APS64_ORACLE_HOME=/erppgfb2/erpapp/appl/mso/11.5.0/bin/nls8174

APS64_ORA_NLS33=/erppgfb2/erpapp/appl/mso/11.5.0//bin/nls8174/ocommon/nls/admin/data

ORA_NLS33=/erppgfb2/erpapp/appl/mso/11.5.0/bin/nls8174/ocommon/nls/admin/data

# End customizations

That looked ok.  But wait, I thought what is the current shell:

$ echo $SHELL

/bin/sh

If the shell is sh, setting an environment variable with equal to notation only, doesn't work.

So if you say:

APS64_ORACLE_HOME=/erppgfb2/erpapp/appl/mso/11.5.0/bin/nls8174

and don't do an

export APS64_ORACLE_HOME

It won't register with the sh shell.  If the shell is ksh, then it can be registered without the export command.

I added the export commands:

# Begin customizations

#

#   Add any custom variables or additional variable settings in this section.

# For example, you may wish to update your PATH to ensure it lists the

# directory containing the "jre" executable, typically $OA_JRE_TOP/bin.

#

# Make sure PATH does not include other versions of the JRE.

# Make sure PATH does not include the JDK.

APS64_USE_EXPORT=NO;export APS64_USE_EXPORT

APS64_ORACLE_HOME=$MSO_TOP/bin/nls8174;export APS64_ORACLE_HOME

APS64_ORA_NLS33=$MSO_TOP//bin/nls8174/ocommon/nls/admin/data;export APS64_ORA_NLS33

#ORA_NLS33=/erppgfb2/erpapp/appl/mso/11.5.0/bin/nls8174/ocommon/nls/admin/data;

export ORA_NLS33

# End customizations

I commented ORA_NLS33 as Oracle doesn't recommend it, but we have seen it work in some situations.  However it was working in production without this variable, so I commented it.

To test, this Ashish ran a small plan to check if it worked.  The small plan completed successfully without a signal 11 error.  He tested with full plan which takes 1 hour to complete, and that also succeeded.