FNDLIBR segmentation fault on RHEL3 after upgrade to 11.5.10.2

Owen pinged me today. After doing an upgrade on an RHEL3 box, the concurrent managers refused to start up. This was the error logged in the concurrent manager logs:

Starting RCVOLTM Concurrent Manager : 10-JUL-2008 21:11:51
sh: line 71: 19678 Segmentation fault FNDLIBR FND CPMGR "FNDCPMBR sysmgr=\"\" $maxreq $sleep $pmon $quesiz $diag logfile=$logfile $target" STOP
$sysmanager
STOP

Metalink Note 343249.1 describes the same issue. However we were already on 11.5.10.2. And the patches suggested were already present. I searched some more and found Metalink Note 264950.1, which suggests applying 806 interop patch on RHEL boxes:

When GSM is not enabled the concurrent managers start up without issue, if GSM is enabled
and the concurrent managers are started I see the following in the log file:
.
Starting STANDARD Concurrent Manager : 15-DEC-2003 19:28:10
sh: line 71: 7884 Segmentation fault (core dumped) FNDLIBR FND CPMGR
"FNDCPMBR sysmgr=\"\" $maxreq $sleep $pmon $quesiz $diag logfile=$logfile
$target"

Fix

Apply patch 3293983, 8.0.6.3 RHEL3.0 APPLICATIONS INTEROPERABILITY PATCH

3293983 was obsoleted by 3830807. So I asked Owen to apply this patch. However he had already gone through this note and applied this patch. He said that applying the patch had no effect. I had a gut feeling that this was the root cause. So I opened the patch readme of 3293983:

#  You must login to your Linux machine as the Oracle software owner before
#  applying this patch. Your setting for ORACLE_HOME must be the value for the
#  Forms 6i 8.0.6 based ORACLE_HOME and you must have write previlages to this
#  area.
#
#  [1] To apply the patch, unzip the PSE container file:
#        % unzip p3830807_8063_LINUX.zip
#
#  [2] Set your current directory to the directory where the patch
#      is located, eg:
#
#        % cd 3830807
#
#  [3] Add execute permission to the script supplied in this patch
#
#        % chmod u+x patch.sh
#
#  [4] Check the value of LD_LIBRARY_PATH.
#
#      This patch will relink reports60 binaries.
#
#      Reports has both link-time and run-time dependency with libjava.so
#      so you need to include
#         $ORACLE_HOME/network/jre11/lib/linux/native_threads    OR
#         $ORACLE_HOME/network/jre11/lib/i686/native_threads
#      in $LD_LIBRARY_PATH before running patch.sh
#
#      Please check your files under $ORACLE_HOME/network/jre11/lib to see
#      which one of the above is appropriate on your system. The same
#      $LD_LIBRARY_PATH should be used at run-time.
#
#  [5] Run the script supplied by this patch
#
#        % ./patch.sh
#
#  [6] Relink all Applications executables. In AD Administration, choose
#      "Relink Applications programs" from the "Generate Applications Files"
#      menu.
#
#  The installation of the patch is now complete.

I asked owen whether he had relinked all Apps executables, and he said that he had not read the readme of the patch. On realizing the missed step, he relinked all executables through adadmin and restarted the concurrent managers. All the concurrent managers started without issue. The problem was resolved.