Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Monday, November 16, 2015

sqlplus core dumps with segmentation fault error in OEL 6.6 when you connect to DB

We have used OEL 6.6 image in our latest build.  When we cloned an EBS R12.2 instance that was on OEL 5.7 to this new server that has OEL 6.6, During the clone, adcfgclone.pl was failing. On further checks, we discovered that sqlplus is crashing with segmentation fault error whenever we tried to connect to database:

sqlplus /nolog
conn apps/apps
Segmentation Fault

So, I suggested the DBAs to do strace sqlplus apps/apps.  The strace revealed many missing libraries:

We had another working OEL 6.4 instance where we checked for these libraries, and all of them were present.

The locate command was used to locate the full directory paths of the missing libraries

locate libnss_sss.so.2
/lib/libnss_sss.so.2

/lib/libnss_sss.so.2
/lib/libnss_files.so.2
/lib/libociei.so
/lib/libc.so.6
/lib/libgcc_s.so.1
/lib/libnsl.so.1
/lib/libpthread.so.0

Then rpm -qf command was used to find out the rpm that would have the library:

$ rpm -qf /lib/libnss_sss.so.2
sssd-client-1.11.6-30.el6_6.3.i686
$ rpm -qf /lib/libnss_files.so.2
glibc-2.12-1.149.el6_6.9.i686
$ rpm -qf /lib/libociei.so
error: file /lib/libociei.so: No such file or directory
$ rpm -qf /lib/libc.so.6
glibc-2.12-1.149.el6_6.9.i686
$ rpm -qf /lib/libgcc_s.so.1
libgcc-4.4.7-3.el6.i686
$ rpm -qf /lib/libnsl.so.1
glibc-2.12-1.149.el6_6.9.i686
$ rpm -qf /lib/libpthread.so.0
glibc-2.12-1.149.el6_6.9.i686
$ rpm -qf /lib/libm.so.6
glibc-2.12-1.149.el6_6.9.i686
$ rpm -qf /lib/libdl.so.2
glibc-2.12-1.149.el6_6.9.i686

Since 10.1.2 home is 32-bit in EBS R12.1 and 12.2, all the libraries needed to be 32-bit.

Except for sssd-client, the other rpms were present.  64-bit version of sssd-client was present and whenver we tried to install the 32-bit rpm it would give this error, as the operating system thinks that it is already installed:

# yum install sssd-client.i686
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sssd-client.i686 0:1.12.4-47.el6 will be installed
--> Finished Dependency Resolution
Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:

         1. You have an upgrade for sssd-client which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of sssd-client of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude sssd-client.otherarch ... this should give you an error
            message showing the root cause of the problem.

         2. You have multiple architectures of sssd-client installed, but
            yum can only see an upgrade for one of those arcitectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.

         3. You have duplicate versions of sssd-client installed already.
            You can use "yum check" to get yum show these errors.

       ...you can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).

       Protected multilib versions: sssd-client-1.12.4-47.el6.i686 != sssd-client-1.11.6-30.el6_6.4.x86_64


# rpm -qa | grep sssd-client
sssd-client-1.11.6-30.el6_6.4.x86_64

Eventually we installed it with force option

# rpm -Uvh --force /tmp/sssd-client-1.11.6-30.el6_6.3.i686.rpm

# rpm -qa | grep sssd-client
sssd-client-1.11.6-30.el6_6.3.i686
sssd-client-1.11.6-30.el6_6.4.x86_64

pam-ldap was one of the other rpms that was installed for other missing libraries.  Surprisingly, sssd-client and pam-ldap rpms are not mentioned as pre-requisites in support.oracle.com article:
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1) 

4 comments:

Леонид said...

We use OEL 6.7 and I don't see /lib/libnss_sss.so.2 in strace output while using sqlplus from 10.1.2 home.

It could be local problem related to strange sssd daemon configuration.

Vikram Das said...

It may be due to the fact that we are using central LDAP for our unix authentication. It is very much possible that when sqlplus is linked on a unix box that uses local users instead of central LDAP users, these libraries are not needed.

Neeli Raghava said...

Hi Vikram,

we are getting the same issue while login to EBS apps.
while connecting with non-user LDAP. we faced segmentation fault issue.

After searching so many blogs and support docs, didn't find any solution.

we installed the RPM 32 bit which was missing. after installing our problem solved.
Thank you very much for the solution which you have provided and the knowledge you shared to entire DBA world.

we installed the below RPM to fix.

sssd-client-1.11.6-30.el6_6.3.i686.rpm


error :

[oracle@dev01lvs appl]$ sqlplus apps/apps

SQL*Plus: Release 10.1.0.5.0 - Production on Tue Dec 13 07:16:33 2015

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Segmentation fault


Thanks
NRVK

prashanth said...

Hi Vikram,

Thanks a lot for the solution. I have faced this issue while installing 12c(32 bit) client on RHEL 7.2(64 bit) platform. I have verified with all available metalink solutions not worked for me.We struggled by this almost a day.

Finally i was able to fix it by installing 32 bit RPMs:

sssd-client-1.11.6-30.el6_6.3.i686.rpm and its dependency RPMs

I wonder by this because these rpms are not even mentioned as pre-requisites from oracle support. Thanks a lot for sharing