sqlplus core dumps with segmentation fault error in OEL 6.6 when you connect to DB

We have used OEL 6.6 image in our latest build.  When we cloned an EBS R12.2 instance that was on OEL 5.7 to this new server that has OEL 6.6, During the clone, adcfgclone.pl was failing. On further checks, we discovered that sqlplus is crashing with segmentation fault error whenever we tried to connect to database:

sqlplus /nolog
conn apps/apps
Segmentation Fault

So, I suggested the DBAs to do strace sqlplus apps/apps.  The strace revealed many missing libraries:

We had another working OEL 6.4 instance where we checked for these libraries, and all of them were present.

The locate command was used to locate the full directory paths of the missing libraries

locate libnss_sss.so.2

/lib/libnss_sss.so.2

/lib/libnss_sss.so.2
/lib/libnss_files.so.2
/lib/libociei.so
/lib/libc.so.6
/lib/libgcc_s.so.1
/lib/libnsl.so.1
/lib/libpthread.so.0

Then rpm -qf command was used to find out the rpm that would have the library:

$ rpm -qf /lib/libnss_sss.so.2

sssd-client-1.11.6-30.el6_6.3.i686

$ rpm -qf /lib/libnss_files.so.2

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libociei.so

error: file /lib/libociei.so: No such file or directory

$ rpm -qf /lib/libc.so.6

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libgcc_s.so.1

libgcc-4.4.7-3.el6.i686

$ rpm -qf /lib/libnsl.so.1

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libpthread.so.0

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libm.so.6

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libdl.so.2

glibc-2.12-1.149.el6_6.9.i686

Since 10.1.2 home is 32-bit in EBS R12.1 and 12.2, all the libraries needed to be 32-bit.

Except for sssd-client, the other rpms were present.  64-bit version of sssd-client was present and whenver we tried to install the 32-bit rpm it would give this error, as the operating system thinks that it is already installed:

# yum install sssd-client.i686

Loaded plugins: security

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package sssd-client.i686 0:1.12.4-47.el6 will be installed

--> Finished Dependency Resolution

Error:  Multilib version problems found. This often means that the root

       cause is something else and multilib version checking is just

       pointing out that there is a problem. Eg.:

         1. You have an upgrade for sssd-client which is missing some

            dependency that another package requires. Yum is trying to

            solve this by installing an older version of sssd-client of the

            different architecture. If you exclude the bad architecture

            yum will tell you what the root cause is (which package

            requires what). You can try redoing the upgrade with

            --exclude sssd-client.otherarch ... this should give you an error

            message showing the root cause of the problem.

         2. You have multiple architectures of sssd-client installed, but

            yum can only see an upgrade for one of those arcitectures.

            If you don't want/need both architectures anymore then you

            can remove the one with the missing update and everything

            will work.

         3. You have duplicate versions of sssd-client installed already.

            You can use "yum check" to get yum show these errors.

       ...you can also use --setopt=protected_multilib=false to remove

       this checking, however this is almost never the correct thing to

       do as something else is very likely to go wrong (often causing

       much more problems).

       Protected multilib versions: sssd-client-1.12.4-47.el6.i686 != sssd-client-1.11.6-30.el6_6.4.x86_64

# rpm -qa | grep sssd-client

sssd-client-1.11.6-30.el6_6.4.x86_64

Eventually we installed it with force option

# rpm -Uvh --force /tmp/sssd-client-1.11.6-30.el6_6.3.i686.rpm

# rpm -qa | grep sssd-client

sssd-client-1.11.6-30.el6_6.3.i686

sssd-client-1.11.6-30.el6_6.4.x86_64

pam-ldap was one of the other rpms that was installed for other missing libraries.  Surprisingly, sssd-client and pam-ldap rpms are not mentioned as pre-requisites in support.oracle.com article:

Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1) 

vnc xterm window appears without header

The program twm is "the window manager" that is responsible for the window header that appears on top of terminal windows of xterm.  In newer OEL 6 builds, when xterm is launched, the header doesn't appear as twm fails to launch.  If you try launching twm, it gives this error and exits to unix prompt:

twm: unable to open fontset "-adobe-helvetica-bold-r-normal--*-120-*-*-*-*-*-*"

I found a solution on http://ubuntuforums.org/archive/index.php/t-1596636.html :

It was reported here for fedora: https://bugzilla.redhat.com/show_bug.cgi?id=509639. The workaround is to execute it with a specific shell variable:

$ DISPLAY=:vnc port number
$ export DISPLAY
$ xhost +
$ LANG=C
$ export LANG
twm &

twm launches fine after this.

Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured

Today, during a cutover when we were moving one of our ERP instance on Cisco UCS VMware VMs to Exalogic and Exadata, I got a call from Bimal.  The extranet iSupplier URL had been configured, but whenever any user logged in, they were seeing the following error instead of the iSupplier OAF Home page:

Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured

A search on support.oracle.com showed many hits.  I went through a few of them and ruled out the solutions given. This article sounded promising: Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured (Doc ID 1474474.1).

The solution suggested:

There is  a hardware load-balancer for a multi-tier environment on place, as well as an SSL accelerator.

     For R12, there is a context variable, s_enable_sslterminator, that was set to "#".

     This should be null for e-Business R12 using specific hardwarementioned before.


1. Set  context variable, s_enable_sslterminator to null,

2. Re-ran autoconfig,

3. Re-test Single sign-ons via IE and Firefox now works as expected.

I asked the DBAs to check the value of s_enable_sslterminator:

grep s_enable_sslterminator

and sure enough the value was #

As per article Enabling SSL or TLS in Oracle E-Business Suite Release 12 (Doc ID 376700.1), the value of s_enable_sslterminator should be made null if you are using an SSL accelerator.  In our case we use SSL certificate on the Load Balancer and never on Web servers.

The DBAs removed the #
Ran autoconfig
Deregistered SSO
Registered SSO

The user was able to login after that.