Oracle Apps Technology

Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Monday, November 16, 2015

sqlplus core dumps with segmentation fault error in OEL 6.6 when you connect to DB

We have used OEL 6.6 image in our latest build.  When we cloned an EBS R12.2 instance that was on OEL 5.7 to this new server that has OEL 6.6, During the clone, was failing. On further checks, we discovered that sqlplus is crashing with segmentation fault error whenever we tried to connect to database:

sqlplus /nolog
conn apps/apps
Segmentation Fault

So, I suggested the DBAs to do strace sqlplus apps/apps.  The strace revealed many missing libraries:

We had another working OEL 6.4 instance where we checked for these libraries, and all of them were present.

The locate command was used to locate the full directory paths of the missing libraries



Then rpm -qf command was used to find out the rpm that would have the library:

$ rpm -qf /lib/
$ rpm -qf /lib/
$ rpm -qf /lib/
error: file /lib/ No such file or directory
$ rpm -qf /lib/
$ rpm -qf /lib/
$ rpm -qf /lib/
$ rpm -qf /lib/
$ rpm -qf /lib/
$ rpm -qf /lib/

Since 10.1.2 home is 32-bit in EBS R12.1 and 12.2, all the libraries needed to be 32-bit.

Except for sssd-client, the other rpms were present.  64-bit version of sssd-client was present and whenver we tried to install the 32-bit rpm it would give this error, as the operating system thinks that it is already installed:

# yum install sssd-client.i686
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sssd-client.i686 0:1.12.4-47.el6 will be installed
--> Finished Dependency Resolution
Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:

         1. You have an upgrade for sssd-client which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of sssd-client of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude sssd-client.otherarch ... this should give you an error
            message showing the root cause of the problem.

         2. You have multiple architectures of sssd-client installed, but
            yum can only see an upgrade for one of those arcitectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.

         3. You have duplicate versions of sssd-client installed already.
            You can use "yum check" to get yum show these errors. can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).

       Protected multilib versions: sssd-client-1.12.4-47.el6.i686 != sssd-client-1.11.6-30.el6_6.4.x86_64

# rpm -qa | grep sssd-client

Eventually we installed it with force option

# rpm -Uvh --force /tmp/sssd-client-1.11.6-30.el6_6.3.i686.rpm

# rpm -qa | grep sssd-client

pam-ldap was one of the other rpms that was installed for other missing libraries.  Surprisingly, sssd-client and pam-ldap rpms are not mentioned as pre-requisites in article:
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1) 

twm: unable to open fontset "-adobe-helvetica-bold-r-normal--*-120-*-*-*-*-*-*"

While launching twm, it gives this error and exits to unix prompt:

twm: unable to open fontset "-adobe-helvetica-bold-r-normal--*-120-*-*-*-*-*-*"

I found a solution on :

It was reported here for fedora: The workaround is to execute it with a specific shell variable:

$ export LANG
twm &

twm launches fine after this.

Saturday, November 14, 2015

Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured

Today, during a cutover when we were moving one of our ERP instance on Cisco UCS VMware VMs to Exalogic and Exadata, I got a call from Bimal.  The extranet iSupplier URL had been configured, but whenever any user logged in, they were seeing the following error instead of the iSupplier OAF Home page:

Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured

A search on showed many hits.  I went through a few of them and ruled out the solutions given. This article sounded promising: Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured (Doc ID 1474474.1).

The solution suggested:

There is  a hardware load-balancer for a multi-tier environment on place, as well as an SSL accelerator.

     For R12, there is a context variable, s_enable_sslterminator, that was set to "#".

     This should be null for e-Business R12 using specific hardwarementioned before.

1. Set  context variable, s_enable_sslterminator to null,

2. Re-ran autoconfig,

3. Re-test Single sign-ons via IE and Firefox now works as expected.

I asked the DBAs to check the value of s_enable_sslterminator:

grep s_enable_sslterminator

and sure enough the value was #

As per article Enabling SSL or TLS in Oracle E-Business Suite Release 12 (Doc ID 376700.1), the value of s_enable_sslterminator should be made null if you are using an SSL accelerator.  In our case we use SSL certificate on the Load Balancer and never on Web servers.

The DBAs removed the #
Ran autoconfig
Deregistered SSO
Registered SSO

The user was able to login after that.

Wednesday, October 21, 2015

How To Install Latest Verisign G5 Root Certificates

Dhananjay pinged me today and told me that for their Paypal integration, they had to upgrade to Verisign G5 root certificate.  This was the message from Paypal:

Global security threats are constantly changing, and the security of our merchants continues to be our highest priority. To guard against current and future threats, we are encouraging our merchants to make the following upgrades to their integrations:
  1. Update your integration to support certificates using the SHA-256 algorithm. PayPal is upgrading SSL certificates on all Live and Sandbox endpoints from SHA-1 to the stronger and more robust SHA-256 algorithm.
  2. Discontinue use of the VeriSign G2 Root Certificate. In accordance with industry standards, PayPal will no longer honor secure connections that require the VeriSign G2 Root Certificate for trust validation. Only secure connection requests that are expecting our certificate/trust chain to be signed by the G5 Root Certificate will result in successful secure connections.
For detailed information on these changes, please reference the Merchant Security System Upgrade Guide. For a basic introduction to internet security, we also recommend these short videos on SSL Certificates and Public Key Cryptography.

There is a article published on October 16, 2015 which has detailed steps for 11i and R12.1:

How To Install Latest Verisign Root Certificates For Use With Paypal SDK 4.3.X (Doc ID 874433.1)

The Verisign G5 root certificate can be downloaded from:

Paypal Microsite about this change:

Useful Links

Friday, October 9, 2015

sftp failure due to newline character difference between windows and unix.

Recently I spent almost a full day struggling to make out, why an sftp connection would not work without password, after setting up ssh equivalence.  The keys were correct, the permissions on the directories were correct.  The authorized_keys file looked ok.  I copied the authorized_keys file of another account that was working fine.  When I replaced the authorized_keys after taking backup of original authorized_keys, it started working.  So then I proceeded to check the contents in a hex editor

On the left side you have the authorized_keys file created in Windows.
On the right side you have the same authorized_keys file created in Unix.

If you notice the ends of the lines in the Windows file it shows CR LF, where as unix shows LF.

This difference is well described in the wikipedia article on newline character.

The one mistake I had done this time was create the authorized_keys file in Windows notepad, as I was teaching a Developer how to create authorized_keys file.  Once I used vi on unix to create the authorized_keys file and pasted the same ssh key, sftp started working without prompting for password.  I know that Windows/DOS and Unix have different newline characters.  However, I was not able to apply that knowledge, till I compared the files in hex editor.

Whenever, a techie is able to get to the root cause of a problem, a deep sense of satisfaction is experienced.  I am glad I got the opportunity to troubleshoot and fix the issue by getting to the root cause of the issue.

Tuesday, September 15, 2015

Copycat blog

While doing a google search today I noticed that there is another blog that has copied all content from my blog and posted it as their own content and even kept a similar sounding name: .  I have made a DMCA complaint to google about this.  The google team asked me to provide a list of URLs.  I had to go through the copycat's whole blog and create a spreadsheet with two columns. One column with URL of my original post and second column with the URL of the copycat's blog.  There were 498 entries.  I patiently did it and sent the spreadsheet to google team and got a reply within 2 hours:

In accordance with the Digital Millennium Copyright Act, we have completed processing your infringement notice. We are in the process of disabling access to the content in question at the following URL(s):

The content will be removed shortly.

The Google Team