Oracle Apps Technology

Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Tuesday, November 27, 2007

libucb.so.1 open failed: No such file or directory

I got a mail from Sandeep today reporting this error:

$ adfrmctl.sh start

You are running adfrmctl.sh version 115.37

ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed
Starting forms server for example11i on port 9000.
ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed

adfrmctl.sh: exiting with status 0

I advised him to relink the FND and AD binaries, however adadmin resulted in this error:

$ adadmin
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
Killed


We did a truss to find out in which directories it was searching for libucb.so.1:

$ truss adadmin
execve("/example11i/erpapp/appl/ad/11.5.0/bin/adadmin", 0xFFBEAFAC, 0xFFBEAFB4) argc = 1
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
open("/var/ld/ld.config", O_RDONLY) = 3
fstat(3, 0xFFBEA808) = 0
mmap(0x00000000, 992, PROT_READ, MAP_SHARED, 3, 0) = 0xFF390000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 0xFFBEA890) = 0
resolvepath("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", "/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 1023) = 43
open("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF380000
mmap(0x0D3731A8, 4595712, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFEC00000
mmap(0xFEC00000, 4322968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFEC00000
mmap(0xFF02E000, 144324, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 4317184) = 0xFF02E000
mmap(0xFF052000, 64584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF052000
munmap(0xFF020000, 57344) = 0
memcntl(0xFEC00000, 445716, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libnsl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libnsl.so.1", "/usr/lib/libnsl.so.1", 1023) = 20
open("/usr/lib/libnsl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AFA2C0, 655360, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF280000
mmap(0xFF280000, 582238, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF280000
mmap(0xFF310000, 33256, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 589824) = 0xFF310000
mmap(0xFF31A000, 23312, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF31A000
memcntl(0xFF280000, 84064, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libsocket.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libsocket.so.1", "/usr/lib/libsocket.so.1", 1023) = 23
open("/usr/lib/libsocket.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AF9ED8, 114688, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF360000
mmap(0xFF360000, 40558, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF360000
mmap(0xFF37A000, 4365, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 40960) = 0xFF37A000
munmap(0xFF36A000, 65536) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF350000
memcntl(0xFF360000, 14496, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libgen.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libgen.so.1", "/usr/lib/libgen.so.1", 1023) = 20
open("/usr/lib/libgen.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x1516F4D0, 98304, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF330000
mmap(0xFF330000, 23073, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF330000
mmap(0xFF346000, 2335, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 24576) = 0xFF346000
munmap(0xFF336000, 65536) = 0
memcntl(0xFF330000, 6932, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libdl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libdl.so.1", "/usr/lib/libdl.so.1", 1023) = 19
open("/usr/lib/libdl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B00850, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF270000
mmap(0xFF270000, 2302, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF270000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libc.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libc.so.1", "/usr/lib/libc.so.1", 1023) = 18
open("/usr/lib/libc.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B042E8, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF180000
mmap(0xFF180000, 704200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF180000
mmap(0xFF23C000, 24772, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF23C000
munmap(0xFF22C000, 65536) = 0
memcntl(0xFF180000, 113528, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libaio.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libaio.so.1", "/usr/lib/libaio.so.1", 1023) = 20
open("/usr/lib/libaio.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x0295B288, 106496, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF160000
mmap(0xFF160000, 28909, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF160000
mmap(0xFF178000, 1584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 32768) = 0xFF178000
munmap(0xFF168000, 65536) = 0
memcntl(0xFF160000, 7184, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libm.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libm.so.1", "/usr/lib/libm.so.1", 1023) = 18
open("/usr/lib/libm.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x016884F8, 294912, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF110000
mmap(0xFF110000, 225354, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF110000
mmap(0xFF156000, 7292, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 221184) = 0xFF156000
munmap(0xFF148000, 57344) = 0
memcntl(0xFF110000, 10604, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
write(2, " l d . s o . 1 : a d a".., 77) = 77
munmap(0xFF380000, 8192) = 0
lwp_self() = 1
*** process killed ***

Based on the above it is clear that it is looking for libucb.so.1 in $LD_LIBRARY_PATH.

This is what we did as a workaround:

cd $ORACLE_HOME/lib
ln -s /usr/ucblib/libucb.so.1 libucb.so.1

After this both adadmin and adfrmctl.sh worked fine.

However the question still remains why it happened. I compared adadmin of this particular instance and a vision instance:

$ ldd /vision11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /vision11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1


$ ldd /example11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /example11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libucb.so.1 => /gpswms76/erpapp/8.0.6/lib/libucb.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2
libelf.so.1 => /usr/lib/libelf.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1

The list of libraries being referenced is not same. Clearly something was wrong in the environment when the binaries were relinked.

Metalink Note 164279.1 describes a similar problem:

PROBLEM DESCRIPTION
====================

You receive the following error message when invoking an oracle executable
on Sun Solaris. This applies to all executbles including Database, Networking,
Tools,and Applications.

 ld.so.1: : fatal: libucb.so.1: open failed:
  No such file or directory
 Killed
 

PROBLEM EXPLANATION
======================

The executable was built using the wrong linker ('ld').  The Solaris
operating system comes with two linkers, the SVR4 linker (/usr/ccs/bin/ld)
and SunOS 4.x compatibility linker (/usr/ucb/ld). Oracle requires the use of
the /usr/ccs/bin utitlities to properly link the executables and
if /usr/ucb/ld is used instead of /usr/ccs/bin/ld, executables will fail with
the above error message.

NOTE:   Please see the appropriate product Installation Guide for
       O/S requirments.


SOLUTION DESCRIPTION
=====================

To correct the problem, you'll need to modify your PATH environment variable
so that the /usr/ccs/bin directory comes before /usr/ucb directory, and then
relink the executables that were linked with the wrong linker.

1. Correct the PATH environment variable.
 
 csh
 -----
 % setenv PATH /usr/ccs/bin:${PATH}

 sh, ksh or bash
 -----------
 % PATH=/usr/ccs/bin:$PATH
 % export PATH

2. Ensure the PATH is set correctly.

 % which ld
 /usr/ccs/bin/ld  <== This should be returned if the the PATH is
                            correct.

3. Relink the executables:

 See Note 131321.1 for Data Server relinking instructions.
I am not sure if PATH was incorrectly set when the binaries were relinked. However, I have advised the DBA team to relink all binaries based on the above analysis.
Posted by at
Labels: ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)