In E-Business Suite 11.5.10 the default value of context variable, s_appserverid_authentication, is SECURE. In previous E-Business Suite 11i versions, this was set to OFF.
There are 3 Modes of authentication:
OFF - Server security is not checked. Any application server machine can access the database. Code IDs are also not checked. Use this option on test systems or if you have full control over the software on all machines which can physically access your database.
ON - Some level of trust is required to access the database. Either the application server must be registered with the database or the code must pass a module and version ID known to be trusted. Use this option only if you wish to maintain compatibility with application servers that you cannot yet patch to the code level required for best security.
SECURE - Full trust is required for access to the database. Only registered application server machines may connect to the database, and only trusted code modules may connect to the database.
If you are on Oracle Applications 11.5.10.2, the context variable, s_appserverid_authentication, is the parameter that allows you to skip the framework login. Setting this parameter to "OFF" mode is only to be used for debugging purposes. Any other use, such as leaving it in the "OFF" mode for regular login, is not supported.
Friday, November 30, 2007
Thursday, November 29, 2007
adadmin errors out with ORA-00942: TABLE OR VIEW DOES NOT EXIST
After a clone in one of the environments, whenever adadmin was run the following error appeared:
AD Administration error:
The following ORACLE error:
ORA-00942: table or view does not exist
occurred while executing the SQL statement:
SELECT to_char(APPLSYS.AD_SESSIONS_S.NEXTVAL) FROM SYS.DUAL
Unable to get the current value of sequence APPLSYS.AD_SESSIONS_S
We tried to check for the ownership and existance of the sequence AD_SESSIONS_S:
SQL> SELECT SUBSTR(OWNER,1,12)
OWNER,SUBSTR(OBJECT_NAME,1,45) NAME, SUBSTR(OBJECT_TYPE,1,12) TYPE
FROM DBA_OBJECTS where object_name = 'AD_SESSIONS_S' ;
OWNER NAME TYPE
------------ --------------------------------------------- ------------
APPLSYS AD_SESSIONS_S SEQUENCE
APPS AD_SESSIONS_S SYNONYM
WAVESET AD_SESSIONS_S SYNONYM
APPS_WEB AD_SESSIONS_S SYNONYM
Also tried to execute the query manually:
SELECT to_char(APPLSYS.AD_SESSIONS_S.NEXTVAL) FROM SYS.DUAL;
TO_CHAR(APPLSYS.AD_SESSIONS_S.NEXTVAL)
----------------------------------------
1852
We tried recreating grants and synonyms but it did not work. So we followed note 149987.1 and recreated AD_SESSIONS_S:
Run adodfcmp adutil.odf to re-create the AD_SESSIONs_S
Run adodfcmp for both tables and sequences.
Navigate to $AD_TOP/admin/odf and run the adodfcmp command similar to the
following
$ adodfcmp mode=tables touser=apps/apps priv_schema=system/manager \
> odffile=adutil.odf userid=applsys/apps changedb=yes
$ adodfcmp mode=sequences touser=apps/apps priv_schema=system/manager \
> odffile=adutil.odf userid=applsys/apps changedb=yes
That fixed the error.
AD Administration error:
The following ORACLE error:
ORA-00942: table or view does not exist
occurred while executing the SQL statement:
SELECT to_char(APPLSYS.AD_SESSIONS_S.NEXTVAL) FROM SYS.DUAL
Unable to get the current value of sequence APPLSYS.AD_SESSIONS_S
We tried to check for the ownership and existance of the sequence AD_SESSIONS_S:
SQL> SELECT SUBSTR(OWNER,1,12)
OWNER,SUBSTR(OBJECT_NAME,1,45) NAME, SUBSTR(OBJECT_TYPE,1,12) TYPE
FROM DBA_OBJECTS where object_name = 'AD_SESSIONS_S' ;
OWNER NAME TYPE
------------ --------------------------------------------- ------------
APPLSYS AD_SESSIONS_S SEQUENCE
APPS AD_SESSIONS_S SYNONYM
WAVESET AD_SESSIONS_S SYNONYM
APPS_WEB AD_SESSIONS_S SYNONYM
Also tried to execute the query manually:
SELECT to_char(APPLSYS.AD_SESSIONS_S.NEXTVAL) FROM SYS.DUAL;
TO_CHAR(APPLSYS.AD_SESSIONS_S.NEXTVAL)
----------------------------------------
1852
We tried recreating grants and synonyms but it did not work. So we followed note 149987.1 and recreated AD_SESSIONS_S:
Run adodfcmp adutil.odf to re-create the AD_SESSIONs_S
Run adodfcmp for both tables and sequences.
Navigate to $AD_TOP/admin/odf and run the adodfcmp command similar to the
following
$ adodfcmp mode=tables touser=apps/apps priv_schema=system/manager \
> odffile=adutil.odf userid=applsys/apps changedb=yes
$ adodfcmp mode=sequences touser=apps/apps priv_schema=system/manager \
> odffile=adutil.odf userid=applsys/apps changedb=yes
That fixed the error.
Wednesday, November 28, 2007
ORA-02083:database name has illegal character '.'
Recently I was asked for help in an issue which is related to ASCP.
In OLTP instance, from OM Order Administrator Primary responsibility
1.Open ATP Inquiry Form
2.Give the Item name and Qty
3.click on the Global Availability Button.
4.Following Error message appears and user is unable to view Global Availability:
ORA-02083:database name has illegal character '.'
ORA-06512:at "SYS.DBMS_SYS_SQL", line 909
ORA-06512:at "SYS.DBMS_SQL", line 39
ORA-06512:at "APPS.MSC_SCH_WB", line 842
Official documentation states the cause of ORA-02083 as:
The database name supplied contains an invalid character. For example, there can be no periods, semicolons, single quotes, double quotes, blanks, or non-printable characters in the database name.
First I checked for the global name of OLTP and ASCP instances:
OLTP:
SQL> select * from global_name;
oltp11i.production.example.com
The global name still had the domain for production. So I changed it to:
SQL> alter database rename global name to oltp11i.development.example.com;
Database Altered
SQL> select * from global_name;
oltp11i.development.example.com
ASCP:
SQL>select * from global_name;
ascp11i
If you query dba_db_links table, you'll find that it automatically attaches the domain name derived from global name of the instance.
I dropped and recreated the ASCP hookup DB links without domain name.
ASCP Instance:
SQL> drop database link ASCP2OLTP.DEVELOPMENT.EXAMPLE.COM;
SQL> create database link ASCP2OLTP connect to apps identified by apps using 'oltp11i';
OLTP Instance:
SQL>drop database link OLTP2ASCP.DEVELOPMENT.EXAMPLE.COM;
SQL> create database link OLTP2ASCP connect to apps identified by apps using 'ascp11i';
I checked for the DB link names in MRP_AP_APPS_INSTANCES:
SQL> select M2A_DBLINK,A2M_DBLINK from mrp_ap_apps_instances
2 /
M2A_DBLINK A2M_DBLINK
--------------------------------- --------------------------------
ASCP2OLTP.DEVELOPMENT.EXAMPLE.COM OLTP2ASCP.DEVELOPMENT.EXAMPLE.COM
The DB links had been specified with domain name. I removed the domain names altogether, as the '.' (period) is not allowed in the base db name by the following statement.
SQL> update mrp_ap_apps_instances
2 set m2a_dblink='ASCP2OLTP',A2M_DBLINK='OLTP2ASCP';
1 row updated.
SQL> COMMIT;
Commit complete.
SQL> select M2A_DBLINK,A2M_DBLINK from mrp_ap_apps_instances
2 /
M2A_DBLINK A2M_DBLINK
--------------------------- -----------------------
ASCP2OLTP OLTP2ASCP
After these changes the error changed to this:
ORA-02019: connection description for remote database not found
ORA-06512: at "APPS.MSC_ATP_PROC", line 3519
ORA-06512: at line 1
ORA-06512: at "APPS.MSC_ATPSOURCES_NEW", line 53
ORA-06512: at "APPS.MSC_SCH_WB", line 660
ORA-06512: at "APPS.ORDER_SCH_WB", line 54
------------------------------------------------
FRM-40735: WHEN-BUTTON-PRESSED trigger raised unhandled exception ORA-02019.
Tracing the error from forms side or the session from DB side is not logging this error. Still investigating this one.
Akhilesh solved this one for me. This is what he did:
On OLTP instance:
Checked the value of profile option MRP:ATP Database Link and it was oltp2ascp.development.example.com. So we removed the domain name and made it oltp2ascp
On ASCP Instance:
Logged in with Advanced Planning Administrator responsibility
Admin > Instances
The DB link values in the fields "From Source to APS" and "From APS to Source" had the dblink with domain name. Removed the domain name. Saved.
Voila, the issue is solved. Error message no longer appears and Global Availability form opens on clicking the button.
In OLTP instance, from OM Order Administrator Primary responsibility
1.Open ATP Inquiry Form
2.Give the Item name and Qty
3.click on the Global Availability Button.
4.Following Error message appears and user is unable to view Global Availability:
ORA-02083:database name has illegal character '.'
ORA-06512:at "SYS.DBMS_SYS_SQL", line 909
ORA-06512:at "SYS.DBMS_SQL", line 39
ORA-06512:at "APPS.MSC_SCH_WB", line 842
Official documentation states the cause of ORA-02083 as:
The database name supplied contains an invalid character. For example, there can be no periods, semicolons, single quotes, double quotes, blanks, or non-printable characters in the database name.
First I checked for the global name of OLTP and ASCP instances:
OLTP:
SQL> select * from global_name;
oltp11i.production.example.com
The global name still had the domain for production. So I changed it to:
SQL> alter database rename global name to oltp11i.development.example.com;
Database Altered
SQL> select * from global_name;
oltp11i.development.example.com
ASCP:
SQL>select * from global_name;
ascp11i
If you query dba_db_links table, you'll find that it automatically attaches the domain name derived from global name of the instance.
I dropped and recreated the ASCP hookup DB links without domain name.
ASCP Instance:
SQL> drop database link ASCP2OLTP.DEVELOPMENT.EXAMPLE.COM;
SQL> create database link ASCP2OLTP connect to apps identified by apps using 'oltp11i';
OLTP Instance:
SQL>drop database link OLTP2ASCP.DEVELOPMENT.EXAMPLE.COM;
SQL> create database link OLTP2ASCP connect to apps identified by apps using 'ascp11i';
I checked for the DB link names in MRP_AP_APPS_INSTANCES:
SQL> select M2A_DBLINK,A2M_DBLINK from mrp_ap_apps_instances
2 /
M2A_DBLINK A2M_DBLINK
--------------------------------- --------------------------------
ASCP2OLTP.DEVELOPMENT.EXAMPLE.COM OLTP2ASCP.DEVELOPMENT.EXAMPLE.COM
The DB links had been specified with domain name. I removed the domain names altogether, as the '.' (period) is not allowed in the base db name by the following statement.
SQL> update mrp_ap_apps_instances
2 set m2a_dblink='ASCP2OLTP',A2M_DBLINK='OLTP2ASCP';
1 row updated.
SQL> COMMIT;
Commit complete.
SQL> select M2A_DBLINK,A2M_DBLINK from mrp_ap_apps_instances
2 /
M2A_DBLINK A2M_DBLINK
--------------------------- -----------------------
ASCP2OLTP OLTP2ASCP
After these changes the error changed to this:
ORA-02019: connection description for remote database not found
ORA-06512: at "APPS.MSC_ATP_PROC", line 3519
ORA-06512: at line 1
ORA-06512: at "APPS.MSC_ATPSOURCES_NEW", line 53
ORA-06512: at "APPS.MSC_SCH_WB", line 660
ORA-06512: at "APPS.ORDER_SCH_WB", line 54
------------------------------------------------
FRM-40735: WHEN-BUTTON-PRESSED trigger raised unhandled exception ORA-02019.
Tracing the error from forms side or the session from DB side is not logging this error. Still investigating this one.
Akhilesh solved this one for me. This is what he did:
On OLTP instance:
Checked the value of profile option MRP:ATP Database Link and it was oltp2ascp.development.example.com. So we removed the domain name and made it oltp2ascp
On ASCP Instance:
Logged in with Advanced Planning Administrator responsibility
Admin > Instances
The DB link values in the fields "From Source to APS" and "From APS to Source" had the dblink with domain name. Removed the domain name. Saved.
Voila, the issue is solved. Error message no longer appears and Global Availability form opens on clicking the button.
Tuesday, November 27, 2007
ld.so.1: vi: fatal: relocation error
While trying to solve the issue in the previous post, I had modified the LD_LIBRARY_PATH to include /usr/ucblib directory. This was the result:
$vi
ld.so.1: vi: fatal: relocation error: file /usr/bin/vi: symbol cur_term: referenced symbol not found
unset LD_LIBRARY_PATH
vi worked fine. /usr/ucblib directory should not be in the $LD_LIBRARY_PATH. If it is, you'll get this error.
$vi
ld.so.1: vi: fatal: relocation error: file /usr/bin/vi: symbol cur_term: referenced symbol not found
unset LD_LIBRARY_PATH
vi worked fine. /usr/ucblib directory should not be in the $LD_LIBRARY_PATH. If it is, you'll get this error.
libucb.so.1 open failed: No such file or directory
I got a mail from Sandeep today reporting this error:
$ adfrmctl.sh start
You are running adfrmctl.sh version 115.37
ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed
Starting forms server for example11i on port 9000.
ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed
adfrmctl.sh: exiting with status 0
I advised him to relink the FND and AD binaries, however adadmin resulted in this error:
$ adadmin
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
Killed
We did a truss to find out in which directories it was searching for libucb.so.1:
$ truss adadmin
execve("/example11i/erpapp/appl/ad/11.5.0/bin/adadmin", 0xFFBEAFAC, 0xFFBEAFB4) argc = 1
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
open("/var/ld/ld.config", O_RDONLY) = 3
fstat(3, 0xFFBEA808) = 0
mmap(0x00000000, 992, PROT_READ, MAP_SHARED, 3, 0) = 0xFF390000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 0xFFBEA890) = 0
resolvepath("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", "/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 1023) = 43
open("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF380000
mmap(0x0D3731A8, 4595712, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFEC00000
mmap(0xFEC00000, 4322968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFEC00000
mmap(0xFF02E000, 144324, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 4317184) = 0xFF02E000
mmap(0xFF052000, 64584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF052000
munmap(0xFF020000, 57344) = 0
memcntl(0xFEC00000, 445716, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libnsl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libnsl.so.1", "/usr/lib/libnsl.so.1", 1023) = 20
open("/usr/lib/libnsl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AFA2C0, 655360, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF280000
mmap(0xFF280000, 582238, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF280000
mmap(0xFF310000, 33256, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 589824) = 0xFF310000
mmap(0xFF31A000, 23312, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF31A000
memcntl(0xFF280000, 84064, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libsocket.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libsocket.so.1", "/usr/lib/libsocket.so.1", 1023) = 23
open("/usr/lib/libsocket.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AF9ED8, 114688, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF360000
mmap(0xFF360000, 40558, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF360000
mmap(0xFF37A000, 4365, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 40960) = 0xFF37A000
munmap(0xFF36A000, 65536) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF350000
memcntl(0xFF360000, 14496, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libgen.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libgen.so.1", "/usr/lib/libgen.so.1", 1023) = 20
open("/usr/lib/libgen.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x1516F4D0, 98304, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF330000
mmap(0xFF330000, 23073, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF330000
mmap(0xFF346000, 2335, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 24576) = 0xFF346000
munmap(0xFF336000, 65536) = 0
memcntl(0xFF330000, 6932, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libdl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libdl.so.1", "/usr/lib/libdl.so.1", 1023) = 19
open("/usr/lib/libdl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B00850, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF270000
mmap(0xFF270000, 2302, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF270000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libc.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libc.so.1", "/usr/lib/libc.so.1", 1023) = 18
open("/usr/lib/libc.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B042E8, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF180000
mmap(0xFF180000, 704200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF180000
mmap(0xFF23C000, 24772, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF23C000
munmap(0xFF22C000, 65536) = 0
memcntl(0xFF180000, 113528, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libaio.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libaio.so.1", "/usr/lib/libaio.so.1", 1023) = 20
open("/usr/lib/libaio.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x0295B288, 106496, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF160000
mmap(0xFF160000, 28909, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF160000
mmap(0xFF178000, 1584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 32768) = 0xFF178000
munmap(0xFF168000, 65536) = 0
memcntl(0xFF160000, 7184, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libm.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libm.so.1", "/usr/lib/libm.so.1", 1023) = 18
open("/usr/lib/libm.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x016884F8, 294912, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF110000
mmap(0xFF110000, 225354, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF110000
mmap(0xFF156000, 7292, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 221184) = 0xFF156000
munmap(0xFF148000, 57344) = 0
memcntl(0xFF110000, 10604, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
write(2, " l d . s o . 1 : a d a".., 77) = 77
munmap(0xFF380000, 8192) = 0
lwp_self() = 1
*** process killed ***
Based on the above it is clear that it is looking for libucb.so.1 in $LD_LIBRARY_PATH.
This is what we did as a workaround:
cd $ORACLE_HOME/lib
ln -s /usr/ucblib/libucb.so.1 libucb.so.1
After this both adadmin and adfrmctl.sh worked fine.
However the question still remains why it happened. I compared adadmin of this particular instance and a vision instance:
$ ldd /vision11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /vision11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
$ ldd /example11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /example11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libucb.so.1 => /gpswms76/erpapp/8.0.6/lib/libucb.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2
libelf.so.1 => /usr/lib/libelf.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
The list of libraries being referenced is not same. Clearly something was wrong in the environment when the binaries were relinked.
Metalink Note 164279.1 describes a similar problem:
$ adfrmctl.sh start
You are running adfrmctl.sh version 115.37
ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed
Starting forms server for example11i on port 9000.
ld.so.1: FNDSVCRG: fatal: libucb.so.1: open failed: No such file or directory
Killed
adfrmctl.sh: exiting with status 0
I advised him to relink the FND and AD binaries, however adadmin resulted in this error:
$ adadmin
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
Killed
We did a truss to find out in which directories it was searching for libucb.so.1:
$ truss adadmin
execve("/example11i/erpapp/appl/ad/11.5.0/bin/adadmin", 0xFFBEAFAC, 0xFFBEAFB4) argc = 1
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
open("/var/ld/ld.config", O_RDONLY) = 3
fstat(3, 0xFFBEA808) = 0
mmap(0x00000000, 992, PROT_READ, MAP_SHARED, 3, 0) = 0xFF390000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libclntsh.so.1.0", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 0xFFBEA890) = 0
resolvepath("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", "/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", 1023) = 43
open("/example11i/erpapp/8.0.6/lib/libclntsh.so.1.0", O_RDONLY) = 3
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF380000
mmap(0x0D3731A8, 4595712, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFEC00000
mmap(0xFEC00000, 4322968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFEC00000
mmap(0xFF02E000, 144324, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 4317184) = 0xFF02E000
mmap(0xFF052000, 64584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF052000
munmap(0xFF020000, 57344) = 0
memcntl(0xFEC00000, 445716, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libnsl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libnsl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libnsl.so.1", "/usr/lib/libnsl.so.1", 1023) = 20
open("/usr/lib/libnsl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AFA2C0, 655360, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF280000
mmap(0xFF280000, 582238, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF280000
mmap(0xFF310000, 33256, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 589824) = 0xFF310000
mmap(0xFF31A000, 23312, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF31A000
memcntl(0xFF280000, 84064, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libsocket.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libsocket.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libsocket.so.1", "/usr/lib/libsocket.so.1", 1023) = 23
open("/usr/lib/libsocket.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15AF9ED8, 114688, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF360000
mmap(0xFF360000, 40558, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF360000
mmap(0xFF37A000, 4365, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 40960) = 0xFF37A000
munmap(0xFF36A000, 65536) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF350000
memcntl(0xFF360000, 14496, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libgen.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libgen.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libgen.so.1", "/usr/lib/libgen.so.1", 1023) = 20
open("/usr/lib/libgen.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x1516F4D0, 98304, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF330000
mmap(0xFF330000, 23073, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF330000
mmap(0xFF346000, 2335, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 24576) = 0xFF346000
munmap(0xFF336000, 65536) = 0
memcntl(0xFF330000, 6932, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libdl.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libdl.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libdl.so.1", "/usr/lib/libdl.so.1", 1023) = 19
open("/usr/lib/libdl.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B00850, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF270000
mmap(0xFF270000, 2302, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF270000
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libc.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libc.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libc.so.1", "/usr/lib/libc.so.1", 1023) = 18
open("/usr/lib/libc.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x15B042E8, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF180000
mmap(0xFF180000, 704200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF180000
mmap(0xFF23C000, 24772, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF23C000
munmap(0xFF22C000, 65536) = 0
memcntl(0xFF180000, 113528, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libaio.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libaio.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libaio.so.1", "/usr/lib/libaio.so.1", 1023) = 20
open("/usr/lib/libaio.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x0295B288, 106496, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF160000
mmap(0xFF160000, 28909, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF160000
mmap(0xFF178000, 1584, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 32768) = 0xFF178000
munmap(0xFF168000, 65536) = 0
memcntl(0xFF160000, 7184, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libm.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libm.so.1", 0xFFBEA890) = 0
resolvepath("/usr/lib/libm.so.1", "/usr/lib/libm.so.1", 1023) = 18
open("/usr/lib/libm.so.1", O_RDONLY) = 3
mmap(0xFF380000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0x016884F8, 294912, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF110000
mmap(0xFF110000, 225354, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF110000
mmap(0xFF156000, 7292, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 221184) = 0xFF156000
munmap(0xFF148000, 57344) = 0
memcntl(0xFF110000, 10604, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/example11i/erpapp/8.0.6/network/jre11/lib/sparc/native_threads/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/appl/cz/11.5.0/bin/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/dt/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/openwin/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/opt/SUNWcluster/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/example11i/erpapp/8.0.6/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
stat("/usr/lib/libucb.so.1", 0xFFBEA890) Err#2 ENOENT
ld.so.1: adadmin: fatal: libucb.so.1: open failed: No such file or directory
write(2, " l d . s o . 1 : a d a".., 77) = 77
munmap(0xFF380000, 8192) = 0
lwp_self() = 1
*** process killed ***
Based on the above it is clear that it is looking for libucb.so.1 in $LD_LIBRARY_PATH.
This is what we did as a workaround:
cd $ORACLE_HOME/lib
ln -s /usr/ucblib/libucb.so.1 libucb.so.1
After this both adadmin and adfrmctl.sh worked fine.
However the question still remains why it happened. I compared adadmin of this particular instance and a vision instance:
$ ldd /vision11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /vision11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
$ ldd /example11i/erpapp/appl/ad/11.5.0/bin/adadmin
libclntsh.so.1.0 => /example11i/erpapp/8.0.6/lib/libclntsh.so.1.0
libnsl.so.1 => /usr/lib/libnsl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libgen.so.1 => /usr/lib/libgen.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libc.so.1 => /usr/lib/libc.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libm.so.1 => /usr/lib/libm.so.1
libucb.so.1 => /gpswms76/erpapp/8.0.6/lib/libucb.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2
libelf.so.1 => /usr/lib/libelf.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
The list of libraries being referenced is not same. Clearly something was wrong in the environment when the binaries were relinked.
Metalink Note 164279.1 describes a similar problem:
PROBLEM DESCRIPTION
====================
You receive the following error message when invoking an oracle executable
on Sun Solaris. This applies to all executbles including Database, Networking,
Tools,and Applications.
ld.so.1: : fatal: libucb.so.1: open failed:
No such file or directory
Killed
PROBLEM EXPLANATION
======================
The executable was built using the wrong linker ('ld'). The Solaris
operating system comes with two linkers, the SVR4 linker (/usr/ccs/bin/ld)
and SunOS 4.x compatibility linker (/usr/ucb/ld). Oracle requires the use of
the /usr/ccs/bin utitlities to properly link the executables and
if /usr/ucb/ld is used instead of /usr/ccs/bin/ld, executables will fail with
the above error message.
NOTE: Please see the appropriate product Installation Guide for
O/S requirments.
SOLUTION DESCRIPTION
=====================
To correct the problem, you'll need to modify your PATH environment variable
so that the /usr/ccs/bin directory comes before /usr/ucb directory, and then
relink the executables that were linked with the wrong linker.
1. Correct the PATH environment variable.
csh
-----
% setenv PATH /usr/ccs/bin:${PATH}
sh, ksh or bash
-----------
% PATH=/usr/ccs/bin:$PATH
% export PATH
2. Ensure the PATH is set correctly.
% which ld
/usr/ccs/bin/ld <== This should be returned if the the PATH is
correct.
3. Relink the executables:
See Note 131321.1 for Data Server relinking instructions. I am not sure if PATH was incorrectly set when the binaries were relinked. However, I have advised the DBA team to relink all binaries based on the above analysis.
Monday, November 26, 2007
ORA-1403 when forms is launched from self service
In one of our instances, we are getting this error in Firefox whenever we try to launch forms from self service modules:
ORA-01403: no data found Content-type: text/html; charset=UTF-8 Location: http://justanexample.com:8000/oa_servlets/oracle.apps.fnd.sso.AppsLogin?requestUrl=APPSHOMEPAGE&cancelUrl=http%3A%2F%2F11ioltp.justanexample.com%3A8013%2Foa_servlets%2Foracle.apps.fnd.sso.AppsLogin&errText=Your%20session%20is%20no%20longer%20valid.%20Please%20login%20again.
This error only appears when we try the option from Firefox. Internet Explorer is silent and doesn't show any error.
I have asked the DBA team to enable trace on ORA-01403 in init.ora through this event to find out:
event = "1403 trace name errorstack: 10046 trace name context forever, level 12"
Some data purging had been done by DBAs, after which this error is occuring.
After trace was enabled, this is what we found in trace file:
PARSING IN CURSOR #3 len=272 dep=1 uid=173 oct=3 lid=173 tim=5586109696263 hv=1880614960 ad='9ab642f8'
SELECT FPOV.PROFILE_OPTION_VALUE FROM FND_PROFILE_OPTION_VALUES FPOV, FND_PROFILE_OPTIONS FPO WHERE FPO.PROFILE_OPTION_NAME =
'AFCORE_LOGGING_ENABLED' AND FPO.PROFILE_OPTION_ID = FPOV.PROFILE_OPTION_ID AND FPOV.LEVEL_ID = 10001 AND FPO.APPLICATION_ID
= FPOV.APPLICATION_ID
END OF STMT
PARSE #3:c=0,e=165,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696257
BINDS #3:
EXEC #3:c=0,e=126,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696528
FETCH #3:c=0,e=101,p=0,cr=5,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696670
*** 2007-11-26 15:39:02.788
ksedmp: internal or fatal error
ORA-01403: no data found
Current SQL statement for this session:
begin
owa.user_id := :userid;
owa.password := :passwd;
owa.ip_address := :ipaddress;
owa.hostname := :hostname;
owa.init_cgi_env(:n__, :nm__, :v__);
if (owa_custom.authorize = TRUE) then
:authorized := 'yes';
else
:authorized := 'no';
end if;
:realm := owa.protection_realm;
end;
----- PL/SQL Call Stack -----
object line object
handle number name
398b4f6f0 23 package body APPS.FND_CORE_LOG
398b4f6f0 158 package body APPS.FND_CORE_LOG
39cbe1d38 213 package body APPS.FND_PROFILE
39cbe1d38 1548 package body APPS.FND_PROFILE
39cbe1d38 1720 package body APPS.FND_PROFILE
39cbe1d38 2357 package body APPS.FND_PROFILE
398b8bbd0 35 package body APPS.FND_WEB_CONFIG
398b8bbd0 214 package body APPS.FND_WEB_CONFIG
398b8bbd0 411 package body APPS.FND_WEB_CONFIG
39b7cdf08 33 package body APPS.OWA_CUSTOM
39b7d8b00 7 anonymous block
----- Call Stack Trace -----
Still investigating this one.
ORA-01403: no data found Content-type: text/html; charset=UTF-8 Location: http://justanexample.com:8000/oa_servlets/oracle.apps.fnd.sso.AppsLogin?requestUrl=APPSHOMEPAGE&cancelUrl=http%3A%2F%2F11ioltp.justanexample.com%3A8013%2Foa_servlets%2Foracle.apps.fnd.sso.AppsLogin&errText=Your%20session%20is%20no%20longer%20valid.%20Please%20login%20again.
This error only appears when we try the option from Firefox. Internet Explorer is silent and doesn't show any error.
I have asked the DBA team to enable trace on ORA-01403 in init.ora through this event to find out:
event = "1403 trace name errorstack: 10046 trace name context forever, level 12"
Some data purging had been done by DBAs, after which this error is occuring.
After trace was enabled, this is what we found in trace file:
PARSING IN CURSOR #3 len=272 dep=1 uid=173 oct=3 lid=173 tim=5586109696263 hv=1880614960 ad='9ab642f8'
SELECT FPOV.PROFILE_OPTION_VALUE FROM FND_PROFILE_OPTION_VALUES FPOV, FND_PROFILE_OPTIONS FPO WHERE FPO.PROFILE_OPTION_NAME =
'AFCORE_LOGGING_ENABLED' AND FPO.PROFILE_OPTION_ID = FPOV.PROFILE_OPTION_ID AND FPOV.LEVEL_ID = 10001 AND FPO.APPLICATION_ID
= FPOV.APPLICATION_ID
END OF STMT
PARSE #3:c=0,e=165,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696257
BINDS #3:
EXEC #3:c=0,e=126,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696528
FETCH #3:c=0,e=101,p=0,cr=5,cu=0,mis=0,r=0,dep=1,og=4,tim=5586109696670
*** 2007-11-26 15:39:02.788
ksedmp: internal or fatal error
ORA-01403: no data found
Current SQL statement for this session:
begin
owa.user_id := :userid;
owa.password := :passwd;
owa.ip_address := :ipaddress;
owa.hostname := :hostname;
owa.init_cgi_env(:n__, :nm__, :v__);
if (owa_custom.authorize = TRUE) then
:authorized := 'yes';
else
:authorized := 'no';
end if;
:realm := owa.protection_realm;
end;
----- PL/SQL Call Stack -----
object line object
handle number name
398b4f6f0 23 package body APPS.FND_CORE_LOG
398b4f6f0 158 package body APPS.FND_CORE_LOG
39cbe1d38 213 package body APPS.FND_PROFILE
39cbe1d38 1548 package body APPS.FND_PROFILE
39cbe1d38 1720 package body APPS.FND_PROFILE
39cbe1d38 2357 package body APPS.FND_PROFILE
398b8bbd0 35 package body APPS.FND_WEB_CONFIG
398b8bbd0 214 package body APPS.FND_WEB_CONFIG
398b8bbd0 411 package body APPS.FND_WEB_CONFIG
39b7cdf08 33 package body APPS.OWA_CUSTOM
39b7d8b00 7 anonymous block
----- Call Stack Trace -----
Still investigating this one.
Subscribe to:
Posts (Atom)
Posts
