autoconfig fails on adgendbc.sh with ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution. Errors are grouped by directory and phase.
The report format is:


[SETUP PHASE]
AutoConfig could not successfully execute the following scripts:
Directory: $COMMON_TOP/admin/install/$TWO_TASK
adgendbc.sh INSTE8_SETUP 1


AutoConfig is exiting with status 1

When we tried running adgendbc.sh manually, these errors occurred:

Error occurred during initialization of VM
Could not reserve enough space for object heap
UPDATE call failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
Error occurred during initialization of VM
Could not reserve enough space for object heap
Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
Error occurred during initialization of VM
Could not reserve enough space for object heap
Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
Error occurred during initialization of VM
Could not reserve enough space for object heap
DBC generation failed with exit code 1

Based on Metalink Note 394550.1:

Cause

The issue is caused java caching feature enabled.

The following parameter was true in jserv.properties

wrapper.bin.parameters=-DLONG_RUNNING_JVM=true



.

Solution

To implement the solution, please execute the following steps:

1.Take a backup of jserv.properties.
2.Edit the value DLONG_RUNNING_JVM to false.
3.The following should be the change...
wrapper.bin.parameters=-DLONG_RUNNING_JVM=true
to
wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

I modifed jserv.properties and re-ran adgendbc.sh. New errors came up this time:

adgendbc.sh started at Fri Jul 11 14:18:47 EDT 2008


Enter the APPS username: apps

Enter the APPS password:

SQL*Plus: Release 8.0.6.0.0 - Production on Fri Jul 11 14:18:54 2008

(c) Copyright 1999 Oracle Corporation. All rights reserved.

Enter value for 1: Enter value for 2: Enter value for 3: Connected.
Updated profile option value - 1 row(s) updated
Application Id : 0
Profile Name : APPS_DATABASE_ID
Level Id : 10001
New Value : $CONTEXT_NAME
Old Value : $CONTEXT_NAME

PL/SQL procedure successfully completed.


Commit complete.

Disconnected from Oracle9i Enterprise Edition Release 9.2.0.5.0 - 64bit Producti on
With the Partitioning option
JServer Release 9.2.0.5.0 - Production


Params=fnd_jdbc_stmt_cache_free_mem=TRUE
fnd_jdbc_buffer_min=1
fnd_jdbc_buffer_max=5
fnd_jdbc_buffer_decay_interval=300
fnd_jdbc_buffer_decay_size=5
fnd_jdbc_usable_check=false
fnd_jdbc_context_check=true
fnd_jdbc_plsql_reset=false


Unique constraint error (00001) is OK if key already exists
Application server ID already exists for this host - loading..
ADD executed successfully - $FND_SECURE/$CONTEXT_NAME.dbc

Trying to update information ...

java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

UPDATE call failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Generating $FND_SECURE/$TWO_TASK.domainname_$TWO_TASK.dbc
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Generating $FND_SECURE/$CONTEXT_NAME.dbc
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Updating Server Security Authentication
java.sql.SQLException: Io exception: Invalid number format for port number
Database connection to jdbc:oracle:thin:@host_name:port_number:database failed
Updating Server Security Authentication failed with exit code 1
adgendbc.sh exiting with status 1
ERRORCODE = 1 ERRORCODE_END

We saw various metalink notes which pointed to similar errors. We also upgraded to TXK Rollup S to get rid of the error, but to no avail. The common error during the run of adgendbc.sh was:

APPS.FND_APP_SERVER_PKG", line 357

So I decided to check what is present in line 357:

SQL> select line,text
2 from dba_source
3 where line between 357 and 370
4 and name='FND_APP_SERVER_PKG';

LINE
----------
TEXT
--------------------------------------------------------------------------------
357
select SUPPORT_CP, SUPPORT_FORMS, SUPPORT_WEB, SUPPORT_ADMIN,

358
SUPPORT_DB, PLATFORM_CODE

359
into l_support_cp, l_support_forms, l_support_web, l_support_admin,


LINE
----------
TEXT
--------------------------------------------------------------------------------
360
l_support_db, l_platform

361
from fnd_nodes

362
where server_address = p_address;


Based on the above query I checked for the value of server_address in fnd_nodes and found that two node had the same IP address as per fnd_nodes:

SQL> select node_name,server_address
2 from fnd_nodes;

NODE_NAME SERVER_ADDRESS
------------------------------ ------------------------------
PHYSICALDB 192.168.70.29
VIRTUALDB 192.168.70.29
AUTHENTICATION *
APPTIER 192.168.70.69

I pinged each server and found that the correct IP address of Physical DB was different. So I changed it immediately.

SQL> update fnd_nodes
2 set server_address='192.168.70.31'
3 where node_name='PHYSICALDB';

1 row updated.

SQL> commit;

Commit complete.

After this adgendbc.sh ran without errors.