R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade

This problem occurs during rapidwiz where it fails to create apache instance and the error is logged in deploytech.log:

from deploytech.log:

ERROR: Failed to create the EBS domain.
Failed to deploy the E-Business Suite domain

Oracle would usually advise you to re-install. However I found a way to do it by tracking the rapidwiz log files to see what all scripts it calls to create it manually:   

Create missing apache instance, not created by rapidwiz

Reference :

. ./EBSapps.env run

cd $FMW_HOME/opmnctl/bin

./opmnctl createinstance -oracleInstance $FMW_HOME/webtier/instances/EBS_web_ebs122_OHS1

-adminhost admin.justanexample.com -adminPort 7002 -adminRegistration OFF

Creating Oracle Instance directories...Done

Recording OPMN ports reservations...Done

Bootstrapping OPMN configuration files...Done

Instantiating opmnctl for direct usage...Done

Skipping instance registration

Command succeeded.

./opmnctl createcomponent -componentType OHS -componentName EBS_web_ebs122

-oracleInstance $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1

Creating empty component directories...Done

Provisioning OHS files for EBS_web_ebs122

  Copying OHS files from ORACLE_HOME to ORACLE_INSTANCE locations

  Customizing httpd.conf

  Adding component's process control to OPMN...Done

Skipping EBS_web_ebs122 component registration.

Command succeeded.

cd $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/bin

./opmnctl registerinstance –adminhost admin.justanexample.com -adminPort 7002

Command requires login to weblogic admin server (alpcmfapq11.comfin.ge.com):

  Username: weblogic

  Password:

Deploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl admin.justanexample.com:7002 -username weblogic -name NonJ2EEManagement -source $FMW_HOME/webtier/opmn/applications/NonJ2EEManagement.ear -nostage -deploy -upload -noexit

Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Target state: deploy completed on Server AdminServer

Done

Registering instance

Command succeeded.

Copy these *.conf files to $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/config/OHS/EBS_web_ebs122 obtained from Vision instance :

admin.conf
apps.conf
custom.conf
httpd.conf
mod_wl_ohs.conf
oracle_apache.conf
oracle_apache_ssl.conf
security2.conf
security2_dmz.conf
ssl.conf
ssl_terminator.conf
trusted.conf
url_fw.conf
url_fw_ws.conf
webgate.conf

Global replace the server name and port in the files above.

Run adconfig to regenerate these files

Instantiate the Oracle HTTP Server configuration files admin.conf, ssl.conf, mod_wl_ohs.conf on Run File System.
Note: Since the templates are being instantiated, all customizations will be lost and have to be restored explicitly.
Perform the instantiation with the following commands (the backslash at the end of each line is being used to divide each command across several screen lines for readability). When prompted for the APPS password, press .

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/ssl_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/mod_wl_ohs_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/admin_conf_FMW.tmp

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-outfile=$INST_TOP/logs/appl/rgf/TXK/test.log

apps.conf generated successfully

mod_wl_ohs.conf generated successfully
 

 

R12.2 Dictionary Corrupted error during adop phase=prepare

Do not follow the document 1576086.1.  I followed it and increased my invalid counts from 1000 to 100,000 and wasted 10 hours to get back to where I started.  Instead follow these instructions:

Reference : Ora-04023 Reported while Accessing Valid Objects (Doc ID 1610514.1)

Dictionary Corrupted:

Details are as follows:

OWBSYS          HAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          WB_RTI_OBJECT_CATALOG                          OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

PUBLIC          WB_OLAP_AW_PRECOMPUTE                          OWBSYS          WB_OLAP_AW_PRECOMPUTE                          TS mismatch: 23-SEP-10 05:17:34 26-NOV-12 18:48:25

OWBSYS          ALL_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          RAB_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

please contact Oracle Support and request them to open a bug against Oracle Application Install(166), Component Online Patching(OP)

        [ERROR]     Prepare phase completed with errors/warnings. Please check logfiles

        Log file: /erpodpa1/erpapp/fs_ne/EBSapps/log/adop/3/adop_20140306_005850.log

select owner,object_type,object_name

from dba_objects

where object_name in

(

'HAB_RT_EXEC_PROC_RUN_METRICS',

'WB_RTI_OBJECT_CATALOG',

'UAB_RT_EXEC_PROCESS',

'UAB_RT_EXEC_PROCESS_RUNS',

'UAB_RT_EXEC_PROC_RUN_COUNTS',

'UAB_RT_EXEC_PROC_RUN_METRICS',

'WB_OLAP_AW_PRECOMPUTE',

'ALL_RT_INSTALLATIONS',

'RAB_RT_INSTALLATIONS',

'HAB_RT_EXEC_PROCESS',

'HAB_RT_EXEC_PROCESS_RUNS',

'HAB_RT_EXEC_PROC_RUN_COUNTS'

);

Select against a view  fails with following error.

SQL> Select * from . ;

ORA-04023: could not be validated or authorized

This is issue is due to a timestamp inconsistency in the data dictionary. This timestamp inconsistency will lead to the problem when we  populate the relevant cached objects and later try to select from the affected views.

Run following script to find out the objects which are having timestamp discrepencies.

Connect as / as sysdba

SQL> @?/rdbms/admin/utldtchk.sql

This script is available from 11g onwards.

Recompile all the objects manually returned by this script.

Conn owbsys/owbsys

alter view ALL_RT_INSTALLATIONS compile;

alter view RAB_RT_INSTALLATIONS compile;

alter view HAB_RT_EXEC_PROCESS compile;

alter view HAB_RT_EXEC_PROCESS_RUNS compile;

alter view HAB_RT_EXEC_PROC_RUN_COUNTS compile;

alter view HAB_RT_EXEC_PROC_RUN_METRICS compile;

alter view UAB_RT_EXEC_PROCESS compile;

alter view UAB_RT_EXEC_PROCESS_RUNS compile;

alter view UAB_RT_EXEC_PROC_RUN_COUNTS compile;

alter view UAB_RT_EXEC_PROC_RUN_METRICS compile;

alter function WB_OLAP_AW_PRECOMPUTE compile;

drop public synonym WB_OLAP_AW_PRECOMPUTE;

CREATE PUBLIC SYNONYM WB_OLAP_AW_PRECOMPUTE

FOR OWBSYS.WB_OLAP_AW_PRECOMPUTE;

It will remove the timestamp inconsistency in data dictionary and resolve this issue.

Verify by re-running the script, it should return zero rows.

SQL> @?/rdbms/admin/utldtchk.sql

If you still see occurances, bounce the database and rerun the above script.