R12.2 Documentation link in html format

This link has the R12.2 documentation in HTML format:

https://docs.oracle.com/cd/E26401_01/index.htm 

The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes

I got this error while upgrading an R12.1.3 instance to R12.2.4, when I completed AD.C.Delta 5 patches with November 2014 bundle patches for AD.C and was in the process of applying TXK.C.Delta5 with November 2014 bundle patches for TXK.C :

Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
[START 2015/03/01 04:53:16] Check if services are down
        [INFO] Run admin server is not down
     [WARNING]  Hotpatch mode should only be used when directed by the patch readme.
  [EVENT]     [START 2015/03/01 04:53:17] Performing database sanity checks
    [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .
    Log file: /erppgzb1/erpapp/fs_ne/EBSapps/log/adop/adop_20150301_045249.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)

I was really surprised as I had already run EBS technology codelevel checker (patch 17537119) script checkDBpatch.sh on racnode1.

To investigate I checked inside checkDBpatch.sh and found that it create a table called TXK_TCC_RESULTS.  

SQL> desc txk_tcc_results

 Name                                      Null?    Type

 ----------------------------------------- -------- ----------------------------

 TCC_VERSION                               NOT NULL VARCHAR2(20)

 BUGFIX_XML_VERSION                        NOT NULL VARCHAR2(20)

 NODE_NAME                                 NOT NULL VARCHAR2(100)

 DATABASE_NAME                             NOT NULL VARCHAR2(64)

 COMPONENT_NAME                            NOT NULL VARCHAR2(10)

 COMPONENT_VERSION                         NOT NULL VARCHAR2(20)

 COMPONENT_HOME                                     VARCHAR2(600)

 CHECK_DATE                                         DATE

 CHECK_RESULT                              NOT NULL VARCHAR2(10)

 CHECK_MESSAGE                                      VARCHAR2(4000)

SQL> select node_name from txk_tcc_results;

NODE_NAME

--------------------------------------------------------------------------------

RACNODE1

I ran checkDBpatch.sh again, but the patch failed again with previous error:

   [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .

It was Saturday 5 AM already working through the night.  So I thought, it is better to sleep now and tackle this on Sunday.  On Sunday morning after a late breakfast, I looked at the problem again.  This time, I realized that the error was complaining about racnode1 (in lower case) and the txk_tcc_results table had RACNODE1(in upper case).  To test my hunch, I immediately updated the value:

update txk_tcc_results

set node_name='racnode1' where node_name='RACNODE1';

commit;

I restarted the patch, and it went through.  Patch was indeed failing because it was trying to look for a lower case value.  I will probably log an SR with Oracle, so that they change their code to make the node_name check case insensitive.

Further, I was curious, why node_name was stored in all caps in fnd_nodes and txk_tcc_results.  The file /etc/hosts had it in lowercase.  I tried the hostname command on linux prompt:

$ hostname
RACNODE1

That was something unusual, as in our environment, hostname always returns the value in lowercase.  So I further investigated.
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = RACNODE1

So I changed it

[root@RACNODE1 ~]# sysctl kernel.hostname=RACNODE1
kernel.hostname = racnode1
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = racnode1
[root@RACNODE1 ~]#
[root@RACNODE1 ~]# hostname
racnode1

Logged in again to see if root prompt changed:

[root@racnode1 ~]#

I also checked
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=RACNODE1

Changed it here also:
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=racnode1

I also changed it on racnode2.

cannot set user id: Resource temporarily unavailable or Fork: Retry: Resource Temporarily Unavailable

Amjad reported this error while trying to login to the server:

cannot set user id: Resource temporarily unavailable

In the past he had reported this error:

Fork: Retry: Resource Temporarily Unavailable

This is due to the fact that the user has run out of free stacks.  In OEL 6.x , the stack setting is not done in /etc/security/limits.conf but in the file:

/etc/security/limits.d/90-nproc.conf

The default content in the file is:

cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited

I changed this to:

After

$ cat /etc/security/limits.d/90-nproc.conf

# Default limit for number of user's processes to prevent

# accidental fork bombs.

# See rhbz #432903 for reasoning.

*          soft    nproc     16384

root       soft    nproc     unlimited

$

As soon as this change was made, Amjad was able to login.

ERROR - CLONE-20372 Server port validation failed

Alok and Shoaib pinged me about this error. This error is reported in logs when adcfgclone.pl is run for a R12.2.4 appsTier where the source and target instances are on same physical server.

SEVERE : Jan 27, 2015 3:40:09 PM - ERROR - CLONE-20372   Server port validation failed.
SEVERE : Jan 27, 2015 3:40:09 PM - CAUSE - CLONE-20372   Ports of following servers - oacore_server2(7256),forms_server2(7456),oafm_server2(7656),forms-c4ws_server2(7856),oaea_server1(6856) - are not available.
4:00 PM
SEVERE : Jan 27, 2015 3:40:09 PM - ERROR - CLONE-20372   Server port validation failed.
SEVERE : Jan 27, 2015 3:40:09 PM - CAUSE - CLONE-20372   Ports of following servers - oacore_server2(7256),forms_server2(7456),oafm_server2(7656),forms-c4ws_server2(7856),oaea_server1(6856) - are not available.
SEVERE : Jan 27, 2015 3:40:09 PM - ACTION - CLONE-20372   Provide valid free ports.
oracle.as.t2p.exceptions.FMWT2PPasteConfigException: PasteConfig failed. Make sure that the move plan and the values specified in moveplan are correct

The ports reported are those in the source instance.  Searching on support.oracle.com bug database I found three articles:

EBS 12.2.2.4 RAPID CLONE FAILS WITH ERROR - CLONE-20372 SERVER PORT VALIDATION(Bug ID 20147454)

12.2: N->1 CLONING TO SAME APPS TIER FAILING DUE TO PORT CONFLICT(Bug ID 20389864)

FS_CLONE IS NOT ABLE TO COMPLETE FOR MULTI-NODE SETUP(Bug ID 18460148)

The situation described in the first two bugs is same.  The articles reference each other but don't provide any solution.

Logically thinking, adcfgclone.pl is picking this up from source configuration that is in $COMMON_TOP/clone directory.  So we did grep on subdirectories of $COMMON_TOP/clone:

cd $COMMON_TOP/clone
find . -type f -print | xargs grep 7256

7256 is one of the ports that failed validation.

It is present in

CTXORIG.xml and
FMW/ohs/moveplan.xml
FMW/wls/moveplan.xml

We tried changing the port numbers in CTXORIG.xml and re-tried adcfgclone.pl and it failed again.

So we changed the port numbers of the ports that failed validation in

$COMMON_TOP/clone/FMW/ohs/moveplan.xml and
$COMMON_TOP/clone/FMW/wls/moveplan.xml

cd $FMW_HOME
find . -name detachHome.sh |grep -v Template

The above command returns the detachHome.sh scripts for all the ORACLE_HOMEs inside FMW_HOME.  Executed this to detach all of them.

Removed the FMW_HOME directory

Re-executed
adcfgclone.pl appsTier

It succeeded this time.  Till we get a patch for this bug, we will continue to use this workaround to complete clones.

ERROR: The following required ports are in use: 6801 : WLS OAEA Application Port

Anil pinged me today when his adop phase=fs_clone failed with this error message:

-----------------------------
ERROR: The following required ports are in use:
-----------------------------
6801 : WLS OAEA Application Port
Corrective Action: Free the listed ports and retry the adop operation.

Completed execution : ADOPValidations.java

====================================
Inside _validateETCHosts()...
====================================

This is a bug mentioned in the appendix of article: 

Integrating Oracle E-Business Suite Release 12.2 with Oracle Access Manager 11gR2 (11.1.2) using Oracle E-Business Suite AccessGate (Doc ID 1576425.1)

Bug 19817016

The following errors are encountered when running fs_clone after completing AccessGate and OAM integration and after completing a patch cycle:

Checking  WLS OAEA Application Port on aolesc11:  Port Value = 6801
RC-50204: Error: - WLS OAEA Application Port in use: Port Value = 6801

-----------------------------
ERROR: The following required ports are in use:
-----------------------------
6801 : WLS OAEA Application Port
Corrective Action: Free the listed ports and retry the adop operation.

Workaround:
Stop the oaea managed server on the run file system before performing the fs_clone operation, immediately after the accessgate deployment.

Solution:
This issue will be addressed through Bug 19817016.

If you read the bug:

Bug 19817016 : RUNNING ADOP FS_CLONE FAILS DUE TO PORT CONFLICT BETWEEN RUN AND PATCH EDITION	To Bottom

Bug Attributes

---

Type	B - Defect	Fixed in Product Version
Severity	2 - Severe Loss of Service	Product Version	12.2.4
Status	11 - Code/Hardware Bug (Response/Resolution)	Platform	226 - Linux x86-64
Created	14-Oct-2014	Platform Version	ORACLE LINUX 5
Updated	02-Dec-2014	Base Bug	N/A
Database Version	11.2.0.3	Affects Platforms	Generic
Product Source	Oracle	Knowledge, Patches and Bugs related to this bug

Related Products

---

Line	Oracle E-Business Suite	Family	Applications Technology
Area	Technology Components	Product	1745 - Oracle Applications Technology Stack

Hdr: 19817016 11.2.0.3 FSOP 12.2.4 PRODID-1745 PORTID-226
Abstract: RUNNING ADOP FS_CLONE FAILS DUE TO PORT CONFLICT BETWEEN RUN AND PATCH EDITION
 
*** 10/14/14 11:58 am ***
Service Request (SR) Number:
----------------------------
 
 
Problem Statement:
------------------
Running fs_clone after completing EBS and OAM integration and after 
completing a patch cycle results in fs_clone failing with the following 
errors:
 
Checking  WLS OAEA Application Port on aolesc11:  Port Value = 6801
RC-50204: Error: - WLS OAEA Application Port in use: Port Value = 6801
 
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
6801 : WLS OAEA Application Port
Corrective Action: Free the listed ports and retry the adop operation.
 
Detailed Results of Problem Analysis:
-------------------------------------
The problem is due to the newly added managed server port being the same for 
both the run and patch edition.  Going back to the sequence of steps and 
tracking the port assignment, it showed the following:
 
- deploy accessgate on patch
Creates managed server - oaea_server1:6801
This is the default port and doing this to the patch edition...
 
fs2 - run -> 6801 port
fs1 - patch -> 6801 port
 
- complete OAM registration
- close patching cycle
- cutover
- after cutover, SSO is working
 
fs1 - run -> 6801 port
fs2 - patch -> 6801 port
 
- fs_clone -> fails due to both run(fs1) and patch(fs2) referencing the same 
port 6801
 
Configuration and Version Details:
----------------------------------
OAM - 11.1.2.2.0
WG - 11.1.2.2.0
EAG - 1.2.3
WT - 11.1.1.6.0
 
EBS 12.2.4 w/ AD/TXK delta 5
 
Steps To Reproduce:
-------------------
As part of the EBS integration w/ OAM, we add a managed server for use as the 
EBS AccessGate (EAG) to the existing WLS in EBS.  There is an option to do 
this to both run edition, as well as the patch edition during an active patch 
cycle.  In this case the latter was done.  Here is a summary of the steps 
used:
 
1. Start patch cycle
2. Integrated OID and EBS
3. Cutover
4. Confirmed OID provisioning is working
5. Start patch cycle
6. Apply pre-req EBS patches for OAM
7. Proceed w/ OAM integration on patch file system
8. Cutover
9. Confirmed SSO/OAM is working
10. Run fs_clone -> this is where the issue appears
 
 
Additional Information:
-----------------------
The workaround here is to stop the oaea_server1 managed server operating in 
the run edition on port 6801, and then re-running fs_clone.  Once this is 
done, fs_clone completes and the patch edition now operates on port 6802 for 
the same managed server.
 
For A Severity 1 Bug: Justification and 24x7 Contact Details:
-------------------------------------------------------------
 
 
*** 10/14/14 01:19 pm ***
*** 10/16/14 07:05 am ***
*** 10/16/14 07:05 am ***
*** 10/17/14 01:47 am ***
*** 10/17/14 01:49 am *** 
*** 10/17/14 01:57 am ***
*** 10/17/14 08:47 am ***
*** 10/23/14 12:16 am ***
*** 10/23/14 12:17 am ***
*** 10/26/14 10:07 pm ***
*** 10/27/14 10:06 pm ***
*** 10/27/14 10:09 pm ***
*** 10/30/14 10:40 pm ***
*** 10/30/14 10:49 pm ***
*** 10/30/14 10:49 pm ***
*** 11/05/14 04:30 pm *** 
*** 11/05/14 04:30 pm ***
*** 11/06/14 10:59 am *** 
*** 11/17/14 09:20 pm ***
*** 12/02/14 12:36 am ***
*** 12/02/14 07:26 pm ***

Till a patch is made available, you need to shutdown the oaea managed server and restart fs_clone. So much for keeping all services online and the promise of no outage during fs_clone.

Table TXK_TCC_RESULTS needs to be installed by running the EBS Technology Codelevel Checker (available as patch 17537119).

I got this error while trying to apply a patch in R12.2:

 [EVENT]     [START 2014/11/19 09:18:39] Performing database sanity checks
   [ERROR]     Table TXK_TCC_RESULTS needs to be installed by running the EBS Technology Codelevel Checker (available as patch 17537119).

This table TXK_TCC-RESULTS is created in APPLSYS schema, by the latest version of the checkDBpatch.sh script delivered by 17537119.

So go ahead, download patch 17537119.  

Login as oracle user on your database node.

Source environment

cd $ORACLE_HOME/appsutil

unzip p17537119*

$ ./checkDBpatch.sh 

+===============================================================+ 

| Copyright (c) 2005, 2014 Oracle and/or its affiliates. | 

| All rights reserved. | 

| EBS Technology Codelevel Checker | 

+===============================================================+ 

Executing Technology Codelevel Checker version: 120.18 

Enter ORACLE_HOME value : /exampler122/oracle/11.2.0 

Enter ORACLE_SID value : exampler122

Bugfix XML file version: 120.0.12020000.16 

Proceeding with the checks... 

Getting the database release ... 

Setting database release to 11.2.0.3 

DB connectivity successful. 

The given ORACLE_HOME is RAC enabled. 

NOTE: For a multi-node RAC environment 

- run this tool on all non-shared ORACLE_HOMEs. 

- run this tool on one of the shared ORACLE_HOMEs. 

Created the table to store Technology Codelevel Checker results. 

STARTED Pre-req Patch Testing : Wed Nov 19 10:53:00 EST 2014 

Log file for this session : ./checkDBpatch_7044.log 

Got the list of bug fixes to be applied and the ones to be rolled back. 

Checking against the given ORACLE_HOME 

Opatch is at the required version. 

Found patch records in the inventory. 

All the required one-offs are present in Oracle Database Home 

Stored Technology Codelevel Checker results in the database successfully. 

FINISHED Pre-req Patch Testing : Wed Nov 19 10:53:03 EST 2014 

========================================================= 

1 select owner,table_name from dba_tables 

2* where table_name='TXK_TCC_RESULTS' 

SQL> / 

OWNER TABLE_NAME 

------------------------------ ------------------------------ 

APPLSYS TXK_TCC_RESULTS 

SQL> 

Once you have done this, restart your patch with adop with additional parameter restart=yes