The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes

I got this error while upgrading an R12.1.3 instance to R12.2.4, when I completed AD.C.Delta 5 patches with November 2014 bundle patches for AD.C and was in the process of applying TXK.C.Delta5 with November 2014 bundle patches for TXK.C :

Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
[START 2015/03/01 04:53:16] Check if services are down
        [INFO] Run admin server is not down
     [WARNING]  Hotpatch mode should only be used when directed by the patch readme.
  [EVENT]     [START 2015/03/01 04:53:17] Performing database sanity checks
    [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .
    Log file: /erppgzb1/erpapp/fs_ne/EBSapps/log/adop/adop_20150301_045249.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)

I was really surprised as I had already run EBS technology codelevel checker (patch 17537119) script checkDBpatch.sh on racnode1.

To investigate I checked inside checkDBpatch.sh and found that it create a table called TXK_TCC_RESULTS.  

SQL> desc txk_tcc_results

 Name                                      Null?    Type

 ----------------------------------------- -------- ----------------------------

 TCC_VERSION                               NOT NULL VARCHAR2(20)

 BUGFIX_XML_VERSION                        NOT NULL VARCHAR2(20)

 NODE_NAME                                 NOT NULL VARCHAR2(100)

 DATABASE_NAME                             NOT NULL VARCHAR2(64)

 COMPONENT_NAME                            NOT NULL VARCHAR2(10)

 COMPONENT_VERSION                         NOT NULL VARCHAR2(20)

 COMPONENT_HOME                                     VARCHAR2(600)

 CHECK_DATE                                         DATE

 CHECK_RESULT                              NOT NULL VARCHAR2(10)

 CHECK_MESSAGE                                      VARCHAR2(4000)

SQL> select node_name from txk_tcc_results;

NODE_NAME

--------------------------------------------------------------------------------

RACNODE1

I ran checkDBpatch.sh again, but the patch failed again with previous error:

   [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .

It was Saturday 5 AM already working through the night.  So I thought, it is better to sleep now and tackle this on Sunday.  On Sunday morning after a late breakfast, I looked at the problem again.  This time, I realized that the error was complaining about racnode1 (in lower case) and the txk_tcc_results table had RACNODE1(in upper case).  To test my hunch, I immediately updated the value:

update txk_tcc_results

set node_name='racnode1' where node_name='RACNODE1';

commit;

I restarted the patch, and it went through.  Patch was indeed failing because it was trying to look for a lower case value.  I will probably log an SR with Oracle, so that they change their code to make the node_name check case insensitive.

Further, I was curious, why node_name was stored in all caps in fnd_nodes and txk_tcc_results.  The file /etc/hosts had it in lowercase.  I tried the hostname command on linux prompt:

$ hostname
RACNODE1

That was something unusual, as in our environment, hostname always returns the value in lowercase.  So I further investigated.
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = RACNODE1

So I changed it

[root@RACNODE1 ~]# sysctl kernel.hostname=RACNODE1
kernel.hostname = racnode1
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = racnode1
[root@RACNODE1 ~]#
[root@RACNODE1 ~]# hostname
racnode1

Logged in again to see if root prompt changed:

[root@racnode1 ~]#

I also checked
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=RACNODE1

Changed it here also:
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=racnode1

I also changed it on racnode2.