Correct permissions on /var/tmp

/var/tmp is the default Solaris swap mount. If you notice it has an extra t in its permissions:

$ ls -ld /var/tmp
drwxrwxrwt 16 root sys 8192 Jul 17 17:54 /var/tmp
$

If you ever see /var/tmp without this t

drwxrwxrw 16 root sys 8192 Jul 17 17:54 /var/tmp

These are the permissions and ownership /var/tmp should have:

sudo chown root:root /var/tmp
sudo chmod 1777 /var/tmp

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

After the upgrade from 11.5.9 / 9.2.0.8 to 11.5.10.2/10.2.0.3 we are getting errors in DB alert logs:

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

A search through ORA-600 lookup tool turns up this note:

Subject:	ORA-600 [kcbz_check_objd_typ_3]
	Doc ID:	Note:406787.1	Type:	REFERENCE
	Last Revision Date:	10-JUL-2008	Status:	PUBLISHED

Note: For additional ORA-600 related information please read Note 146580.1

PURPOSE:
 This article represents a partially published OERI note.

 It has been published because the ORA-600 error has been
 reported in at least one confirmed bug.

 Therefore, the SUGGESTIONS section of this article may help
 in terms of identifying the cause of the error.

 This specific ORA-600 error may be considered for full publication
 at a later date. If/when fully published, additional information
 will be available here on the nature of this error.

SUGGESTIONS:

 If the Known Issues section below does not help in terms of identifying
 a solution, please submit the trace files and alert.log to Oracle
 Support Services for further analysis.

 Known Issues:
 Bug# 6405339   See Note 6405339.8
     Corruption / OERI[kcbz_check_objd_typ_3] / OERI:5400 from PQ
     Fixed: 10.2.0.5, 11.1.0.7, 11.2

 Bug# 5689412   See Note 5689412.8
     ORA-600 [kcbz_check_objd_typ_3] after SCN based recovery
     Fixed:

 Bug# 4592596   See Note 4592596.8
     Corruption (ORA-1410) from multi-table insert with direct load
     Fixed: 10.2.0.4, 11.1.0.6

 Bug# 4430244 +  See Note 4430244.8
     Segment advisor can load blocks of dropped objects into buffer cache (KCB OERI errors)
     Fixed: 10.2.0.4, 11.1.0.6

Metalink Note 466049.1 describes summary of Bugs Containing ORA - 00600[kcbz_check_objd_typ_3]

Purpose

This article is intended to

1. Help the reader to understand ORA-600 [kcbz_check_objd_typ_3] error and how is this caused.

2. List the most common bugs of the aforesaid error and their fixes.

Scope

This is a consolidated article based upon various bugs logged for the error
ORA-600 [kcbz_check_objd_typ_3] and gives the reader an overview about the most frequent bugs which the customers are affected with.

The error ORA-600 [kcbz_check_objd_typ_3] is raised because we see different object id in bufffer cache (BH) and on disk .

Summary of Bugs Containing ORA - 00600[kcbz_check_objd_typ_3]

Bug 5466416---Closed as Duplicate of Unpublished Bug 4430244

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] ON INSERT VIA DBMS_STATS OR DBMS_SPACE


Versions affected : 10.2

Fixed releases : 10.2.0.4, 11.0.0.0

Details : Segment advisor code (eg: DBMS_SPACE.OBJECT_GROWTH_TREND)

can load blocks into the cache for Dropped objects as CURRENT leading

to subsequent operations seeing an incorrect (old) version of a block.

This can lead to various internal buffer cache related errors such as

ORA-600 [kcbnew_3] / ORA-600 [kcbz_check_objd_typ_3].

The exact error depends on which code sees the block.

Backportable : yes to 10.2

Symptoms :

1. ORA-600 [kcbnew_3] internal error reported in the alert.log

2. May occur if segment advisor and DROP operation are being run
concurrently on the same object.

Workaround :

sql>alter system flush buffer cache' to flush the buffer in the cache.

Patch Details:

Check Metalink for Patch 4430244 availability.





Bug 5859511 -----> Closed as Duplicate of Unpublished Bug 4592596

Abstract: INTERMITTENT ORA-600 [KCBZ_CHECK_OBJD_TYP_3] FROM PARALLEL SLAVES

Fixed releases : 10.2.0.4, 11.0.0.0

Details:
Corruption can occur using a multi-table insert SQL with
direct load operations. Eg: If the SQL goes parallel.
This can result in subsequent ORA-1410 type errors on selects
from the target table/s.

Symptoms :

1.Select on table fails with ORA-600 [kcbz_check_objd_typ_3] after multi-table insert with direct load was done.

2.Analyze on the table fails with ORA-1410

3.Call Stack :- kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdst_fetch kdstf0010101km
kdsttgr qertbFetch qergiFetch qertqoFetch



Workaround: -
Do not use direct path (APPEND or PQ) in a multi-table insert SQL

Check Metalink for Patch 4592596 availability.

Unpublished Bug 5754708 ----> Closed as Duplicate of Unpublished Bug 4996133

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] FROM "SYS.DBMS_STATS"


Symptoms : -

1. Running in RAC mode
2. Reading the overflow segment of an IOT
3. Call Stack :- kdsgnp kafger qerixGetNonKeyCol qerixFetchByLogicalRowid
Or
Call Stack :- kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdsgrp kdsfbr qertbFetchByRowID


Workaround:
Flush the buffer cache (10g onwards)
or
Restart the instance to clear its buffer cache.

Check Metalink for Patch 4996133 availability.





Bug 5348204

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] IMPORTING TRANSPORTABLE TABLESPACE FROM 10.1


Fixed In Ver: 10.2.0.1

Symptoms :-

1.Cross Platform tablespace Import using Transportable tablespace option fails with following error message

ORA-39083: Object type TABLE failed to create with error
ORA-600: internal error code, arguments: [kcbz_check_objd_type_3]

2.Call Stack :- kcbz_check_objd_typ kcbzib kcbgtcr ktecgsc ktecgetsh ktecgshx ktsscd_segment ktsscf_segment kdicpsc kdicrws

I am still investigating this one. Will update once I get more information.

Fndfxwho.Sql Fails during TXK or ATG patches

Recently while applying TXK Rollup S and ATG patches, one of the workers running fndfxwho.sql failed:

FAILED: file fndfxwho.sql on worker 8 for product fnd username APPLSYS.

adwork008.log showed:

Start time for file is: Wed Jul 09 2008 17:38:27

sqlplus -s APPS/***** @/gpscd140/erpapp/appl/fnd/11.5.0/patch/115/sql/fndfxwho.s
ql NONE
DECLARE
*
ERROR at line 1:
ORA-20002: [WF_NO_USER] NAME=OPERATOR2 ORIG_SYSTEM=NULL ORIG_SYSTEM_ID=NULL
ORA-06512: at "APPS.WF_LOCAL_SYNCH", line 1890
ORA-06512: at line 154

Metalink Note 372651.1 advises skipping the failed worker and applying ATG RUP5 for the fix. However this error repeated itself when ATG RUP5 was being applied. Metalink Note 374105.1 advises applying ATG RUP6:

Symptoms

On 11.5.10.2, when attempting to apply Patch 4334965 ,
the following error occurs.

ERROR
Fndfxwho.sql fails when installing Patch 4334965

sqlplus -s APPS/*****
@$FND_TOP/11.5.0/patch/115/sql/fndfxwho.sql &un_fnd &pw_fnd
DECLARE
*
ERROR at line 1:
ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at line 36

Cause

In the current SQL statement that is executed a supplementary condition is needed:

Current statement:

select created_by, creation_date, last_updated_by, last_update_date
into createdby, creationdate, lastupdatedby,lastupdatedate
from fnd_user_resp_groups_old
where user_id = who_rec.user_id
and responsibility_id = who_rec.responsibility_id
and responsibility_application_id = who_rec.responsibility_application_id;

New condition:

select created_by, creation_date, last_updated_by, last_update_date
into createdby, creationdate, lastupdatedby,lastupdatedate
from fnd_user_resp_groups_old
where user_id = who_rec.user_id
and responsibility_id = who_rec.responsibility_id
and responsibility_application_id = who_rec.responsibility_application_id
and security_group_id = who_rec.security_group_id;


By adding the supplementary condtition the SQL statement will return just one row, as required by
the business model.

This fix is available in $Header: fndfxwho.sql 115.3 delivered in 11i.ATG_PF.H.delta.6
.

Solution

To implement the solution, please execute the following steps:

1. Obtain 11i.ATG_PF.H.delta.6
2. Review the pre-reqs for this patch and apply to test system

autoconfig fails on adgendbc.sh with ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution. Errors are grouped by directory and phase.
The report format is:


[SETUP PHASE]
AutoConfig could not successfully execute the following scripts:
Directory: $COMMON_TOP/admin/install/$TWO_TASK
adgendbc.sh INSTE8_SETUP 1


AutoConfig is exiting with status 1

When we tried running adgendbc.sh manually, these errors occurred:

Error occurred during initialization of VM
Could not reserve enough space for object heap
UPDATE call failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
Error occurred during initialization of VM
Could not reserve enough space for object heap
Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
Error occurred during initialization of VM
Could not reserve enough space for object heap
Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
Error occurred during initialization of VM
Could not reserve enough space for object heap
DBC generation failed with exit code 1

Based on Metalink Note 394550.1:

Cause

The issue is caused java caching feature enabled.

The following parameter was true in jserv.properties

wrapper.bin.parameters=-DLONG_RUNNING_JVM=true



.

Solution

To implement the solution, please execute the following steps:

1.Take a backup of jserv.properties.
2.Edit the value DLONG_RUNNING_JVM to false.
3.The following should be the change...
wrapper.bin.parameters=-DLONG_RUNNING_JVM=true
to
wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

I modifed jserv.properties and re-ran adgendbc.sh. New errors came up this time:

adgendbc.sh started at Fri Jul 11 14:18:47 EDT 2008


Enter the APPS username: apps

Enter the APPS password:

SQL*Plus: Release 8.0.6.0.0 - Production on Fri Jul 11 14:18:54 2008

(c) Copyright 1999 Oracle Corporation. All rights reserved.

Enter value for 1: Enter value for 2: Enter value for 3: Connected.
Updated profile option value - 1 row(s) updated
Application Id : 0
Profile Name : APPS_DATABASE_ID
Level Id : 10001
New Value : $CONTEXT_NAME
Old Value : $CONTEXT_NAME

PL/SQL procedure successfully completed.


Commit complete.

Disconnected from Oracle9i Enterprise Edition Release 9.2.0.5.0 - 64bit Producti on
With the Partitioning option
JServer Release 9.2.0.5.0 - Production


Params=fnd_jdbc_stmt_cache_free_mem=TRUE
fnd_jdbc_buffer_min=1
fnd_jdbc_buffer_max=5
fnd_jdbc_buffer_decay_interval=300
fnd_jdbc_buffer_decay_size=5
fnd_jdbc_usable_check=false
fnd_jdbc_context_check=true
fnd_jdbc_plsql_reset=false


Unique constraint error (00001) is OK if key already exists
Application server ID already exists for this host - loading..
ADD executed successfully - $FND_SECURE/$CONTEXT_NAME.dbc

Trying to update information ...

java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

UPDATE call failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Generating $FND_SECURE/$TWO_TASK.domainname_$TWO_TASK.dbc
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Generating $FND_SECURE/$CONTEXT_NAME.dbc
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1
Updating parameters: fnd_jdbc_stmt_cache_free_mem=TRUE fnd_jdbc_buffer_min=1 fn d_jdbc_buffer_max=5
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_buffer_decay_interval=300 fnd_jdbc_buffer_decay_s ize=5 fnd_jdbc_usable_check=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

Updating parameters: fnd_jdbc_context_check=true fnd_jdbc_plsql_reset=false
DBC argument specified, ignoring SECURE_PATH
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.FND_APP_SERVER_PKG", line 357
ORA-06512: at line 1
- Database error modifying the server

DBC generation failed with exit code 1

Updating Server Security Authentication
java.sql.SQLException: Io exception: Invalid number format for port number
Database connection to jdbc:oracle:thin:@host_name:port_number:database failed
Updating Server Security Authentication failed with exit code 1
adgendbc.sh exiting with status 1
ERRORCODE = 1 ERRORCODE_END

We saw various metalink notes which pointed to similar errors. We also upgraded to TXK Rollup S to get rid of the error, but to no avail. The common error during the run of adgendbc.sh was:

APPS.FND_APP_SERVER_PKG", line 357

So I decided to check what is present in line 357:

SQL> select line,text
2 from dba_source
3 where line between 357 and 370
4 and name='FND_APP_SERVER_PKG';

LINE
----------
TEXT
--------------------------------------------------------------------------------
357
select SUPPORT_CP, SUPPORT_FORMS, SUPPORT_WEB, SUPPORT_ADMIN,

358
SUPPORT_DB, PLATFORM_CODE

359
into l_support_cp, l_support_forms, l_support_web, l_support_admin,


LINE
----------
TEXT
--------------------------------------------------------------------------------
360
l_support_db, l_platform

361
from fnd_nodes

362
where server_address = p_address;


Based on the above query I checked for the value of server_address in fnd_nodes and found that two node had the same IP address as per fnd_nodes:

SQL> select node_name,server_address
2 from fnd_nodes;

NODE_NAME SERVER_ADDRESS
------------------------------ ------------------------------
PHYSICALDB 192.168.70.29
VIRTUALDB 192.168.70.29
AUTHENTICATION *
APPTIER 192.168.70.69

I pinged each server and found that the correct IP address of Physical DB was different. So I changed it immediately.

SQL> update fnd_nodes
2 set server_address='192.168.70.31'
3 where node_name='PHYSICALDB';

1 row updated.

SQL> commit;

Commit complete.

After this adgendbc.sh ran without errors.

FNDLIBR segmentation fault on RHEL3 after upgrade to 11.5.10.2

Owen pinged me today. After doing an upgrade on an RHEL3 box, the concurrent managers refused to start up. This was the error logged in the concurrent manager logs:

Starting RCVOLTM Concurrent Manager : 10-JUL-2008 21:11:51
sh: line 71: 19678 Segmentation fault FNDLIBR FND CPMGR "FNDCPMBR sysmgr=\"\" $maxreq $sleep $pmon $quesiz $diag logfile=$logfile $target" STOP
$sysmanager
STOP

Metalink Note 343249.1 describes the same issue. However we were already on 11.5.10.2. And the patches suggested were already present. I searched some more and found Metalink Note 264950.1, which suggests applying 806 interop patch on RHEL boxes:

When GSM is not enabled the concurrent managers start up without issue, if GSM is enabled
and the concurrent managers are started I see the following in the log file:
.
Starting STANDARD Concurrent Manager : 15-DEC-2003 19:28:10
sh: line 71: 7884 Segmentation fault (core dumped) FNDLIBR FND CPMGR
"FNDCPMBR sysmgr=\"\" $maxreq $sleep $pmon $quesiz $diag logfile=$logfile
$target"

Fix

Apply patch 3293983, 8.0.6.3 RHEL3.0 APPLICATIONS INTEROPERABILITY PATCH

3293983 was obsoleted by 3830807. So I asked Owen to apply this patch. However he had already gone through this note and applied this patch. He said that applying the patch had no effect. I had a gut feeling that this was the root cause. So I opened the patch readme of 3293983:

#  You must login to your Linux machine as the Oracle software owner before
#  applying this patch. Your setting for ORACLE_HOME must be the value for the
#  Forms 6i 8.0.6 based ORACLE_HOME and you must have write previlages to this
#  area.
#
#  [1] To apply the patch, unzip the PSE container file:
#        % unzip p3830807_8063_LINUX.zip
#
#  [2] Set your current directory to the directory where the patch
#      is located, eg:
#
#        % cd 3830807
#
#  [3] Add execute permission to the script supplied in this patch
#
#        % chmod u+x patch.sh
#
#  [4] Check the value of LD_LIBRARY_PATH.
#
#      This patch will relink reports60 binaries.
#
#      Reports has both link-time and run-time dependency with libjava.so
#      so you need to include
#         $ORACLE_HOME/network/jre11/lib/linux/native_threads    OR
#         $ORACLE_HOME/network/jre11/lib/i686/native_threads
#      in $LD_LIBRARY_PATH before running patch.sh
#
#      Please check your files under $ORACLE_HOME/network/jre11/lib to see
#      which one of the above is appropriate on your system. The same
#      $LD_LIBRARY_PATH should be used at run-time.
#
#  [5] Run the script supplied by this patch
#
#        % ./patch.sh
#
#  [6] Relink all Applications executables. In AD Administration, choose
#      "Relink Applications programs" from the "Generate Applications Files"
#      menu.
#
#  The installation of the patch is now complete.

I asked owen whether he had relinked all Apps executables, and he said that he had not read the readme of the patch. On realizing the missed step, he relinked all executables through adadmin and restarted the concurrent managers. All the concurrent managers started without issue. The problem was resolved.

FAILED: file icxwtab.odf during adpatch

Recently while patching, the worker running icxwtab.odf failed:

ATTENTION: All workers either have failed or are waiting:

FAILED: file icxwtab.odf on worker 1.

ATTENTION: Please fix the above failed worker(s) so the manager can continue.

adworker log showed:

Start time for statement below is: Wed Jul 09 2008 17:12:20

CREATE UNIQUE INDEX ICX.ICX_TRANSACTIONS_U1 ON ICX.ICX_TRANSACTIONS
(TRANSACTION_ID) LOGGING STORAGE (INITIAL 4K NEXT 104K MINEXTENTS 1
MAXEXTENTS UNLIMITED PCTINCREASE 0 FREELIST GROUPS 4 FREELISTS 4 ) PCTFREE
10 INITRANS 11 MAXTRANS 255 COMPUTE STATISTICS TABLESPACE ICXX

Statement executed.

AD Worker error:
The index cannot be created as the table has duplicate keys.


Use the following SQL statement to identify the duplicate keys:

SELECT TRANSACTION_ID, count(*)
FROM ICX.ICX_TRANSACTIONS
GROUP BY TRANSACTION_ID
HAVING count(*)>1

AD Worker error:
Unable to compare or correct tables or indexes or keys because of the error above

As specified in Metalink Note 430673.1:

Symptoms

adpatch fails on script icxwtab.odf with the following errors:

ERROR
The table is missing the index ICX_TRANSACTIONS_U1
or index ICX_TRANSACTIONS_U1 exists on another table.
Create it with the statement:

Start time for statement below is: Mon May 07 2007 14:23:44

CREATE UNIQUE INDEX ICX.ICX_TRANSACTIONS_U1 ON ICX.ICX_TRANSACTIONS
(TRANSACTION_ID) LOGGING STORAGE (INITIAL 4K NEXT 104K MINEXTENTS 1
MAXEXTENTS UNLIMITED PCTINCREASE 0 FREELIST GROUPS 4 FREELISTS 4 ) PCTFREE
10 INITRANS 11 MAXTRANS 255 COMPUTE STATISTICS TABLESPACE ICXX

Statement executed.
AD Worker error:
The index cannot be created as the table has duplicate keys.

Use the following SQL statement to identify the duplicate keys:

SELECT TRANSACTION_ID, count(*)
FROM ICX.ICX_TRANSACTIONS
GROUP BY TRANSACTION_ID
HAVING count(*)>1

AD Worker error:
Unable to compare or correct tables or indexes or keys
because of the error above

SPECIFIC DATA
Ran the suggested query, and here is the output:

TRANSACTION_ID COUNT(*)
-------------------------- -------------
148341124 2
431640607 2
555224577 2
1202811809 2

Cause

These duplicate transactions are there because the concurrent program that deletes temporary
session data (program that removes old entries in ICX_SESSIONS and ICX_TRANSACTIONS) is not
executed on a regular basis. As a result, these tables grow in space and there is the possibility
that the sequences cycle and restart, creating duplicate primary keys.

The following justifies how the issue is related to this specific customer:

SELECT TRANSACTION_ID, count(*)
FROM ICX.ICX_TRANSACTIONS
GROUP BY TRANSACTION_ID
HAVING count(*)>1

TRANSACTION_ID COUNT(*)
-------------------------- --------------
148341124 2
431640607 2
555224577 2
1202811809 2

This is explained in the following unpublished bug: Bug 5001287 PERFORMANCE PROBLEM WHEN APPROVING POS WITH ICX_TRANSACTIONS

Solution

To implement the solution, please execute the following steps:

1. Run the purge program:
a. The name of the program is "Purge Inactive Sessions" located under the "Apps for the Web Manager" responsibility.
b. The internal name is ICXDLTMP.
c. Also you can find this SQL script under $ICX_TOP/sql (named ICXDLTMP.sql).

2. Rerun the failed worker (icxwtab.odf).

3. Migrate the solution as appropriate to other environments.

4. This program should be executed at least once a week to clean up ICX_TRANSACTIONS and ICX_SESSION tables, otherwise they will continue to grow.

Running this sql returned two rows:

SQL> SELECT TRANSACTION_ID, count(*)
FROM ICX.ICX_TRANSACTIONS
GROUP BY TRANSACTION_ID
HAVING count(*)>1
2 3 4 5
SQL> /

TRANSACTION_ID COUNT(*)
-------------- ----------
746007924 2

SQL> desc icx_transactions
Name Null? Type
----------------------------------------- -------- ----------------------------
TRANSACTION_ID NOT NULL NUMBER
SESSION_ID NOT NULL NUMBER
RESPONSIBILITY_APPLICATION_ID NUMBER
RESPONSIBILITY_ID NUMBER
SECURITY_GROUP_ID NUMBER
MENU_ID NUMBER


1 select rowid,transaction_id,session_id
2 from icx_transactions
3* where transaction_id='746007924'
SQL> /

ROWID TRANSACTION_ID SESSION_ID
------------------ -------------- ----------
AAAaZsAGeAAAIsjAAU 746007924 499638533
AAAaZsAAbAAAHb4AAV 746007924 888513258

SQL> delete icx_transactions
2 where rowid='AAAaZsAGeAAAIsjAAU'
3 /

1 row deleted.

SQL> commit;

Commit complete.

Failed worker was restarted through adctrl and it went fine.