ORA-600 and ORA-7445 errors after upgrade to Apps 11.5.10.2 and DB 10.2.0.3

After our recent upgrade from 11.5.9/9.2.0.8/Solaris 8 to 11.5.10.2/10.2.0.3/Solaris 10, we have regular occurances of ORA-600 and ORA-7445 errors:

ORA-600 errors:

ORA-00600: internal error code, arguments: [12209], [111], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [139], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [153], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [154], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [156], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [157], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [158], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [162], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [172], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [190], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [193], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [197], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [198], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [217], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [219], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [230], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [292], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [497], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [498], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [71], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [95], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15264], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [17271], [instantiation space leak], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [17281], [1001], [0x3F6A0D310], [], [], [], [], []

ORA-00600: internal error code, arguments: [20445], [6892510], [6901726], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [64], [64], [65], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [7], [0], [8], [], [], [], []

ORA-00600: internal error code, arguments: [kcbzwb_4], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kglhdunp2_2], [0x437813380], [37], [0x47B749B28], [0x437813470], [0x437A1C828], [1000], [2]

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [qctcte1], [0], [], [], [], [], [], []

ORA-7445 errors:

ORA-07445: exception encountered: core dump [peplmufc_Unregister_From_Cleanup()+152] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFF713EB1D8] [] []

ORA-07445: exception encountered: core dump [peplmufc_Unregister_From_Cleanup()+152] [SIGSEGV] [Invalid permissions for mapped object] [0xFFFFFFFF56D9ABB8] [] []

ORA-07445: exception encountered: core dump [rnmobj1()+3320] [SIGSEGV] [Address not mapped to object] [0x000000134] [] []

ORA-07445: exception encountered: core dump [sdfind_rtn_hdr()+908] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF6240A03C] [] []

Some of these errors are causing business critical problems. We have severity 1 SRs open with Oracle to solve these issues. Will update once I learn more.

We disabled automatic SGA on Oracle's suggestion and the errors have disappeared. It is described in more detail in this post.

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

After the upgrade from 11.5.9 / 9.2.0.8 to 11.5.10.2/10.2.0.3 we are getting errors in DB alert logs:

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

A search through ORA-600 lookup tool turns up this note:

Subject:	ORA-600 [kcbz_check_objd_typ_3]
	Doc ID:	Note:406787.1	Type:	REFERENCE
	Last Revision Date:	10-JUL-2008	Status:	PUBLISHED

Note: For additional ORA-600 related information please read Note 146580.1

PURPOSE:
 This article represents a partially published OERI note.

 It has been published because the ORA-600 error has been
 reported in at least one confirmed bug.

 Therefore, the SUGGESTIONS section of this article may help
 in terms of identifying the cause of the error.

 This specific ORA-600 error may be considered for full publication
 at a later date. If/when fully published, additional information
 will be available here on the nature of this error.

SUGGESTIONS:

 If the Known Issues section below does not help in terms of identifying
 a solution, please submit the trace files and alert.log to Oracle
 Support Services for further analysis.

 Known Issues:
 Bug# 6405339   See Note 6405339.8
     Corruption / OERI[kcbz_check_objd_typ_3] / OERI:5400 from PQ
     Fixed: 10.2.0.5, 11.1.0.7, 11.2

 Bug# 5689412   See Note 5689412.8
     ORA-600 [kcbz_check_objd_typ_3] after SCN based recovery
     Fixed:

 Bug# 4592596   See Note 4592596.8
     Corruption (ORA-1410) from multi-table insert with direct load
     Fixed: 10.2.0.4, 11.1.0.6

 Bug# 4430244 +  See Note 4430244.8
     Segment advisor can load blocks of dropped objects into buffer cache (KCB OERI errors)
     Fixed: 10.2.0.4, 11.1.0.6

Metalink Note 466049.1 describes summary of Bugs Containing ORA - 00600[kcbz_check_objd_typ_3]

Purpose

This article is intended to

1. Help the reader to understand ORA-600 [kcbz_check_objd_typ_3] error and how is this caused.

2. List the most common bugs of the aforesaid error and their fixes.

Scope

This is a consolidated article based upon various bugs logged for the error
ORA-600 [kcbz_check_objd_typ_3] and gives the reader an overview about the most frequent bugs which the customers are affected with.

The error ORA-600 [kcbz_check_objd_typ_3] is raised because we see different object id in bufffer cache (BH) and on disk .

Summary of Bugs Containing ORA - 00600[kcbz_check_objd_typ_3]

Bug 5466416---Closed as Duplicate of Unpublished Bug 4430244

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] ON INSERT VIA DBMS_STATS OR DBMS_SPACE


Versions affected : 10.2

Fixed releases : 10.2.0.4, 11.0.0.0

Details : Segment advisor code (eg: DBMS_SPACE.OBJECT_GROWTH_TREND)

can load blocks into the cache for Dropped objects as CURRENT leading

to subsequent operations seeing an incorrect (old) version of a block.

This can lead to various internal buffer cache related errors such as

ORA-600 [kcbnew_3] / ORA-600 [kcbz_check_objd_typ_3].

The exact error depends on which code sees the block.

Backportable : yes to 10.2

Symptoms :

1. ORA-600 [kcbnew_3] internal error reported in the alert.log

2. May occur if segment advisor and DROP operation are being run
concurrently on the same object.

Workaround :

sql>alter system flush buffer cache' to flush the buffer in the cache.

Patch Details:

Check Metalink for Patch 4430244 availability.





Bug 5859511 -----> Closed as Duplicate of Unpublished Bug 4592596

Abstract: INTERMITTENT ORA-600 [KCBZ_CHECK_OBJD_TYP_3] FROM PARALLEL SLAVES

Fixed releases : 10.2.0.4, 11.0.0.0

Details:
Corruption can occur using a multi-table insert SQL with
direct load operations. Eg: If the SQL goes parallel.
This can result in subsequent ORA-1410 type errors on selects
from the target table/s.

Symptoms :

1.Select on table fails with ORA-600 [kcbz_check_objd_typ_3] after multi-table insert with direct load was done.

2.Analyze on the table fails with ORA-1410

3.Call Stack :- kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdst_fetch kdstf0010101km
kdsttgr qertbFetch qergiFetch qertqoFetch



Workaround: -
Do not use direct path (APPEND or PQ) in a multi-table insert SQL

Check Metalink for Patch 4592596 availability.

Unpublished Bug 5754708 ----> Closed as Duplicate of Unpublished Bug 4996133

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] FROM "SYS.DBMS_STATS"


Symptoms : -

1. Running in RAC mode
2. Reading the overflow segment of an IOT
3. Call Stack :- kdsgnp kafger qerixGetNonKeyCol qerixFetchByLogicalRowid
Or
Call Stack :- kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdsgrp kdsfbr qertbFetchByRowID


Workaround:
Flush the buffer cache (10g onwards)
or
Restart the instance to clear its buffer cache.

Check Metalink for Patch 4996133 availability.





Bug 5348204

Abstract: ORA-600 [KCBZ_CHECK_OBJD_TYP_3] IMPORTING TRANSPORTABLE TABLESPACE FROM 10.1


Fixed In Ver: 10.2.0.1

Symptoms :-

1.Cross Platform tablespace Import using Transportable tablespace option fails with following error message

ORA-39083: Object type TABLE failed to create with error
ORA-600: internal error code, arguments: [kcbz_check_objd_type_3]

2.Call Stack :- kcbz_check_objd_typ kcbzib kcbgtcr ktecgsc ktecgetsh ktecgshx ktsscd_segment ktsscf_segment kdicpsc kdicrws

I am still investigating this one. Will update once I get more information.

ORA-600 errors after upgrade from 9.2.0.8 to 10.2.0.3

1.ORA-00600: internal error code, arguments: [16203], [], [], [], [], [], [], []
2.ORA-00600: internal error code, arguments: [17281], [1001], [0x3AE64C028], [], [], [], [], []
3.ORA-00600: internal error code, arguments: [kcbzwb_4], [], [], [], [], [], [], []

1. Possibly bug 5108130:
Making a reference in a PL/SQL block to a synonym at a remote site R1 when
the referent of the synonym was on yet another site R2 could have resulted
in an ORA-600 [16203] error. The PL/SQL block must have been compiled by
a 9.2.0.X or earlier compiler.

2. Possibly bug 5482174:

Deinstantiation of a large number of PLSQL objects that have opened cursors
when session_cached_cursors is non-zero might hit an ORA-600 [17281] [36]
with a call stack of the form
  'ksupop->kgiCallStateCleanUp->kgiLRUResize'
repeated within it a number of times.

Workaround:
 Set session_cached_cursors to zero.

3. Possibly bug 5028099:

ORA-600 [kcbzwb_4] can occur when using in memory undo (IMU)
for private buffers.

Workaround:
 Set "_in_memory_undo"=FALSE