ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV]

We got this error in one of our Production instances whenever we ran Gather Schema Statistics program:

Errors in Database alert log:
$ORACLE_HOME/admin/$CONTEXT_NAME/udump/$TWO_TASK_ora_13052.trc:
ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address
not mapped to object] [0x000000000] [] []

Trace file:
ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address not mapped to object]
[0x000000000] [] []
Current SQL statement for this session:
begin :retval := "CTXSYS"."TEXTOPTSTATS".ODCIStatsCollect(:idx_info,:stat_opt,:stats,:env);
end;
----- PL/SQL Call Stack -----
object line object
handle number name
3b1247878 6 CTXSYS.TEXTOPTSTATS
3b1688f20 1 anonymous block
3b144c860 3379 package body SYS.DBMS_STATS
3b144c860 3639 package body SYS.DBMS_STATS
3b144c860 10584 package body SYS.DBMS_STATS
3b144c860 13153 package body SYS.DBMS_STATS
3b144c860 13202 package body SYS.DBMS_STATS
3a7244100 1 anonymous block
3b11b3e38 676 package body APPS.FND_STATS
3b11b3e38 1487 package body APPS.FND_STATS
3b11b3e38 1004 package body APPS.FND_STATS
3b11b3e38 782 package body APPS.FND_STATS
3c197dab0 1 anonymous block

We had an SR logged with Oracle for this. They noticed that we were not passing estimate percent which defaults to 10%. They asked us to explicitly give estimate percent as 30% and run GSS. The issue was resolved after this. From the SR:

ISSUE CLARIFICATION
====================

Gather Schema Statistics concurrent program issue for ' All ' schemas : it completes with error.


ISSUE VERIFICATION
===================
Verified the issue by the alert log file and trace file , which displays :

ORA-07445: exception encountered: core dump [drumcmf()+184] [SIGSEGV] [Address not mapped to object] [0x000000000]" errors are encountered with core dumps.


CAUSE DETERMINATION
====================
Parameters to run Gather Shema Statistics are incorrect.

CAUSE JUSTIFICATION
====================
The following parameters are set as fillow :

GATHER OPTIONS: GATHER AUTO
Estimate Percent: 10


.
PROPOSED SOLUTION(S)
======================
GATHER OPTIONS: GATHER
Estimate Percent: 30


PROPOSED SOLUTION JUSTIFICATION(S)
====================================
GATHER AUTO : If underlying tables are not in monitoring mode , then "Gather Auto" option will try to change it. Since this is a DDL statement, it requires an exclusive lock on the table. If
table is under heavy load and frequently accessed , then this might take a while.
Note : 331017.1


Estimate Percent : Note.141532.1

Percentage of rows to estimate. If left empty it will default to 10%. The valid range is 0-99. A higher percentage will be more accurate, but take longer to run. If the object(s) that you
are gathering statistics for do not change often or the object(s) has data entered that is very similiar you may choose a lower number. However, if the data changes frequently a larger number entered for this parameter would be recommended to provide a more accurate representation of your data.

SOLUTION / ACTION PLAN
=======================
To implement the solution, please execute the following steps:

1. Schema Name: ALL
Estimate Percent: 30
Backup Flag : NOBACKUP
HISTORY MODE: LAST RUN
GATHER OPTIONS: GATHER
Invalidate Dependent Cursors: Y (Yes)


2. References :
Note. 141532.1 ' How To Gather Schema Statistics for Oracle Applications 11i '
Note. 331017.1 ' Gather Schema Statistics program hangs or fails with ORA-54 error '

ORA-600 and ORA-7445 errors after upgrade to Apps 11.5.10.2 and DB 10.2.0.3

After our recent upgrade from 11.5.9/9.2.0.8/Solaris 8 to 11.5.10.2/10.2.0.3/Solaris 10, we have regular occurances of ORA-600 and ORA-7445 errors:

ORA-600 errors:

ORA-00600: internal error code, arguments: [12209], [111], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [139], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [153], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [154], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [156], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [157], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [158], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [162], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [172], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [190], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [193], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [197], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [198], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [217], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [219], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [230], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [292], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [497], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [498], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [71], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [12209], [95], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15160], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [15264], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [17271], [instantiation space leak], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [17281], [1001], [0x3F6A0D310], [], [], [], [], []

ORA-00600: internal error code, arguments: [20445], [6892510], [6901726], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [64], [64], [65], [], [], [], []

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [7], [0], [8], [], [], [], []

ORA-00600: internal error code, arguments: [kcbzwb_4], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kglhdunp2_2], [0x437813380], [37], [0x47B749B28], [0x437813470], [0x437A1C828], [1000], [2]

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [qctcte1], [0], [], [], [], [], [], []

ORA-7445 errors:

ORA-07445: exception encountered: core dump [peplmufc_Unregister_From_Cleanup()+152] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFF713EB1D8] [] []

ORA-07445: exception encountered: core dump [peplmufc_Unregister_From_Cleanup()+152] [SIGSEGV] [Invalid permissions for mapped object] [0xFFFFFFFF56D9ABB8] [] []

ORA-07445: exception encountered: core dump [rnmobj1()+3320] [SIGSEGV] [Address not mapped to object] [0x000000134] [] []

ORA-07445: exception encountered: core dump [sdfind_rtn_hdr()+908] [SIGBUS] [Object specific hardware error] [0xFFFFFFFF6240A03C] [] []

Some of these errors are causing business critical problems. We have severity 1 SRs open with Oracle to solve these issues. Will update once I learn more.

We disabled automatic SGA on Oracle's suggestion and the errors have disappeared. It is described in more detail in this post.

ORA-7445 in alert log when you open Project Quick Entry form

In one of the 11.5.10.2 instances which was upgraded to 10.2.0.3 Database, the following behavior was observed:

Whenever Project Quick Entry form was accessed, it gave the error:

FRM-40735: ON-ERROR trigger raised unhandled exception ORA-01001.

APP-FND-01564: error 1000 in audfrm

Cause: audfm failed due to ORA-01000 maximum open cursors exceeded
The SQL statement being executed at the time of the error was &SQLSTMT and was executed from the file &ERRFILE.

In alert log the following ORA-7445 was logged:

ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [drurnew()+184] [SIGSEGV] [Address not mapped to object] [0x000000040] [] []
Current SQL statement for this session:
INSERT INTO PA_PROJECT_CTX_SEARCH (PROJECT_ID, CTX_DESCRIPTION, PROGRAM_REQUEST_ID, PROGRAM_APPLICATION_ID, PROGRAM_ID, PROGRAM_UPDATE_DATE, CREATED
_BY, CREATION_DATE, LAST_UPDATED_BY, LAST_UPDATE_DATE, LAST_UPDATE_LOGIN) VALUES
(:B2 , :B1 , NULL, NULL, NULL, NULL, FND_GLOBAL.USER_ID, SYSDATE, FND_GLOBAL.US
ER_ID, SYSDATE, FND_GLOBAL.LOGIN_ID)
----- PL/SQL Call Stack -----
object line object
handle number name
3ec258df8 5 CTXSYS.TEXTINDEXMETHODS
3ec258df8 625 CTXSYS.TEXTINDEXMETHODS
----- Call Stack Trace -----

We logged an SR for this issue and got this response:

CAUSE DETERMINATION
===================
This is unpublished Bug 5587976.

The error is due to a problem with the intermedia functionality.

CAUSE JUSTIFICATION
===================
BUG 5587976 : APPSST: ORA-07445: EXCEPTION ENCOUNTERED: CORE DUMP [DRURNEW()+129] [SIGSEGV]

PROPOSED SOLUTION(S)
====================
Download and install Patch 5587976

Bug has been fixed in 10.2.0.4 patchset

PROPOSED SOLUTION JUSTIFICATION(S)
==================================
SURE 461767.1 : ORA-7445 [Drurnew()] Encountered During Applications Upgrade

SOLUTION / ACTION PLAN
======================
-- To implement the solution, please execute the following steps::
Download and install Patch 5587976

Even after applying this patch the issue was occuring. The DBA team ran gather schema statistics on PA schema from backend.

execute fnd_stats.gather_schema_stats('PA');

After this the issue was resolved. This issue was not occuring in dev/test instances which did not have this patch. It is possible that the error was coming solely due to the absence of statistics.

ORA-7445 in alert log, due to FRM-40735 ORA-01001 in Projects Quick Entry form

Yesterday, I discussed with Akhilesh about an issue being faced in one of the production instances, which has started logging ORA-7445 errors whenever projects quick entry form is accessed which throws FRM-40735: ON-ERROR trigger raised unhandled error: ORA-01001. We analyzed the trace files generated by ORA-7445 :

ORA-07445: exception encountered: core dump [drurnew()+184] [SIGSEGV] [Address n
ot mapped to object] [0x000000040] [] []
Current SQL statement for this session:
INSERT INTO PA_PROJECT_CTX_SEARCH (PROJECT_ID, CTX_DESCRIPTION, PROGRAM_REQUEST_
ID, PROGRAM_APPLICATION_ID, PROGRAM_ID, PROGRAM_UPDATE_DATE, CREATED_BY, CREATIO
N_DATE, LAST_UPDATED_BY, LAST_UPDATE_DATE, LAST_UPDATE_LOGIN) VALUES (:B2 , :B1
, NULL, NULL, NULL, NULL, FND_GLOBAL.USER_ID, SYSDATE, FND_GLOBAL.USER_ID, SYSDA
TE, FND_GLOBAL.LOGIN_ID)
----- PL/SQL Call Stack -----
object line object
handle number name
3ec258df8 5 CTXSYS.TEXTINDEXMETHODS
3ec258df8 625 CTXSYS.TEXTINDEXMETHODS
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------

A query of ORA-7445/ORA-600 lookup utility revealed that this is a known bug # 5587976. This bug is triggered by a bad return from some NLS functions. This fix signals such
problems as an NLS error rather than causing the process to dump.

However we are still exploring why this error is not occuring in Production support instances and have asked Oracle to explain the bug in more detail.