ORA-04063: PACKAGE BODY "SYS.DBMS_UTILITY" HAS ERRORS

At 7 AM on 29th February, I got an SOS call for issues on an instance. After talking to the DBAs, I checked the alert log file:

Doing block recovery for file 1325 block 84015
Block recovery from logseq 2834, block 424302 to scn 10103409529653
Fri Feb 29 06:05:45 2008
Recovery of Online Redo Log: Thread 1 Group 1 Seq 2834 Reading mem 0
Mem# 0: /justanexample/redo1/log1a.dbf
Mem# 1: /justanexample/redo1/log1b.dbf
Block recovery completed at rba 2834.424312.16, scn 2352.1646449464
Doing block recovery for file 453 block 55097
Block recovery from logseq 2834, block 396285 to scn 10103409529653
Fri Feb 29 06:05:45 2008
Recovery of Online Redo Log: Thread 1 Group 1 Seq 2834 Reading mem 0
Mem# 0: /justanexample/redo1/log1a.dbf
Mem# 1: /justanexample/redo1/log1b.dbf
Block recovery completed at rba 2834.424312.16, scn 2352.1646449464
Doing block recovery for file 1418 block 35982
Block recovery from logseq 2834, block 420640 to scn 10103409529653
Fri Feb 29 06:05:45 2008
Recovery of Online Redo Log: Thread 1 Group 1 Seq 2834 Reading mem 0
Mem# 0: /justanexample/redo1/log1a.dbf
Mem# 1: /justanexample/redo1/log1b.dbf
Block recovery completed at rba 2834.424312.16, scn 2352.1646449464
Fri Feb 29 06:05:49 2008
Errors in file /justanexample/oracle/10.2.0/admin/justanexample_justanexample/bdump/justanexample_j000_9484.trc:
ORA-12012: error on auto execute of job 3404
ORA-04063: package body "SYS.DBMS_UTILITY" has errors
ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_UTILITY"
ORA-06512: at "SYS.DBMS_IREFRESH", line 62
ORA-06512: at "SYS.DBMS_REFRESH", line 15
ORA-06512: at "SYS.DBMS_REFRESH", line 195
ORA-06512: at line 1
Fri Feb 29 06:05:49 2008
Errors in file /justanexample/oracle/10.2.0/admin/justanexample_justanexample/bdump/justanexample_j001_16186.trc:
ORA-12012: error on auto execute of job 4155
ORA-04063: package body "SYS.DBMS_UTILITY" has errors
ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_UTILITY"
ORA-06512: at "APPS.WF_EVENT", line 1160
ORA-06512: at "APPS.WF_BES_CLEANUP", line 482
ORA-06512: at line 1

When I checked dba_registry, I found that Oracle packages and types had become invalid.

I asked the DBAs to follow Metalink Note 457861.1:

SQL > @?/rdbms/admin/catalog.sql
SQL > @?/rdbms/admin/catproc.sql
SQL > @?/rdbms/admin/utlrp.sql

After running catalog and catproc, which invalidated all dependencies the no. of invalid objects reached 160,000. utlrp.sql took 10 hours to complete with 24 parallel threads. Even after this 16 objects in sys were invalid. These were types:

TYPE AQ$_DESCRIPTOR
TYPE AQ$_DESCRIPTOR
TYPE AQ$_EVENT_MESSAGE
TYPE AQ$_REG_INFO
TYPE AQ$_REG_INFO
TYPE AQ$_REG_INFO_LIST
TYPE AQ$_REG_INFO_LIST
TYPE AQ$_SRVNTFN_MESSAGE
TYPE AQ$_SRVNTFN_MESSAGE
TYPE AQ$_SRVNTFN_MESSAGE
TYPE DBMS_XPLAN_TYPE
TYPE DBMS_XPLAN_TYPE_TABLE
TYPE LCR$_ROW_LIST
TYPE LCR$_ROW_UNIT TYPE MSG_PROP_T
TYPE XMLGENFORMATTYPE

We had to reset these types manually with the command:
Alter type owner.typename reset;

We are still investigating the root cause for this issue.