SMON: disabling tx recovery

Recently, we had performance issues on a box which was shared by 18 instances. It was a Sun E2900 box with 24 processors and 96 GB RAM. The problem was tracked down to long running FNDLIBR processes which were hogging a lot of memory. Some of these processes were running for more than 30 days. Clearly they were not legitimate. As a result, we shutdown concurrent managers of most of the instances, ran cmclean.sql and restarted the services. On one of the instances, which had 10GB SGA, we reduced the SGA to 6 GB. However when shutdown immediate command was issued, the instance refused to shutdown with this message in the alert log:

SMON: disabling tx recovery

I found Metalink Note 1076161.6 which I am pasting here for reference:

Description
===========
SHUTDOWN NORMAL or SHUTDOWN IMMEDIATE hangs. In the alert.log, you see only
the following:

Shutting down instance (immediate)
License high water mark = 12
Thu Dec  8 18:43:16 1994
alter database  close normal
Thu Dec  8 18:43:17 1994
SMON: disabling tx recovery
SMON: disabling cache recovery
or
     waiting for smon to disable tx recovery

There are no ORA errors or trace files.



Scope & Application
===================
Informational

During a SHUTDOWN IMMEDIATE and SHUTDOWN NORMAL, SMON is cleaning up extents
which are no longer needed and marking them as freed.

Either wait for SMON to clean up the free extents in the database as it
shuts down or perform a SHUTDOWN ABORT to shutdown the instance. A SHUTDOWN
ABORT will not perform a clean shutdown.

Verify that temporary segments are decreasing
---------------------------------------------
To verify that the temporary segments are decreasing have an active session
available in Server Manager or SQLPLUS during the SHUTDOWN IMMEDIATE. Issue the following
query to ensure the database is not hanging, but is actually perform extent
cleanup:

 SVRMGR/SQL> select count(block#) from fet$;
 COUNT(BLOC
 ----------
          7

 SVRMGR/SQL> select count(block#) from uet$;
 COUNT(BLOC
 ----------
        402

After some time has elapsed, reissue the query and see that the values for fet$
have increased while the values or uet$ have decreased:

 SVRMGR/SQL> select count(block#) from fet$;
 COUNT(BLOC
 ----------
         10

 SVRMGR/SQL> select count(block#) from uet$;
 COUNT(BLOC
 ----------
        399

During shutdown the SMON process is cleaning up extents and updating the data
dictionary tables with the marked free extents. As the extents are marked as
freed, they are removed from the table for used extents, UET$ and placed on the
table for free extents, FET$.

How to Avoid creating many Temporary Extents
--------------------------------------------
Once the database has shutdown cleanly, to avoid creating many temporary
extents change the initial and next extent sizes on temporary tablespaces
to a more appropriate size:

 ALTER TABLESPACE  DEFAULT STORAGE (INITIAL M/K NEXT M/K);

Note: If the temporary tablespace is of type TEMPORARY, then this change
will only affect temporary segments created after issuing the above
command. Any existing temporary segments already in the TEMPORARY tablespace
will not be affected till the instance is restarted. On shutdown, existing
temporary segments are dropped. If the TEMPORARY TABLESPACE is of type
PERMANENT, then cleanup is performed by SMON after completion of the process
using it.

Increasing the initial and next extent size will decrease the number of extents
that are allocated to temporary segments. Since there are fewer extents to
deallocate, the database should shutdown more speedily.

Take the following scenario:

A database was subject to large sorts with the following sort parameter in
the "init.ora" file:

      - sort_area_size=1000000

The temporary tablespaces for this database were all created with initial and
next extents sized at 50k and the total database size was about 300mb.

Database sorts will utilize memory as much as possible based on the "init.ora"
parameter "sort_area_size".  Once this memory-based sort area is filled, the
database will utilize the temporary table space associated with the database
user to complete the sort operation.  During a shutdown normal, the database
will attempt to clean up the temporary tablespaces.

If a small extent size is used, then a large number of extents will be created
for a large sort.  The cleanup of the temporary tablespace takes much longer
with a large number of extents.

Note:
=====
You have to do a shutdown abort and then bring the database
back up to run the suggested queries.

For other reasons for slow/hung shutdown see also these notes:

Note 375935.1 - What To Do and Not To Do When 'shutdown immediate' Hangs
Note 428688.1 - Bug 5057695: Shutdown Immediate Very Slow To Close Database.

References:
===========
Note 61997.1 SMON - Temporary Segment Cleanup and Free Space Coalescing
                

Search Words:
=============
hanging
shutdown

We had to issue the command "shutdown abort" to finally shutdown the database. I asked Oracle in an SR how to do the temporary segment cleanup when the database is running, so that when we next shutdown the DB, it comes down without taking too much time. Oracle responded that there is no way to do it.

UPDATE
=======
Hi Vikram,
I reviewed the generated files hoping to see something upnormal, however both findings above do not point to any upnormal issues, hence SMON background process is performing the expected behaviour of cleaning up extents which are no longer needed and marking them as freed.

As the note mentions, there are two options, either to wait until SMON performs its job, or use
shutdown abort (which is something i would not recommend).

Please let me know your thoughts and whether you need us to discuss this over the phone.


Best Regards,
Hany Fawzy
Oracle Support.


Hi,

Is there any way to make SMON do its cleanup job before the actual command of shutdown immediate is given ?

- Vikram


UPDATE
=======
Hi Vikram,

The answer to this question is no, there is however another method, by which you would stop the cleanup before the shutdown, this is actually implemented by using Event 10061, This disables SMON from cleaning up temporary extents. This will leave used (unavailable) space in the tablespace in which the extents reside. You can then point users to another temporary tablesp ace if necessary. The database can then be restarted off-peak without 10061 and
SMON will clean up the temporary extents. The above is actually not recommended, If you perform a shutdown before allowing SMON to clean up all temporary extents in a temporary tablespace, then after the subsequent startup SMON may appear to spin without cleaning up any further extents.


Best Regards,
Hany Fawzy
Oracle Support.

The event 10061 is described in metalink note 21169.1. You need to put this in your init.ora:

event="10061 trace name context forever, level 10"

However, this also didn't help and shutdown immediate still hung up. So we kept doing shutdown abort till the instance was recloned from Production.