usdsop cannot redirect standard output

Akhilesh reported this error to me today where concurrent managers refused to start with this error in the *.mgr logs:

Cause: usdsop encountered an error redirecting standard output for a child process.

Metalink Note 733901.1 describes this problem and gives the cause as:

The cause of this problem has been identified in Bug 7000874. It is caused by a code bug which causes memory corruption on an internal FILENAME variable. As a result of that the FND Service Manager fails to spawn the service manager processes such as the Output Post Processor and many others.

Solution

Apply a patch that includes the fixed file version of afpsms.oc:

• Oracle E-Business Suite 11i
  Patch 7244628 1OFF:7235678:11.5.10.2:11.5.10.2:GSM Services Not Running
  --> Controlled release patch on top of patch 4676589 11i.ATG_PF.H.RUP4.
• Oracle E-Business Suite Release 12
  Patch 7000874 GSM Services Not Running
  --> Controlled release patch on top of the 12.0.3 code level.

Note: Both patches are released under the terms of 'By Support' which implies that the patch is queryable on Metalink, but downloads of these patches requires a password. Please contact Global Customer Support to receive a password to download the patch.

Workaround

In case the issue is encountered on a lower code level and thus a patch is not available, one may consider to implement the following workaround:

1. Stop the concurrent managers.
2. Decreased the sequence value of FND_CONCURRENT_PROCESSES_S to 500000 (decrease from 8 to 6 digits).
   
   
   • Determine the current sequence value:
     
     SELECT applsys.fnd_concurrent_processes_s.NEXTVAL
FROM DUAL;
     
     
   • Set the INCREMENT BY value of the sequence to the value of (fnd_concurrent_processes_s.NEXTVAL - 500000) * -1). For example, if the current value is 10019473 then the INCREMENT value will be -9519473 (= (10019473 - 500000) * -1).
     
     ALTER SEQUENCE applsys.fnd_concurrent_processes_s INCREMENT BY -9519473;
     
     
   • Decrease the current sequence value by requesting a next value from the sequence:
     
     SELECT fnd_concurrent_processes_s.NEXTVAL
FROM DUAL;
     
     This will return a value of 500000.
     
     
   • Reset the INCREMENT BY value to 1:
     
     ALTER SEQUENCE fnd_concurrent_processes_s INCREMENT BY 1;
     
     
3. Determine the concurrent manager log files that will need to be removed manually from the Concurrent Processing tier. The easiest way is to spool the output to a spool file so that it can be execute afterwards from the command line (or to be archived if needed).
   
   SELECT 'rm ' || logfile_name
FROM fnd_concurrent_processes
WHERE concurrent_process_id >= 500000;
   
   
4. Delete the rows from the FND_CONCURRENT_PROCESSES table:
   
   DELETE FROM fnd_concurrent_processes
WHERE concurrent_process_id >= 500000;
   
   
5. Delete the rows from the FND_ENV_CONTEXT table:
   
   DELETE FROM fnd_env_context
WHERE concurrent_process_id >= 500000;
   
   
6. Start the concurrent managers.

Once these actions are completed, the service managers such as the Workflow related ones as well as the Output Post Processor will start up successfully.

Note: The impact of the workaround affects only the Concurrent Processing log files. If preferred, these can be kept for a temporary time in case it may be of interest. The existing concurrent requests which ran in the past are not affected meaning that report data will not be lost by completing this workaround.