R12.2 Single file system

With the release of AD and TXK Delta 6, Oracle has provided the feature of single file system on development instances for R12.2. Here's what they have mentioned in support.oracle.com article: Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.6 and R12.TXK.C.Delta.6 (Doc ID 1983782.1)

Enhancements in AD and TXK Delta 6

4. New and Changed Features

Oracle E-Business Suite Technology Stack and Oracle E-Business Suite Applications DBA contain the following new or changed features in R12.AD.C.Delta.6 and R12.TXK.C.Delta.6.

4.1 Support for single file system development environments

• A normal Release 12.2 online patching environment requires one application tier file system for the run edition, and another for the patch edition. This dual file system architecture is fundamental to the patching of Oracle E-Business Suite Release 12.2 and is necessary for production environments and test environments that are meant to be representative of production. This enhancement makes it possible to have a development environment with a single file system, where custom code can be built and tested. A limited set of adop phases and modes are available to support downtime patching of such a development environment. Code should then be tested in standard dual file system test environments before being applied to production.

More details are provided in Oracle E-Business Suite Maintenance Guide, Chapter: Patching Procedures):
http://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531065.htm#6169002 

Support for Single File System Development Environments
A normal Release 12.2 online patching environment requires two application tier file systems, one for the run edition and another for the patch edition. This dual file system architecture is fundamental to patching of Oracle E-Business Suite Release 12.2, and is necessary both for production environments and test environments that are intended to be representative of production. This feature makes it possible to create a development environment with a single file system, where custom code can be built and tested. The code should then always be tested in a standard dual file system test environment before being applied to production.
You can set up a single file system development environment by installing Oracle E-Business Suite Release 12.2 in the normal way, and then deleting the $PATCH_BASE directory with the command:

$ rm -rf $PATCH_BASE

A limited set of adop phases and modes are available to support patching of a single file system development environment. These are:

·         apply phase in downtime mode

·         cleanup phase

Specification of any other phase or mode will cause adop to exit with an error.
The following restrictions apply to using a single file system environment:

·         You can only use a single file system environment for development purposes.

·         You cannot use online patching on a single file system environment.

·         You can only convert an existing dual file system environment to a single file system: you cannot directly create a single file system environment via Rapid Install or cloning.

·         There is no way to convert a single file system environment back into a dual file system.

·         You cannot clone from a single file system environment.

You Are Trying To Access a Page That Is No Longer Active.The Referring Page May Have Come From a Previous Session. Please Select Home To Proceed

Shahed pinged me about this error.  It was coming after logging in.  This R12.1.3 instance had just migrated from an old server to a new one. Once you logged in this error would be displayed:

You Are Trying To Access a Page That Is No Longer Active.The Referring Page May Have Come From a Previous Session. Please Select Home To Proceed

The hits on support.oracle.com were not helpful, but a gave a clue that it may have something to do with session cookie.  So I used Firefox to check http headers.  If you press Ctrl+Shift+K, you will get a panel at the bottom of the browser. Click on Network tab, click on the AppsLocalLogin.jsp and on the right side of the pane, you'll see a cookie tab.

The domain appearing in the cookie tab was from the old server.  So I checked:

select session_cookie_domain from icx_parameters;
olddomain.justanexample.com

So I nullified it:

update icx_parameters set session_cookie_domain=null;

commit;

Restarted Apache

cd $ADMIN_SCRIPTS_HOME
adapcctl.sh stop
adapcctl.sh start

No more error.  I was able to log in and so was Shahed.

Chrome and E-Business Suite

Dhananjay came to me today.  He said that his users were complaining about forms not launching after upgrading to the latest version of Chrome. On launching forms they got this error:

/dev60cgi/oracle forms engine Main was not found on this server

I recalled that Google Chrome team had announced that they would not support java going forward. Googling with keywords chrome java brought this page:

https://java.com/en/download/faq/chrome.xml#npapichrome

It states that:

NPAPI support by Chrome

The Java plug-in for web browsers relies on the cross platform plugin architecture NPAPI, which has long been, and currently is, supported by all major web browsers. Google announced in September 2013 plans to remove NPAPI support from Chrome by "the end of 2014", thus effectively dropping support for Silverlight, Java, Facebook Video and other similar NPAPI based plugins. Recently, Google has revised their plans and now state that they plan to completely remove NPAPI by late 2015. As it is unclear if these dates will be further extended or not, we strongly recommend Java users consider alternatives to Chrome as soon as possible. Instead, we recommend Firefox, Internet Explorer and Safari as longer-term options. As of April 2015, starting with Chrome Version 42, Google has added an additional step to configuring NPAPI based plugins like Java to run — see the section Enabling NPAPI in Chrome Version 42 and later below.

Enabling NPAPI in Chrome Version 42 and later

As of Chrome Version 42, an additional configuration step is required to continue using NPAPI plugins.

1. In your URL bar, enter:
   chrome://flags/#enable-npapi 
2. Click the Enable link for the Enable NPAPI configuration option.
3. Click the Relaunch button that now appears at the bottom of the configuration page.

Developers and System administrators looking for alternative ways to support users of Chrome should see this blog, in particular "Running Web Start applications outside of a browser" and "Additional Deployment Options" section.

Once Dhananjay did the above steps, Chrome started launching forms again.  He quickly gave these steps to all his users who had upgraded to the latest version of Chrome (version 42) and it started working form them too.

Oracle doesn't certify E-Business Suite forms on Chrome.  Only self service pages of E-Business Suite are certified on Google Chrome.

opatch hangs on /sbin/fuser oracle

Pipu pinged me today about opatch hanging. The opatch log showed this:

[Apr 11, 2015 5:24:13 PM]    Start fuser command /sbin/fuser $ORACLE_HOME/bin/oracle at Sat Apr 11 17:24:13 EDT 2015

I had faced this issue once before, but was not able to recall what was the solution.  So I started fresh.

As oracle user:

/sbin/fuser $ORACLE_HOME/bin/oracle hung

As root user

/sbin/fuser $ORACLE_HOME/bin/oracle hung

As root user

lsof hung.

Google searches about it brought up a lot of hits about NFS issues.  So I did df -h.

df -h also hung.

So I checked /var/log/messages and found many messages like these:

Apr 11 19:44:42 erpserver kernel: nfs: server share.justanexample.com not responding, still trying

That server has a mount called /R12.2stage that has the installation files for R12.2.

So I tried unmounting it:

umount /R12.2stage

Device Busy

umount -f /R12.2stage

Device Busy

umount -l /R12.2stage

df -h didn't hang any more.

Next I did strace /sbin/fuser $ORACLE_HOME/bin/oracle and it stopped here:

open("/proc/12854/fdinfo/3", O_RDONLY)  = 7

fstat(7, {st_mode=S_IFREG|0400, st_size=0, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b99de014000

read(7, "pos:\t0\nflags:\t04002\n", 1024) = 20

close(7)                                = 0

munmap(0x2b99de014000, 4096)            = 0

getdents(4, /* 0 entries */, 32768)     = 0

close(4)                                = 0

stat("/proc/12857/", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0

open("/proc/12857/stat", O_RDONLY)      = 4

read(4, "12857 (bash) S 12853 12857 12857"..., 4096) = 243

close(4)                                = 0

readlink("/proc/12857/cwd", "11.2.0.4/examples (deleted)"..., 4096) = 27

rt_sigaction(SIGALRM, {0x411020, [ALRM], SA_RESTORER|SA_RESTART, 0x327bc30030}, {SIG_DFL, [ALRM], SA_RESTORER|SA_RESTART, 0x327bc30030}, 8) = 0

alarm(15)                               = 0

write(5, "@\20A\0\0\0\0\0", 8)          = 8

write(5, "\20\0\0\0", 4)                = 4

write(5, "/proc/12857/cwd\0", 16)       = 16

write(5, "\220\0\0\0", 4)               = 4

read(6,  

It stopped here. So I did Ctrl+C

# 

# ps -ef |grep 12857

oracle   12857 12853  0 Apr10 pts/2    00:00:00 -bash

root     21688  2797  0 19:42 pts/8    00:00:00 grep 12857

Killed this process

# kill -9 12857

Again I did strace /sbin/fuser $ORACLE_HOME/bin/oracle and it stopped at a different process this time that was another bash process.  I killed that process also.

I executed it for 3rd time: strace /sbin/fuser $ORACLE_HOME/bin/oracle

This time it completed.

Ran it without strace

/sbin/fuser $ORACLE_HOME/bin/oracle

It came out in 1 second.

Then I did the same process for lsof

strace lsof

and killed those processes were it was getting stuck.  Eventually lsof also worked.

Pipu retried opatch and it worked fine.

Stale NFS mount was the root cause of this issue.  It was stale because the source server was down for Unix security patching during weekend.

 

adoafmctl.sh hangs

Rajesh and Shahed called me about this error where after a reboot of the servers, adoafmctl.sh wouldn't start.  It gave errors like these:

You are running adoafmctl.sh version 120.6.12000000.3 
Starting OPMN managed OAFM OC4J instance ... 
adoafmctl.sh: exiting with status 152 
adoafmctl.sh: check the logfile 
$INST_TOP/logs/appl/admin/log/adoafmctl.txt for more information

adoafmctl.txt showing:

ias-component/process-type/process-set:
default_group/oafm/default_group/
Error
--> Process (index=1,uid=349189076,pid=15039)
time out while waiting for a managed process to start
Log:
$INST_TOP/logs/ora/10.1.3/opmn/default_group~oafm~default_group~1
07/31/09-09:14:28 :: adoafmctl.sh: exiting with status 152
================================================================================
07/31/09-09:14:40 :: adoafmctl.sh version 120.6.12000000.3
07/31/09-09:14:40 :: adoafmctl.sh: Checking the status of OPMN managed OAFM OC4J instance
Processes in Instance: SID_machine.machine.domain
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
default_group | oafm | N/A | Down

Solution:

1. Shutdown all Middle tier services and ensure no defunct processes exist running the following from the operating system:

# ps -ef | grep

If one finds any, kill these processes.

2. Navigate to $INST_TOP/ora/10.1.3/opmn/logs/states directory. It contains hidden file .opmndat:

# ls -lrt .opmndat

3. Delete this file .opmndat after making a backup of it:

# rm .opmndat

4. Restart the services.

5. Re-test the issue.

This resolved the issue.