Oracle Apps Technology

Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Wednesday, April 15, 2015

You Are Trying To Access a Page That Is No Longer Active.The Referring Page May Have Come From a Previous Session. Please Select Home To Proceed

Shahed pinged me about this error.  It was coming after logging in.  This R12.1.3 instance had just migrated from an old server to a new one. Once you logged in this error would be displayed:

You Are Trying To Access a Page That Is No Longer Active.The Referring Page May Have Come From a Previous Session. Please Select Home To Proceed

The hits on support.oracle.com were not helpful, but a gave a clue that it may have something to do with session cookie.  So I used Firefox to check http headers.  If you press Ctrl+Shift+K, you will get a panel at the bottom of the browser. Click on Network tab, click on the AppsLocalLogin.jsp and on the right side of the pane, you'll see a cookie tab.

The domain appearing in the cookie tab was from the old server.  So I checked:

select session_cookie_domain from icx_parameters;
olddomain.justanexample.com

So I nullified it:

update icx_parameters set session_cookie_domain=null;

commit;

Restarted Apache

cd $ADMIN_SCRIPTS_HOME
adapcctl.sh stop
adapcctl.sh start

No more error.  I was able to log in and so was Shahed.

Chrome and E-Business Suite

Dhananjay came to me today.  He said that his users were complaining about forms not launching after upgrading to the latest version of Chrome. On launching forms they got this error:

/dev60cgi/oracle forms engine Main was not found on this server

I recalled that Google Chrome team had announced that they would not support java going forward. Googling with keywords chrome java brought this page:

https://java.com/en/download/faq/chrome.xml#npapichrome

It states that:

NPAPI support by Chrome

The Java plug-in for web browsers relies on the cross platform plugin architecture NPAPI, which has long been, and currently is, supported by all major web browsers. Google announced in September 2013 plans to remove NPAPI support from Chrome by "the end of 2014", thus effectively dropping support for Silverlight, Java, Facebook Video and other similar NPAPI based plugins. Recently, Google has revised their plans and now state that they plan to completely remove NPAPI by late 2015. As it is unclear if these dates will be further extended or not, we strongly recommend Java users consider alternatives to Chrome as soon as possible. Instead, we recommend Firefox, Internet Explorer and Safari as longer-term options. As of April 2015, starting with Chrome Version 42, Google has added an additional step to configuring NPAPI based plugins like Java to run — see the section Enabling NPAPI in Chrome Version 42 and later below.

Enabling NPAPI in Chrome Version 42 and later

As of Chrome Version 42, an additional configuration step is required to continue using NPAPI plugins.
  1. In your URL bar, enter:
    chrome://flags/#enable-npapi 
  2. Click the Enable link for the Enable NPAPI configuration option.
  3. Click the Relaunch button that now appears at the bottom of the configuration page.
Developers and System administrators looking for alternative ways to support users of Chrome should see this blog, in particular "Running Web Start applications outside of a browser" and "Additional Deployment Options" section.
Once Dhananjay did the above steps, Chrome started launching forms again.  He quickly gave these steps to all his users who had upgraded to the latest version of Chrome (version 42) and it started working form them too.
Oracle doesn't certify E-Business Suite forms on Chrome.  Only self service pages of E-Business Suite are certified on Google Chrome.

Saturday, April 11, 2015

opatch hangs on /sbin/fuser oracle

Pipu pinged me today about opatch hanging. The opatch log showed this:

[Apr 11, 2015 5:24:13 PM]    Start fuser command /sbin/fuser $ORACLE_HOME/bin/oracle at Sat Apr 11 17:24:13 EDT 2015

I had faced this issue once before, but was not able to recall what was the solution.  So I started fresh.

As oracle user:

/sbin/fuser $ORACLE_HOME/bin/oracle hung

As root user

/sbin/fuser $ORACLE_HOME/bin/oracle hung

As root user

lsof hung.

Google searches about it brought up a lot of hits about NFS issues.  So I did df -h.

df -h also hung.

So I checked /var/log/messages and found many messages like these:

Apr 11 19:44:42 erpserver kernel: nfs: server share.justanexample.com not responding, still trying

That server has a mount called /R12.2stage that has the installation files for R12.2.

So I tried unmounting it:

umount /R12.2stage
Device Busy

umount -f /R12.2stage
Device Busy

umount -l /R12.2stage

df -h didn't hang any more.

Next I did strace /sbin/fuser $ORACLE_HOME/bin/oracle and it stopped here:

open("/proc/12854/fdinfo/3", O_RDONLY)  = 7
fstat(7, {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b99de014000
read(7, "pos:\t0\nflags:\t04002\n", 1024) = 20
close(7)                                = 0
munmap(0x2b99de014000, 4096)            = 0
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
stat("/proc/12857/", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/12857/stat", O_RDONLY)      = 4
read(4, "12857 (bash) S 12853 12857 12857"..., 4096) = 243
close(4)                                = 0
readlink("/proc/12857/cwd", "11.2.0.4/examples (deleted)"..., 4096) = 27
rt_sigaction(SIGALRM, {0x411020, [ALRM], SA_RESTORER|SA_RESTART, 0x327bc30030}, {SIG_DFL, [ALRM], SA_RESTORER|SA_RESTART, 0x327bc30030}, 8) = 0
alarm(15)                               = 0
write(5, "@\20A\0\0\0\0\0", 8)          = 8
write(5, "\20\0\0\0", 4)                = 4
write(5, "/proc/12857/cwd\0", 16)       = 16
write(5, "\220\0\0\0", 4)               = 4
read(6,  

It stopped here. So I did Ctrl+C
# ps -ef |grep 12857
oracle   12857 12853  0 Apr10 pts/2    00:00:00 -bash
root     21688  2797  0 19:42 pts/8    00:00:00 grep 12857

Killed this process

# kill -9 12857

Again I did strace /sbin/fuser $ORACLE_HOME/bin/oracle and it stopped at a different process this time that was another bash process.  I killed that process also.

I executed it for 3rd time: strace /sbin/fuser $ORACLE_HOME/bin/oracle

This time it completed.

Ran it without strace

/sbin/fuser $ORACLE_HOME/bin/oracle

It came out in 1 second.

Then I did the same process for lsof

strace lsof

and killed those processes were it was getting stuck.  Eventually lsof also worked.

Pipu retried opatch and it worked fine.

Stale NFS mount was the root cause of this issue.  It was stale because the source server was down for Unix security patching during weekend.
 

Friday, April 3, 2015

adoafmctl.sh hangs

Rajesh and Shahed called me about this error where after a reboot of the servers, adoafmctl.sh wouldn't start.  It gave errors like these:

You are running adoafmctl.sh version 120.6.12000000.3 
Starting OPMN managed OAFM OC4J instance ... 
adoafmctl.sh: exiting with status 152 
adoafmctl.sh: check the logfile 
$INST_TOP/logs/appl/admin/log/adoafmctl.txt for more information

adoafmctl.txt showing:
ias-component/process-type/process-set:
default_group/oafm/default_group/
Error
--> Process (index=1,uid=349189076,pid=15039)
time out while waiting for a managed process to start
Log:
$INST_TOP/logs/ora/10.1.3/opmn/default_group~oafm~default_group~1
07/31/09-09:14:28 :: adoafmctl.sh: exiting with status 152
================================================================================
07/31/09-09:14:40 :: adoafmctl.sh version 120.6.12000000.3
07/31/09-09:14:40 :: adoafmctl.sh: Checking the status of OPMN managed OAFM OC4J instance
Processes in Instance: SID_machine.machine.domain
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
default_group | oafm | N/A | Down

Solution:

1. Shutdown all Middle tier services and ensure no defunct processes exist running the following from the operating system:
# ps -ef | grep

If one finds any, kill these processes.
2. Navigate to $INST_TOP/ora/10.1.3/opmn/logs/states directory. It contains hidden file .opmndat:
# ls -lrt .opmndat
3. Delete this file .opmndat after making a backup of it:
# rm .opmndat
4. Restart the services.

5. Re-test the issue.

This resolved the issue.

Monday, March 23, 2015

R12.2 Documentation link in html format

This link has the R12.2 documentation in HTML format:

https://docs.oracle.com/cd/E26401_01/index.htm 

Sunday, March 1, 2015

The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes

I got this error while upgrading an R12.1.3 instance to R12.2.4, when I completed AD.C.Delta 5 patches with November 2014 bundle patches for AD.C and was in the process of applying TXK.C.Delta5 with November 2014 bundle patches for TXK.C :

Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
[START 2015/03/01 04:53:16] Check if services are down
        [INFO] Run admin server is not down
     [WARNING]  Hotpatch mode should only be used when directed by the patch readme.
  [EVENT]     [START 2015/03/01 04:53:17] Performing database sanity checks
    [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .
    Log file: /erppgzb1/erpapp/fs_ne/EBSapps/log/adop/adop_20150301_045249.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)

I was really surprised as I had already run EBS technology codelevel checker (patch 17537119) script checkDBpatch.sh on racnode1.

To investigate I checked inside checkDBpatch.sh and found that it create a table called TXK_TCC_RESULTS.  

SQL> desc txk_tcc_results
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 TCC_VERSION                               NOT NULL VARCHAR2(20)
 BUGFIX_XML_VERSION                        NOT NULL VARCHAR2(20)
 NODE_NAME                                 NOT NULL VARCHAR2(100)
 DATABASE_NAME                             NOT NULL VARCHAR2(64)
 COMPONENT_NAME                            NOT NULL VARCHAR2(10)
 COMPONENT_VERSION                         NOT NULL VARCHAR2(20)
 COMPONENT_HOME                                     VARCHAR2(600)
 CHECK_DATE                                         DATE
 CHECK_RESULT                              NOT NULL VARCHAR2(10)
 CHECK_MESSAGE                                      VARCHAR2(4000)

SQL> select node_name from txk_tcc_results;

NODE_NAME
--------------------------------------------------------------------------------
RACNODE1

I ran checkDBpatch.sh again, but the patch failed again with previous error:

   [ERROR]     The EBS Technology Codelevel Checker (available as Patch 17537119) needs to be run on the following nodes: .

It was Saturday 5 AM already working through the night.  So I thought, it is better to sleep now and tackle this on Sunday.  On Sunday morning after a late breakfast, I looked at the problem again.  This time, I realized that the error was complaining about racnode1 (in lower case) and the txk_tcc_results table had RACNODE1(in upper case).  To test my hunch, I immediately updated the value:

update txk_tcc_results
set node_name='racnode1' where node_name='RACNODE1';

commit;

I restarted the patch, and it went through.  Patch was indeed failing because it was trying to look for a lower case value.  I will probably log an SR with Oracle, so that they change their code to make the node_name check case insensitive.

Further, I was curious, why node_name was stored in all caps in fnd_nodes and txk_tcc_results.  The file /etc/hosts had it in lowercase.  I tried the hostname command on linux prompt:

$ hostname
RACNODE1

That was something unusual, as in our environment, hostname always returns the value in lowercase.  So I further investigated.
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = RACNODE1

So I changed it

[root@RACNODE1 ~]# sysctl kernel.hostname=RACNODE1
kernel.hostname = racnode1
[root@RACNODE1 ~]# sysctl kernel.hostname
kernel.hostname = racnode1
[root@RACNODE1 ~]#
[root@RACNODE1 ~]# hostname
racnode1

Logged in again to see if root prompt changed:

[root@racnode1 ~]#

I also checked
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=RACNODE1

Changed it here also:
[root@tsgld5811 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTNAME=racnode1

I also changed it on racnode2.