weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
        at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
        at weblogic.store.io.file.Heap.open(Heap.java:325)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

A search on support.oracle.com for keywords
resulted in this hit:

E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)

WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.

Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
  
Specifically for EBS R12.2 I did this:

cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
 cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT

adadminsrvctl.sh start

After keying in apps, system and weblogic passwords, it finally started up.

Have to see if this is a recurrent workaround or fixes the issue for good.

R12.2 Failure of server APACHE bridge

Failure of server APACHE bridge:

---

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

If there is an issue with the oacore managed server, meaning apache is not able to connect to the oacore managed server, you'll see this error.

It is possible that the managed server oacore is not yet started, and you try to access the EBS URL.
It is possible that the managed server for oacore is not able to start up because of a port conflict.

R12.2 Apache and Weblogic log file locations

Apache Logs

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_*/*log

OPMN Log
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

Weblogic Logs

$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*
$EBS_DOMAIN_HOME/servers/forms*/logs/*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*
$EBS_DOMAIN_HOME/sysman/log/*

Source: Oracle® E-Business Suite
Maintenance Guide
Release 12.2

adpreclone.pl fails with segmentation fault, bus error, out of memory

Manoj pinged me about this issue last night.  Whenever he was trying to do adpreclone.pl dbTier or AppsTier, when he entered the apps password, the process would crash with either segmentation fault, bus error or out of memory.

I tried executing top and it was also ending up with segmentation fault some of the time.  At the times it would start, it would give a flashing

I checked the server memory and it had 73 GB free memory.

Eventually  I opened up the adpreclone.pl script in vi to study what it was doing.  At one place, I noticed that it was looking for the environment variable $USER.  When I checked, on the server, echo $USER, did not return anything.  I checked on other working servers and echo $USER returned the unix applmgr username.

So I manually set

USER=applmgr
export USER
echo $USER

perl adpreclone.pl appsTier

Entered apps password, and it worked fine.

This was a new server. Seems like unix team missed something during this build.  I tried looking up /etc/profile, but wasn't able to find out why the environment variable USER was not set.  I'll update this post, when I discover the reason.

R12.2.2 is now available on edelivery.oracle.com

At last, R12.2.2 is now available on edelivery.oracle.com

550 5.7.1 Relaying denied. IP name lookup failed

Recently we configured a BigIP virtual for SMTP called smtp.justanexample.com

Whenever I tried sending mail through this it would complain:

# telnet smtp.justanexample.com 25


Trying 192.168.10.201...

Connected to smtp.justanexample.com
Escape character is '^]'.

220 smtp1.justanexample.com ESMTP Sendmail 8.13.8/8.13.8; Mon, 15 Apr 2013 22:34:54 -0400

mail from:vikram.das@justanexample.com

250 2.1.0 vikram.das@justanexample.com... Sender ok

rcpt to:vikram.das@justanexample.com

550 5.7.1 vikram.das@justanexample.com... Relaying denied. IP name lookup failed [192.168.10.2]

Doing an nslookup on this name failed.  If I did https://192.168.10.2 it would go to the BigIP user interface.  This IP address was somewhere related to the BigIP device, and did not have a hostname in DNS.  I have sent a mail to the network team for further investigation about this IP.

The usual practice for allowing relay from certain hosts is to add those hosts to /etc/mail/local-host-names.  However since this IP was not present in DNS, I added the following line to /etc/mail/access on all smtp nodes :

Connect:192.168.10.2             RELAY

and bounced sendmail on Linux

/etc/init.d/sendmail restart

The Relaying Denied is not coming anymore.