Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps

Tuesday, January 7, 2014

weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
        at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
        at weblogic.store.io.file.Heap.open(Heap.java:325)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)


A search on support.oracle.com for keywords
resulted in this hit:

E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)

WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.

Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\

  
Specifically for EBS R12.2 I did this:

cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
 cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT

adadminsrvctl.sh start

After keying in apps, system and weblogic passwords, it finally started up.

Have to see if this is a recurrent workaround or fixes the issue for good.

8 comments:

Yogesh said...

Thanks a lot. It helped me.

Unknown said...

It was saved my time.. Thank you

TheAnkh said...

Do you know the reason why this happens?

Unknown said...

Do you know why this error happens?

Gaurav Srivastava said...

What is the root cause of these error? I need to avoid such scenarios in production. What should be taken care from dev perspective that we don't reach to this state.

Unknown said...

Thanks a lot! this really helped!

Irman said...

I guess the root cause is WebLogic Server did not stop cleanly.

Vinod said...

Also, when there is change of permission on .DAT files, or chnage of ownershiip of .DAT files.
The scenario will occur.

Vinod