After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.
I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:
BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
A search on support.oracle.com for keywords
resulted in this hit:
E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)
WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.
Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
Specifically for EBS R12.2 I did this:
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT
adadminsrvctl.sh start
After keying in apps, system and weblogic passwords, it finally started up.
Have to see if this is a recurrent workaround or fixes the issue for good.
I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:
BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
A search on support.oracle.com for keywords
resulted in this hit:
E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)
WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.
Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
Specifically for EBS R12.2 I did this:
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT
adadminsrvctl.sh start
After keying in apps, system and weblogic passwords, it finally started up.
Have to see if this is a recurrent workaround or fixes the issue for good.
8 comments:
Thanks a lot. It helped me.
It was saved my time.. Thank you
Do you know the reason why this happens?
Do you know why this error happens?
What is the root cause of these error? I need to avoid such scenarios in production. What should be taken care from dev perspective that we don't reach to this state.
Thanks a lot! this really helped!
I guess the root cause is WebLogic Server did not stop cleanly.
Also, when there is change of permission on .DAT files, or chnage of ownershiip of .DAT files.
The scenario will occur.
Vinod
Post a Comment