After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.
I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:
BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
A search on support.oracle.com for keywords
resulted in this hit:
E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)
WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.
Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
Specifically for EBS R12.2 I did this:
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT
adadminsrvctl.sh start
After keying in apps, system and weblogic passwords, it finally started up.
Have to see if this is a recurrent workaround or fixes the issue for good.
I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:
BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
A search on support.oracle.com for keywords
resulted in this hit:
E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)
WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.
Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
Specifically for EBS R12.2 I did this:
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT
adadminsrvctl.sh start
After keying in apps, system and weblogic passwords, it finally started up.
Have to see if this is a recurrent workaround or fixes the issue for good.