Weblogic hangs at BEA-002014 IIOP subsystem enabled

After a server reboot, OTM Weblogic would not start up. It was forever waiting after showing these lines in weblogic.log:

####Sep 15, 2011 5:22:38 PM EDT Info IIOP weblogic.justanexample.com gc3-appotms1 [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' WLS Kernel 1316121758429 BEA-002014 IIOP subsystem enabled.

A google search with keywords "Weblogic hang BEA-002014 IIOP subsystem enabled" led me to

https://forums.oracle.com/forums/thread.jspa?threadID=923601

Vivek Tripathi gave a solution two years ago in 2009:

Hmmmm ... Last line in the logs are
"<2/07/2009 08:54:10 PM CDT> "

Usually after enabling IIOP subsystem weblogic server initiates "security". I will suggest to rename the "ldap" directory present under location "\servers\\data" and start the server.
This issue may be due to corrupt LDAP data.

I stopped Apache and Weblogic

cd $GLOG_HOME/weblogic/domains/otm/servers/gc3-appotms1/data

mv ldap ldap.old

Restarted Apache and Weblogic

Weblogic came up fine after this.

[HTTP:101216]Servlet: "action" failed to preload on startup in Web application: "ccg.war".

While deploying ccg in a freshly installed Weblogic 10.3.5, this error was coming:

[HTTP:101216]Servlet: "action" failed to preload on startup in Web application: "ccg.war". java.lang.NullPointerException at org.apache.commons.digester.Digester.getXMLReader(Digester.java:899) at org.apache.commons.digester.Digester.parse(Digester.java:1647) at org.apache.struts.action.ActionServlet.parseModuleConfigFile(ActionServlet.java:708) at org.apache.struts.action.ActionServlet.initModuleConfig(ActionServlet.java:670) at org.apache.struts.action.ActionServlet.init(ActionServlet.java:329) at javax.servlet.GenericServlet.init(GenericServlet.java:241) at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:283) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:64) at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58) at weblogic.servlet.internal.StubLifecycleHelper.(StubLifecycleHelper.java:48) at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:539) at weblogic.servlet.internal.WebAppServletContext.preloadServlet(WebAppServletContext.java:1985) at weblogic.servlet.internal.WebAppServletContext.loadServletsOnStartup(WebAppServletContext.java:1959) at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1878) at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3153) at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1508) at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:482) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52) at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119) at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200) at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52) at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119) at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:27) at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:636) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52) at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:205) at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:43) at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161) at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:569) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:323) at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:844) at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1253) at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:440) at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:163) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:195) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:13) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:68) at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209) at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

The issue is caused by the following setup: Existence of ‘xercesImpl.jar’ file in the ccg-war directory (Location: /WEB-INF/lib) caused errors on the Weblogic 11g application server while deplying CCG applications.

As per My Oracle Support article "CCG 5.5.1 Deployment With Weblogic 10.3.0.1/11g Fails [ID 1265899.1]"

This is explained in the following bug:
Bug 10251676 - CCG 5.5.1 DEPLOYMENT WITH WEBLOGIC 10.3.0.1 FAILS

Solution
1. Delete Weblogic cache and restart it before applying the workaround below.
2. Remove the ‘xercesImpl.jar’ file from ccg-war directory (Location: /WEB-INF/lib)
3. Redeploy CCG applications on Weblogic 11g application server. (Refer to the CCG Install Guide v5.5.1 for detailed instructions.

Weblogic Cache exists in $MW_HOME/user_projects/domains/ccg/servers/AdminServer/tmp
I had named my domain ccg. You can replace that with your domain name in the above path.

So I did this:

Stop Weblogic Server
rm /ccg/WEB-INF/lib/xercesImpl.jar
rm -rf $MW_HOME/user_projects/domains/ccg/servers/AdminServer/tmp
mkdir $MW_HOME/user_projects/domains/ccg/servers/AdminServer/tmp
Start Weblogic Server

The issue was resolved.

GRC An error occurred during the login process

On a freshly installed Oracle EGRC(Enterprise Governance Risk and Compliance) 8.6.3 on OEL 5.6, I got this error in front end:

An error occurred during the login process.

On checking $MW_HOME/grc863/grc/log/grc.log, I found this error:

2011-09-13 14:27:25,529 ERROR [el.Default (self-tuning)'] LoginServiceImpl:233 Error during User Authentication : No Configuration was registered that can handle the configuration named JdbcAuth
java.lang.IllegalArgumentException: No Configuration was registered that can handle the configuration named JdbcAuth
at com.bea.common.security.jdkutils.JAASConfiguration.getAppConfigurationEntry(JAASConfiguration.java:130)
at oracle.apps.grc.security.AuthenticationEnforcer.authenticate(AuthenticationEnforcer.java:77)
at oracle.apps.grc.ui.login.server.LoginServiceImpl.isLogin(LoginServiceImpl.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
at oracle.apps.grc.ui.common.server.context.AACGRemoteServiceServlet.processCall(AACGRemoteServiceServlet.java:73)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.apps.grc.ui.webapp.servlet.ApplicationResourceRequestFilter.doFilter(ApplicationResourceRequestFilter.java:262)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)


I checked startManagedWeblogic.sh script and found that there was a space before jaas.conf

JAVA_OPTIONS="-Xms8192m -Xmx16384m -Xss512k -XX:MaxPermSize=512m -Doracle.security.jps.auth.debug=true -Doracle.security.jps.auth.debug.verbose=true -Djava.security.debug=access,failure -Djava.security.auth.login.config="/GE/erpgrca1/grcapp/weblogic/grc863/grc/WEB-INF/ jaas.config" -Djava.awt.headless=true -Dweblogic.security.SSL.trustedCAKeyStore="/GE/erpgrca1/grcapp/weblogic/wlserver_10.3/server/lib/cacerts" ${JAVA_OPTIONS}"
export JAVA_OPTIONS

I started Weblogic again and a new error showed up in grc.log:

2011-09-13 14:43:05,456 ERROR [el.Default (self-tuning)'] LoginServiceImpl:233 Error during User Authentication : No Configuration was registered that can handle the configuration named JdbcAuth
java.lang.IllegalArgumentException: No Configuration was registered that can handle the configuration named JdbcAuth
at com.bea.common.security.jdkutils.JAASConfiguration.getAppConfigurationEntry(JAASConfiguration.java:130)
at oracle.apps.grc.security.AuthenticationEnforcer.authenticate(AuthenticationEnforcer.java:77)
at oracle.apps.grc.ui.login.server.LoginServiceImpl.isLogin(LoginServiceImpl.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
at oracle.apps.grc.ui.common.server.context.AACGRemoteServiceServlet.processCall(AACGRemoteServiceServlet.java:73)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.apps.grc.ui.webapp.servlet.ApplicationResourceRequestFilter.doFilter(ApplicationResourceRequestFilter.java:262)

I went through the documentation again and realized that I needed to do this:

If you are installing EGRCC to perform analysis in Oracle EBS or PeopleSoft instances:
1. Stop the Administration Server.
2. Open the file StartWebLogic.sh in a text editor. The file resides in the /user_projects/domains//bin directory.
3. Add the following line to the file, replacing with the full path to the grc863 directory you created earlier (see step 5 of ―Preparing Additional Files‖ on page 2-7).

SAVE_JAVA_OPTIONS="-Xms512m -Xmx4096m -Xss512k -XX:MaxPermSize=512m -Doracle.security.jps.auth.debug=true -Doracle.security.jps.auth.debug.verbose=true -Djava.security.debug=access,failure -Djava.security.auth.login.config="/grc/WEB-INF/ jaas.config" -Djava.awt.headless=true -Dweblogic.security.SSL.trustedCAKeystore="/wl server_10.3/server/lib/cacerts" ${JAVA_OPTIONS}"

You may use a maximum memory setting (-Xmx) larger than 4096m if your server has enough memory to support the larger value.
4. Start the Administration Server

Once I did the abve steps, I was able to login to GRC with the default username/password : admin/admin

Sendmail connects to localhost only

In a new Solaris build, we go this problem about sendmail. It was connecting on sendmail port 25 only if hostname was localhost or 127.0.0.1. It was not listening on all the other IPs plumbed on the box. Here's what I did to make it listen on all IPs:

Stop sendmail
svcadm disable sendmail

cd /etc/mail
chmod 600 sendmail.cf

Search for this line:
O DaemonPortOptions=NAME=NoMTA4, Family=inet, Addr=127.0.0.1

Remove , Addr=127.0.0.1 so that the line now reads:
O DaemonPortOptions=NAME=NoMTA4, Family=inet

chmod 444 sendmail.cf

Start Sendmail
svcadm enable sendmail

Retest by doing

telnet hostname_of_server 25

If it still doesn't work, do this:

svccfg -s sendmail setprop config/local_only=false

svcadm refresh sendmail

svcadm restart sendmail