A connection to the server could not be established: null (user=DBA.ADMIN)

This morning I got a call from various people complaining about OTM not allowing them to login. Anyone who tried logging in got this error:

A connection to the server could not be established: null (user=any_valid_otm_user)

I checked $GLOG_HOME/logs/weblogic/console.log.0:


INFO | 2011/12/14 09:40:07 | -- OTM Event: serverReady
INFO | 2011/12/14 09:47:01 | javax.management.MBeanException: MBean invoke failed: weblogic.management.mbeanservers.edit.EditTimedOutException:
INFO | 2011/12/14 09:47:01 |
INFO | 2011/12/14 09:47:01 | javax.management.MBeanException: MBean invoke failed: weblogic.management.mbeanservers.edit.EditTimedOutException:
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:205)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
INFO | 2011/12/14 09:47:01 | at javax.management.remote.rmi.RMIConnectionImpl_1033_WLStub.invoke(Unknown Source)
INFO | 2011/12/14 09:47:01 | at weblogic.management.remote.common.RMIConnectionWrapper$16.run(ClientProviderBase.java:876)
INFO | 2011/12/14 09:47:01 | at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
INFO | 2011/12/14 09:47:01 | at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
INFO | 2011/12/14 09:47:01 | at weblogic.security.Security.runAs(Security.java:61)
INFO | 2011/12/14 09:47:01 | at weblogic.management.remote.common.RMIConnectionWrapper.invoke(ClientProviderBase.java:874)
INFO | 2011/12/14 09:47:01 | at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
INFO | 2011/12/14 09:47:01 | at glog.util.jmx.JMXConnection.call(JMXConnection.java:104)
INFO | 2011/12/14 09:47:01 | at glog.util.j2ee.remote.WLManagementHelper$Edit.(WLManagementHelper.java:54)
INFO | 2011/12/14 09:47:01 | at glog.util.j2ee.jdbc.WLConnectionPoolInstance.create(WLConnectionPoolInstance.java:139)
INFO | 2011/12/14 09:47:01 | at glog.util.j2ee.jdbc.ASConnectionPoolInstance.(ASConnectionPoolInstance.java:52)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.AppServerConnectionPoolInstance.(AppServerConnectionPoolInstance.java:9)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.AppServerConnectionPool.getNewInstance(AppServerConnectionPool.java:71)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.AppServerConnectionPool.getInstance(AppServerConnectionPool.java:61)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.AppServerConnectionPool.reserve(AppServerConnectionPool.java:34)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.DataSource.reserve(DataSource.java:143)
INFO | 2011/12/14 09:47:01 | at glog.util.datasource.DataSource.reserve(DataSource.java:137)
INFO | 2011/12/14 09:47:01 | at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:301)
INFO | 2011/12/14 09:47:01 | at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.addRef(T2SharedConnection.java:291)
INFO | 2011/12/14 09:47:01 | at glog.util.jdbc.noserver.T2SharedConnection.open(T2SharedConnection.java:57)
INFO | 2011/12/14 09:47:01 | at glog.server.useraccess.UserAccessSessionBean.getUserAccess(UserAccessSessionBean.java:154)
INFO | 2011/12/14 09:47:01 | at glog.server.useraccess.UserAccessSessionServerSideEJBWrapper.getUserAccess(UserAccessSessionServerSideEJBWrapper.java:87)
INFO | 2011/12/14 09:47:01 | at glog.server.useraccess.UserAccessSessionHome_uzvgzz_EOImpl.getUserAccess(UserAccessSessionHome_uzvgzz_EOImpl.java:1600)
INFO | 2011/12/14 09:47:01 | at glog.server.useraccess.UserAccessSessionHome_uzvgzz_EOImpl_WLSkel.invoke(Unknown Source)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
INFO | 2011/12/14 09:47:01 | at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
INFO | 2011/12/14 09:47:01 | at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
INFO | 2011/12/14 09:47:01 | at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
INFO | 2011/12/14 09:47:01 | at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
INFO | 2011/12/14 09:47:01 | at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

I googled and found a few hits and in one of them found this line:

When I'm trying to deploy my project on weblogic I've got this error:
2007-02-09 16:05:15 CET Error Deployer BEA-149181 Deployment Service encountered error while aborting the edit session before start. Initiato
r for this edit session is 'principals=[kernel identity = -1632281053]'. Error is 'weblogic.management.provider.EditNotEditorException: Not edit lock
owner'
weblogic.management.provider.EditNotEditorException: Not edit lock owner

On seeing those words: Not edit lock owner, I realized that the previous day, I was trying to create SFTP File control in OTM Weblogic console. It wasn't letting me create a new one, till I clicked Lock and Edit. So I quickly logged into weblogic console and clicked on Release Configuration button. The subsequent logins to OTM started working after this.

OTM HTTP-400 Bad Request

Some users reported HTTP-400 Bad Request errors in ERPOTMD1. Here's what was happening:

1. If the first window you open was an OTM instance, everything worked fine.

2. If you started E-Business Suite and then accessed OTM, HTTP-400 Bad Request would result.

3. This was happening in IE8, Firefox6.

4. Apache error.log showed:

[Wed Aug 24 14:41:43 2011] [error] [client 3.75.208.77] request failed: error reading the headers


I googled and found this:

http://allthingsunix.inside.quest.com/thread.jspa?threadID=4667

This post advised:

request headers can be quite long, try setting

LimitRequestFieldsize 12392


in the server context.

See http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestfieldsize

So I checked the value of this variable in Apache httpd.conf and found that it was set to:

LimitRequestFieldSize 2048
LimitRequestLine 4096


I changed it to

LimitRequestFieldSize 20480
LimitRequestLine 40960


And bounced Apache and Weblogic.

Problem was resolved. For official confirmation from Oracle about this fix, I logged SR. Oracle OTM Development was ok with this change.

OTM java.lang.NullPointerException

In OTM we got this error reported by end users:

On many screens for example User Manager > New Button > key in all details > Click Finish.

Users Finder > Results

Results
Errors
Create Another Modify Another

Error

java.lang.NullPointerException

Stack Trace
java.lang.NullPointerException

java.lang.NullPointerException
at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.setAutoCommit(T2SharedConnection.java:421)
at glog.util.jdbc.noserver.T2SharedConnection$ReferencedConnection.release(T2SharedConnection.java:326)
at glog.util.jdbc.noserver.T2SharedConnection.close(T2SharedConnection.java:61)
at glog.database.security.dbrealm.RdbmsRealm.createUser(RdbmsRealm.java:642)
at glog.database.security.dbrealm.RdbmsRealm.createUser(RdbmsRealm.java:652)
at glog.database.security.dbrealm.CachingRealm.newUser(CachingRealm.java:483)
at glog.database.security.SecurityUserImpl.create(SecurityUserImpl.java:65)
at glog.database.security.SecurityUtil.createUser(SecurityUtil.java:52)
at glog.database.security.SecuritySessionBean.createUser(SecuritySessionBean.java:40)
at glog.database.security.SecuritySessionServerSideEJBWrapper.createUser(SecuritySessionServerSideEJBWrapper.java:167)
at glog.database.security.SecuritySessionHome_snlvan_EOImpl.createUser(SecuritySessionHome_snlvan_EOImpl.java:5947)
at glog.database.security.SecuritySessionHome_snlvan_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)






Fix Errors


This was due to the fact that DB Listener was down. Once DB listener was brought up, this error disappeared.

Files in which OTM stores IP address and port

During the recent upgrade of OTM v6.1.0 to OTM v6.2.1, I discovered that OTM stores IP address of its servers in the following files:

$GLOG_HOME/tomcat/conf/server.xml
$GLOG_HOME/weblogic/domains/otm/config/config.xml.fresh
$GLOG_HOME/apache/conf/httpd.conf
$GLOG_HOME/apache/conf/workers.properties


For Apache and Weblogic services, OTM listens on virtual IP of the instance, so there is no port conflict as the Virtual_IP:Port remains unique.

However, the Tomcat component listens on localhost on port 8007 by default. So if you have two OTM instances share the same physical box as application tier, you need to change the default port of 8007 to some other port which is available. This change is made in $GLOG_HOME/tomcat/conf/server.xml.

Oracle Transportation Management (OTM) installation on Solaris 10

We got a new project which uses OTM. Here are the high level steps for installation of OTM:

1. Go to edelivery.oracle.com, enter your information and login.
2. Select E-Business Suite in dropdown
3. Scroll Down to Oracle Transportation Management
4. Download OTM and Weblogic Server
5. Download RDBMS 11gR2 Server and Client from Oracle Technology Network
6. Install 11gR2 Database and create a blank database
7. Install Weblogic Server
8. Install OTM
9. Execute the file $OTM_HOME/install/gc3env.sh
10. Go to $OTM_HOME/glog/oracle/script8 and edit a file called create_all.sql modify the line which reads:
41 SELECT DECODE(UPPER(nvl('&&4','Y')),'Y','./import_content.sh','N','imp
ort_content.cmd') ws_content_import

Change ./import_content.sh to . ./import_content.sh

If you don't do this, this script fails. This issue is documented in My Oracle Support Articles:
Installation or Oracle Transportation Management 6.0 on Solaris [ID 974412.1]
Cannot Log on to a new OTM 6.0 Installation [ID 975529.1]