DBUA is unable to connect to X11

In one of the iterations of 10g upgrade, dbua wouldn't launch and give this error:

Exception in thread "main" java.lang.InternalError: Can't connect to X11 window server using 'DB001:6' as the value of the DISPLAY variable.
[Loaded java.lang.StackTraceElement from /gpsess76/oracle/10.2.0/jdk/jre/lib/rt.jar]
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:134)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:62)
at java.awt.Font.initializeFont(Font.java:308)
at java.awt.Font.(Font.java:344)
at oracle.ewt.lwAWT.LWComponent.(Unknown Source)
at oracle.sysman.assistants.util.wizard.WizardObject.(WizardObject.java:64)
at oracle.sysman.assistants.dbma.ui.DBMAWizard.(DBMAWizard.java:184)
at oracle.sysman.assistants.dbma.ui.UICompManager.(UICompManager.java:153)
at oracle.sysman.assistants.dbma.Dbma.getCompManager(Dbma.java:154)
at oracle.sysman.assistants.dbma.Dbma.execute(Dbma.java:98)
at oracle.sysman.assistants.dbma.Dbma.statusMain(Dbma.java:199)
at oracle.sysman.assistants.dbma.Dbma.main(Dbma.java:180)
[Loaded java.lang.Shutdown from /gpsess76/oracle/10.2.0/jdk/jre/lib/rt.jar]
[Loaded java.lang.Shutdown$Lock from /gpsess76/oracle/10.2.0/jdk/jre/lib/rt.jar]

The X environment was properly set. DISPLAY variable was correct. We were able to launch xclock from the same session without issues. We were also able to start dbua of other instances in the same vnc session, we followed metalink note 443693.1 (on March 24 and 25 2008 this note was available, now this note is not visible) which suggests
modifying dbua: Remove the ‘$JRE_OPTION’ where there is a definition for this
‘–d64’ flag and it will work fine using the default –d32 data model.

We removed this block from dbua as suggested in 443693.1
JRE_OPTION=
if [ $PLATFORM = Solaris ]; then
MACH_HARDWARE=`/bin/uname -i`
case $MACH_HARDWARE in
i86pc)
LD_LIBRARY_PATH=/opt/ORCLcluster/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
;;
*)
JRE_OPTION=-d64
LD_LIBRARY_PATH_64=/opt/ORCLcluster/lib:$LIB_DIR:$ORACLE_HOME/network/lib:$L
D_LIBRARY_PATH_64
export LD_LIBRARY_PATH_64
;;
esac
fi

After this DBUA launched fine. We got errors later in the upgrade process like this one which is logged in the DBUA trace.log:

/dev11i/oracle/10.2.0/oui/lib/solaris64/liboraInstaller.so: wrong ELF class: ELFCLASS64 followed by a long list of java class errors.

DBUA jumped from 8% to 87% in 1 second after this error and after discussions with Oralce we had to follow the manual route of upgrading through catupgrd.sql.

In the morning today, we faced the same issue of DBUA refusing to launch in another instance. This time we did not hurry with a solution but waited and discussed. Subu asked us to compare the instances where it was running fine. We checked ulimit -a on working and non-working instances. On the instances dbua was working nofiles was set to 65536, on instance where dbua was throwing X11 error, nofiles was set to 2048. So we inserted this command in the .profile of the oracle user:

ulimit -n 65536

Logged in again as oracle user and checked ulimit
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 65536
memory(kbytes) unlimited

When we launched dbua after this, it launched without any errors. The issue was resolved. It seems that 64 bit dbua process tries to open more than 2048 files, and unfortunately the 2049th file happens to be an X11 window, because of which we were getting those errors. After resetting nofiles to 65536, it was able to open those files.