Unable to locally verify the issuer's authority

Chuka pinged me when he got this error in Qualys logs after installation of qualys agent on a server

2017-07-24 15:23:08.363 [qualys-cloud-agent][232147]:[Information]:Webservice uri: https://qagpublic.qg1.apps.qualys.com/CloudAgent/v1.1/customer/9c0e25de-0221-5af6-e040-10ac13043f6a/agent/e6ea9ca7-a662-4a4d-8412-2ec70077d7a9/CAPI

2017-07-24 15:23:08.497 [qualys-cloud-agent][232147]:[Information]:Finished curl request

2017-07-24 15:23:08.497 [qualys-cloud-agent][232147]:[Error]:Http request failed:Peer certificate cannot be authenticated with given CA certificates: SSL certificate problem: unable to get local issuer certificate

2017-07-24 15:23:08.497 [qualys-cloud-agent][232147]:[Error]:Http request failed: error code: 0

2017-07-24 15:23:08.497 [qualys-cloud-agent][232147]:[Error]:CAPI request failed:

2017-07-24 15:23:08.497 [qualys-cloud-agent][232147]:[Error]:CAPI event failed

2017-07-24 15:23:08.500 [qualys-cloud-agent][232147]:[Information]:Next event: INTERVAL_EVENT_CAPI, time left: 100 seconds

I suggested that we try setting http_proxy and https_proxy environment variables and check with wget and curl

We did that and got the error:

Unable to locally verify the issuer's authority

On access.redhat.com, I found https://access.redhat.com/solutions/37323 that described a similar issue on RHEL5, we are on OEL6:

Getting "Unable to locally verify the issuer's authority" error from wget in RHEL 5

 SOLUTION VERIFIED - Updated February 6 2013 at 11:52 AM - 

English 

Environment

• Red Hat Enterprise Linux (RHEL) 5.5

Issue

• When downloading from certain SSL URLs, wget throws the error "Unable to locally verify the issuer's authority".

Resolution

• This issue has been resolved as of RHEL 5.7 in openssl-0.9.8e-20.el5, for more information see the following errata: http://rhn.redhat.com/errata/RHBA-2011-1010.html

Root Cause

• The openssl /etc/pki/tls/certs/ca-bundle.crt file in RHEL 5.5 is outdated.

Diagnostic Steps

Reproduction steps:

$ wget https://www.internetx.com/
--2010-07-24 15:30:14--  https://www.internetx.com/
Resolving www.internetx.com... 85.236.36.48, 2001:4178:2:10::54, 2001:4178:2:10::55, ...
Connecting to www.internetx.com|85.236.36.48|:443... connected.
ERROR: cannot verify www.internetx.com's certificate, issued by '/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA':
  Unable to locally verify the issuer's authority.
To connect to www.internetx.com insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

So we compared the size of the file  /etc/ssl/certs/ca-certificates.crt on a working server and this server.  Sure enough the file size was different.  On further investigation we found that there was a backup called ca-bundle.crt.newrpm that had the correct size.  So we took a backup of ca-bundle.crt and replaced it with the newer file.  The error stopped coming.  Even though the ca-certificates rpm version was same on both servers, someone had replaced the ca-bundle.crt file with the older version.  We would have to investigate, why that was done, before this solution can be implemented, as it is possible that the new certificates.crt file broke something, because of which it was replaced by the older version.  We should always have the latest ca-bundle.crt, so that we have the latest root certificates from the Certificate Authorities.

acroread on Linux 6

As per https://access.redhat.com/errata/RHSA-2013:1402 : Adobe Reader (acroread) allows users to view and print documents in Portable Document Format (PDF). Adobe Reader 9 reached the end of its support cycle on June 26, 2013, and will not receive any more security updates. Future versions of Adobe Acrobat Reader will not be available with Red Hat Enterprise Linux.

Some of our ERPs still use acroread for pasta printing.  When we did the upgrade of OS to OEL 6, we had to reinstall adobe by getting the rpm from adobe site ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/ that I got from https://www.reddit.com/r/linux/comments/2hsgq6/linux_version_of_adobe_reader_no_longer/   Fortunately, adobe still hosts it. It is also available from Red Hat if you have a subscription:

Red Hat Enterprise Linux Server 6

SRPM
x86_64
acroread-9.5.5-1.el6_4.1.i686.rpm	SHA-256: ac0934cfe887c6f49238ebff2c963adc33c1531019660bbfa4f8852d725136a2	Download
acroread-plugin-9.5.5-1.el6_4.1.i686.rpm	SHA-256: 8116e41e4825f74478731a7e970482274db5c9d4b9489b9f70ca4e1a278e526b	Download
i386
acroread-9.5.5-1.el6_4.1.i686.rpm	SHA-256: ac0934cfe887c6f49238ebff2c963adc33c1531019660bbfa4f8852d725136a2	Download
acroread-plugin-9.5.5-1.el6_4.1.i686.rpm	SHA-256: 8116e41e4825f74478731a7e970482274db5c9d4b9489b9f70ca4e1a278e526b	Download

After installing the rpm with rpm -ivh AdbeRdr9.5.5-1_i486linux_enu.rpm command, we got dependency error for libxml2.so.  On installing libxml2.so, we got error for other missing dependencies.  So eventually we did yum install AdbeRdr9.5.5-1_i486linux_enu.rpm which installed about 80 rpms that acroread needs to function.

After installation, we kept getting this error:

dirname: missing operand

The post on http://www.linuxquestions.org/questions/slackware-14/error-message-dirname-missing-operand-when-starting-acroread-827012/ recommended a fix:

The issue was fixed by commenting line 529 in the bash script /opt/Adobe/Reader/bin/acrobat

529 #  [ -n "${MozPath}" ] || mozillaPath="`readlink "$MozPath" | xargs dirname`"

The DBAs were getting this error when they used sudo to login as applmgr:

Adobe Reader does not need to be run as a privileged user. Please remove 'sudo' from the beginning of the command

The post on https://forums.freebsd.org/threads/17345/ recommended that we comment line 331,332,333 and 334:

# Do not allow launch using 'sudo'.
#if [ "${ACRO_ALLOW_SUDO+set}" != "set" -a \( "${SUDO_USER+set}" = "set" -o "${SUDO_UID+set}" = "set" -o "${SUDO_GID}" = "set" \) ]; then
# printf "%s\n" "Adobe Reader does not need to be run as a privileged user. Please remove 'sudo' from the beginning of the command."
# exit 1
#fi

That fixed the sudo issue.

I have asked the DBA team to reach out to the Developers to stop using acroread and migrate to an alternative that is natively supported in Linux.  Here's more from Red Hat on their security advisory https://access.redhat.com/errata/RHSA-2013:1402 :

Red Hat advises users to reconsider further use of Adobe Reader for Linux,
as it may contain known, unpatched security issues. Alternative PDF
rendering software, such as Evince and KPDF (part of the kdegraphics
package) in Red Hat Enterprise Linux 5, or Evince and Okular (part of the
kdegraphics package) in Red Hat Enterprise Linux 6, should be
considered. These packages will continue to receive security fixes.

Red Hat will no longer provide security updates to these packages and
recommends that customers not use this application on Red Hat Enterprise
Linux effective immediately.

Demantra setup.exe does nothing on Windows 7

Last evening, Mukhtiar called me to assist in Demantra installation on Windows. By the time we got on screen share, he had already downloaded patch 2195481 and installed JDK 1.8.0_77. However, when he clicked setup.exe, nothing happened.  We tried these :

1. Right click on setup.exe > Click Properties > Compatibility Tab > Run this program in compatibility mode for Windows XP
2. Run the program as Administrator
3. Tried changing registy settings as per some microsoft technotes

However nothing worked.

We located a file called setup.lax that was a configuration file for the installer and switched on stderr=console

However it showed a command window that used to disappear in a flash.

We tried uninstalling JDK 1.8.0_77 and installing JDK 1.7

Setting environment variable JAVA_HOME as described Demantra installation guide on https://docs.oracle.com/cd/E26401_01/doc.122/e22516/T573380T574948.htm

JAVA_HOME System Environment Variable

If the client machine does not have JDK version 1.6 or 1.7 installed, the Installer prompts you to download and install it. After installation, set JAVA_HOME to the JDK installation directory (for example, C:\Program Files\Java\jdk1.6.0_16).

Nothing worked.

Mukhtiar located a blog that advised setting java heap size to 512m by setting this environment variable:

_JAVA_OPTIONS = -Xmx512M

The Java Virtual Machine takes two command line arguments which set the initial and maximum heap sizes: -Xms and -Xmx. You can add a system environment variable named _JAVA_OPTIONS, and set the heap size values there

As soon as this environment variable was set, setup.exe finally launched.  All this time it was erroring out as the default 256MB was insufficient for starting the installer.

There was information about Out of Memory error on the installation guide too, but we had missed that:

Out of Memory Error Message

Verify memory settings:

1. From the Windows Start menu, choose Settings, Control Panel, and then Java. The Java Control Panel dialog box appears.
2. Click the Java Tab.
3. From the Java Applet Runtime Settings section, click View.
4. Verify that the value of the field Java Runtime Parameters is at least -Xmx256M and is based on the client machine hardware/RAM capacity. Oracle recommends value -Xmx512M for heavy-duty client side use.

Presenting in Collaborate 2016

I am presenting in OAUG Collaborate 2016 about our experiences with running E-Business Suite on Engineered Systems - Exadata, Exalogic, ZFS and PCA.  Session Details:

https://collaborate.zerista.com/event/member/223628

Please attend, if you are coming to OAUG Collaborate.

Extranet login redirects to intranet URL

For an old 11.5.10.2 ERP, we are moving from the architecture of "EBS application server in DMZ" to the architecture of "Reverse Proxy in DMZ and EBS application server in intranet".  After doing all configurations, we hit the classic issue where, you login through extranet url visible on public internet which redirects to intranet url.

So https://extranet.example.com asks for SSO details and after keying in SSO username and password goes to http://intranet.example.com.

The support.oracle.com article DMZ Configuration with Oracle E-Business Suite 11i (Doc ID 287176.1) has listed 4 checks which could be the reason for this issue:

H6: Redirection to an Incorrect Server During Login

If you are getting redirected to an incorrect server during the login process, check the following:

• Whether the hirearchy type of the profile options mentioned in Section 5.1 is set to SERVRESP .

select PROFILE_OPTION_NAME,HIERARCHY_TYPE from fnd_profile_options where profile_option_name in 
('APPS_WEB_AGENT','APPS_SERVLET_AGENT','APPS_JSP_AGENT','APPS_FRAMEWORK_AGENT' ,'ICX_FORMS_LAUNCHER','ICX_DISCOVERER_LAUNCHER','ICX_DISCOVERER_VIEWER_LAUNCHER','HELP_WEB_AGENT','APPS_PORTAL','CZ_UIMGR_URL','ASO_CONFIGURATOR_URL','QP_PRICING_ENGINE_URL','TCF:HOST');

PROFILE_OPTION_NAME                               HIERARCHY_TYPE
----------------------------------------                               --------------------------------
APPS_FRAMEWORK_AGENT                         SERVRESP
APPS_JSP_AGENT                                         SERVRESP
APPS_PORTAL                                         SERVRESP
APPS_SERVLET_AGENT                                 SERVRESP
APPS_WEB_AGENT                                         SERVRESP
ASO_CONFIGURATOR_URL                         SERVRESP
CZ_UIMGR_URL                                         SERVRESP
HELP_WEB_AGENT                                         SERVRESP
ICX_DISCOVERER_LAUNCHER                 SERVRESP
ICX_DISCOVERER_VIEWER_LAUNCHER SERVRESP
ICX_FORMS_LAUNCHER                         SERVRESP
QP_PRICING_ENGINE_URL                         SERVRESP
TCF:HOST                                                 SERVRESP


All good on this point


• Whether the profile option values for the fnd profile options (APPS_FRAMEWORK_AGENT, APPS_WEB_AGENT, APPS_JSP_AGENT, APPS_SERVLET_AGENT) are pointing to the correct node. Replace the node_id with the node_id of the external and internal web tier. For example:

select fnd_profile.value_specific('APPS_FRAMEWORK_AGENT',null,null,null,null,) from dual;

This query returned https://extranet.example.com


• Whether the dbc file pointed to by the JVM parameter (JTFDBCFILE) in jserv.properties exists.

wrapper.bin.parameters=-DJTFDBCFILE=

This was incorrect.  It was pointing to the intranet jdbc file location.


• Whether the value of the parameter APPL_SERVER_ID set in the dbc file for the node is the same as the value of the server_id in the fnd_nodes table.

  select node_name,node_id,server_id from fnd_nodes;

  This was overwritten in the dbc file, with appl_server_id of intranet when autoconfig was done on intranet and overwritten with appl_server_id of extranet when autoconfig was done on extranet, as the DBC file location and name were same for both intranet and extranet.

I asked the DBA team to manually correct the dbc file name inside $IAS_CONFIG_HOME/Apache/Apache/Jserv/etc/jserv.properties

and create a file of that name in $FND_SECURE/$CONTEXT_NAME.dbc on the extranet node and bounce services.  Once that was done, we tested and it worked. No more redirection to intranet URL.

Then I asked them to correct the s_dbc_file_name variable in the context file of extranet node. Run autoconfig on extranet, verify the value of dbcfile in jserv.properties DJTFDBCFILE parameter, verify that the DBC file had the server_id of the extranet node.  Restart all services.

Checked again, and it worked again.

So apart from checking the values of context file variables like s_webentryhost, s_webentrydomain, s_active_port, you also need to check the value of s_dbc_file while verifying the setups for extranet configuration. This can happen in 11i , R12.1 and R12.2 also.

Calling all Apps DBAs doing 11i to R12.x upgrades

At this time of the year during holidays, the Apps DBA community is busy doing upgrades as longer downtimes are possible.  In case you are facing any issues, please feel free to write to me at my email: oracleappstechnology@gmail.com .  I will be glad to hear from you and help you.