The Workflow mailer works in this way:
A connection is made from application tier server to the mail server on port 25 (SMTP)
When the smtp server gets the mail from application server, it first checks whether the application server's IP exists in /etc/hosts of the sendmail server. In our environments, we usually configure the concurrent tier as the mail server by starting the sendmail server on it.
Unless and until you have the entries for your application web servers in /etc/hosts, sendmail will not accept the mail for relay to any destination except its own domain.
For example, if I connect to port 25 and send a mail to a different domain
telnet vip.justanexample.com 25
Connected to dbtier.justanexample.com
Escape character is '^]'.
220 physicalserver.justanexample.com ESMTP Sendmail 8.13.8+Sun/8.13.8; Sat, 15 Nov 2008 10:33:45 -0500 (EST)
MAIL FROM: email@example.com
250 2.1.0 firstname.lastname@example.org... Sender ok
RCPT TO: email@example.com
550 5.7.1 firstname.lastname@example.org... Relaying denied
However if the @ in the mail is the mail server name then relaying is ok.
RCPT TO: email@example.com
250 2.1.5 firstname.lastname@example.org... Recipient ok
The rule is
1. Mail will be accepted from all IPs as long as the server name after the @ in email address is the mail server or the hostnames present in /etc/mail/local-host-names. For our example, make sure that you have the logical and physical hostnames of all application servers and the DB cluster name in /etc/mail/local-host-names.
2. Mail will be accepted for relay from those IPs whose IP and hostnames are correctly mentioned in /etc/hosts file of the mail server in this format:
IP address servername.domainname servername
192.168.1.17 vip.justanexample.com vip
If you take care of these two things, you'll not see the 550 5.7.1 Relaying denied message in Oracle Apps 11i.