Dual-instance sendmail with amavisd-new
---------------------------------------
  Mark Martinec,  2003-05-06
  (based on initial research by Ricardo Stella)
  updated on: 2003-06-04

The most recent version of this document can be found at:
  http://www.ijs.si/software/amavisd/README.sendmail-dual

==========================================================================

The setup is very similar to the one described in README.sendmail
(by Rainer Link) in section 'Scanning incoming/outgoing and relayed mail',
except that it uses SMTP protocol over inet socket (instead of pipes
to commands) to transfer files between MTA and amavisd-new and back,
and that it uses a permanently running second sendmail instance
in 'queue only' delivery mode, instead of bringing it up every time
a new checked mail comes from amavisd.

==========================================================================

Camparing the setup described in this document with the sendmail milter
setup, as described in README.milter:

milter - reasons in favour:
- can REJECT on the original SMTP session, instead of generating a bounce
  (sending a non-delivery notification _after_ the mail has been enqueued);

- only one sendmail daemon need be running, only one config file needed,
  no additional queue area needed  (although starting with sendmail 8.12
  more than one queue area is already a norm: clientmqueue, queue groups;
  and MSP already uses a different .cf file);


dual-MTA - reasons in favour:
- Full amavisd-new functionality is available, including adding spam and
  virus information header fields, adding address extensions and removing
  certain recipients from delivery while delivering the same message to
  the rest (*_lovers); All this is not available when using amavis-milter
  helper program.

- Content scanning need not be performed at the time of mail reception.
  This allows better control on CPU-intensive content filtering: mail
  checking can be streamlined and performed at optimum throughput setting
  (number of content checker processes) so as not to overwhelm host resources,
  instead of leaving it at the mercy of the current number of incomming
  SMTP sessions, where available controls are mostly based on the system
  load. Typically the number of incomming SMTP sessions (tiny processes)
  is desired to be many times above the number of content filtering
  processes (heavy resource consumers).

- No helper programs needed, MTA communicates with amavisd-new directly
  via SMTP, saves on creating one directory and one file, and deleting it
  later (at the cost of one additional transfer);

- Receiving sendmail daemon (MTA-RX) need not run as root (using option
  RunAsUser) since it does not need to run any local delivery agents (LDA)
  or to access user .forward files. This prevents external SMTP clients
  from talking directly to a process running as root.

==========================================================================


The following setup is described in this document:

      ............................   ............................
      : sendmail instance MTA-RX :   : sendmail instance MTA-TX :
      :                          :   :                          :
 25 -----> \ (mqueue-rx)         :   : (mqueue)          / -------> forward
587 ----->  > -queue-            :   : -queue-   -------<       :
 ^    :    /     | MAIL_HUB,     :   :   |               \ -------> local
 |    :          v SMART_HOST    :   :   ^                      :  delivery
msp   ...........|................   ....|.......................
                 |                       ^ loopback interface
                 |                       | port 10025
 loopback interf.v port 10024            |
            .....|.......................|.............
            : $inet_socket_port=10024    |            :
            :                            |            :
            : $forward_method='smtp:127.0.0.1:10025'  :
            : $notify_method ='smtp:127.0.0.1:10025'  :
            :                                         :
            :    amavisd-new                          :
            ...........................................


The setup is based on the recent sendmail (8.12.9) with its set of m4
configuration macros. Because of several security problems with earlier
versions of sendmail it is advised to stick to the most recent version,
although the functionality needed for this setup has long been available.
If a particular macro or feature is not available with some older version,
it is usually possible to achieve the same by manually writing a new
'mailer' specification and/or tweaking the .cf file.


We'll prepare two sendmail daemon instances (processes), let's call them
MTA-RX (receiving, accepting) and MTA-TX (transmitting, delivering).

For convenience we keep the name of the configuration file and the
queue (spool) area at default names for one mailer instance, and choose
non-default names for the other. Let's choose the MTA-TX to keep default
names, and supply non-default names to MTA-RX explicitly. This will make
admin utilities like mailq, newaliases, hoststat and purgestat operate
on the outgoing mailer instance unless explicitly told otherwise.
It can just as well be the other way around. 

MTA-RX (receiving mailer) will be responsible for accepting mail
from the Internet on port 25, accepting local submissions on port 587
(optionally), and for local submissions via sendmail program.
It will forward all mail (both for local and for nonlocal recipients)
via SMTP protocol to 127.0.0.1 (a loopback interface) on tcp port 10024,
where amavisd daemon will be listening. 
- its queue:       /var/spool/mqueue-rx
- its config file: /etc/mail/sendmail-rx.cf, /etc/mail/submit.cf
- the source (.mc) of the configuration file: thishost-rx.mc

MTA-TX (transmitting mailer) will be responsible for accepting checked
mail and notifications from amavisd-new via SMTP (or LMTP) on the
loopback interface (127.0.0.1) at tcp port 10025, and will forward all mail
to its final destinations, either for local delivery, or delivering
outgoing mail to the Internet or to other internal mailers.
- its queue:         /var/spool/mqueue
- its config file:   /etc/mail/sendmail.cf
- the source (.mc) of the configuration file: thishost-tx.mc

Inbetween the two MTAs an amavisd daemon will accept mail via SMTP
protocol on tcp port 10024, check it, and forward checked mail and
notifications via SMTP to MTA-TX. 

If you already have an existing sendmail installation, you already
have a queue directory /var/spool/mqueue and the configuration file(s)
(.mc source and the compiled .cf file). Most of the existing settings
in your .mc file can be reused, and are to be moved to the new files
thishost-rx.mc or thishost-tx.mc, or (some of them) to both.
The settings pertaining to receiving mail should go to thishost-rx.mc,
settings pertaining to delivering mail (locally or to other mailers)
should go to thishost-tx.mc, and general settings should go to both.

The file names thishost-rx.mc and thishost-tx.mc are arbitrary, they only
serve as source (to the m4 macro processor) for producing .cf files,
which control sendmail's behaviour. Sendmail never uses .mc files directly.


MTA-TX already got its queue directory during sendmail installation.

For MTA-RX a new queue directory needs to be created where incomming
mail can be collected. Use the same ownership and protection as used
for /var/spool/mqueue, e.g:
  # mkdir /var/spool/mqueue-rx
  # chown root:wheel /var/spool/mqueue-rx
  # chmod 700 /var/spool/mqueue-rx

SECURITY NOTE:
  starting with sendmail 8.12 it is possible to start sendmail daemon
  as root and let it drop privileges (become user specified by RunAsUser)
  after binding to port 25. This is normally used by MSP, and it can
  just as well be used by MTA-RX, since it has no need to access user
  mailboxes and .forward files. To use this feature, specify user and
  group in the macro confRUN_AS_USER (file thishost-rx.mc), and set the
  ownership of mqueue-rx to this user:
  # chown smmsp:smmsp /var/spool/mqueue-rx



More complex queue setup is possible if needed, like separating
sendmail work area and coredump area from actual queues.
For details about queue groups see sendmail documentation.


Create file thishost-rx.mc:
---cut-here------------------------------
dnl To be used for MTA-RX, the first MTA instance (receiving mail)

dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.

dnl Specify here also access controls, relayable domains, anti-spam measures
dnl including milter settings if needed, mail submission settings,
dnl client authentication, resource controls like confREFUSE_LA, confDELAY_LA,
dnl maximum mail size and header size, confMIN_FREE_BLOCKS, as well as
dnl other settings needed for receiving mail. 
dnl
dnl NOTE:
dnl   confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same
dnl   setting at MTA-TX, to quench down clients when disk space is low,
dnl   and not to stop processing of the already received mail.


define(`confRUN_AS_USER',`smmsp:smmsp')dnl  Drop privileges (see SECURITY NOTE)

define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl  Non-default pid file
define(`STATUS_FILE', `/etc/mail/stat-rx')dnl    Non-default stat file
define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl   Non-default queue area
define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are reasonable

dnl Match the number of queue runners (R=) to the number of amavisd-new child
dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is way too much!
QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl

dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024
FEATURE(stickyhost)dnl  Keep envelope addr "u@local.host" when fwd to MAIL_HUB
define(`MAIL_HUB',  `esmtp:[127.0.0.1]')dnl  Forward all local mail to amavisd
define(`SMART_HOST',`esmtp:[127.0.0.1]')dnl  Forward all other mail to amavisd

define(`confDELIVERY_MODE',`q')dnl     Delivery mode: queue only (a must!)
define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl  To tcp port 10024 instead of 25
MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl  Speak LMTP (this is optional)
define(`SMTP_MAILER_MAXMSGS',`10')dnl  Max no. of msgs in a single connection
define(`confTO_DATAFINAL',`20m')dnl    20 minute timeout for content checking
DAEMON_OPTIONS(`Name=MTA-RX')dnl       Daemon name used in logged messages

dnl Disable local delivery, as all local mail will go to MAIL_HUB
undefine(`ALIAS_FILE')dnl     No aliases file, all local mail goes to MAIL_HUB
define(`confFORWARD_PATH')dnl Empty search path for .forward files

MAILER(smtp)
---end-----------------------------------



Create file thishost-tx.mc:
---cut-here------------------------------
dnl To be used for MTA-TX, the second MTA instance
dnl (delivering outgoing and local mail)

dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.

dnl Specify here also the required outgoing mail processing and
dnl local delivery settings such as mailertables, required mailers,
dnl aliases, local delivery mailer settings, deliverey mode, ...
dnl Don't use milters here - for all common purposes they belong to MTA-RX.

FEATURE(`no_default_msa')dnl  No need for another msa, MTA-RX already has one
DAEMON_OPTIONS(`Addr=127.0.0.1, Port=10025, Name=MTA-TX')dnl Listen on lo:10025
define(`confSMTP_LOGIN_MSG', `$w.tx.$m Sendmail $v/$Z; $b')dnl
define(`confTO_IDENT', `0')dnl  Disable IDENT

MAILER(smtp)
MAILER(local)
---end-----------------------------------


Now macro-expand .mc files into .cf files:
(adjust the path if needed to where your cf/m4/cf.m4 file resides)

# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-rx.mc >/etc/mail/sendmail-rx.cf
# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-tx.mc >/etc/mail/sendmail.cf

Start MTA-RX and MTA-TX daemons:
# /usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf -L sm-mta-rx -bd -qp
# /usr/sbin/sendmail                            -L sm-mta-tx -bd -q30m

Start queue runner for the MSP client queue as usual if using it:
# /usr/sbin/sendmail -Ac -L sm-msp-queue -q10m


Start amavisd-new:
# amavisd

Test if MTA-RX is listening:
# telnet localhost 25
QUIT

Test if MTA-RX is listening on MSA port 587 (a newer sendmail invention)
# telnet localhost 587
QUIT

Test if MTA-TX is listening:
# telnet localhost 10025
QUIT

Test if amavisd is listening:
# telnet localhost 10024
QUIT



For convenience some shell alises may be defined:

alias mailq-rx='mailq -C/etc/mail/sendmail-rx.cf'
alias mailq-tx='mailq'
alias sendmail-rx='/usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf'
alias sendmail-tx='/usr/sbin/sendmail'


All done!


NOTES

- In amavisd.conf follow the 'POSTFIX or EXIM V4 or dual MTA setup',
  which is also the default.

- The $final_*_destiny should not specify D_REJECT,
  the D_BOUNCE (or D_PASS or D_DISCARD) is preferred.

- To make MTA-RX content-check only some mail but not all, one may use
  mailertables instead of MAIL_HUB and SMART_HOST. For example setting
  some recipient domains to be passed to MTA-TX at 127.0.0.1:10025
  directly (e.g. via mailer 'esmtp'), while sending all the rest to
  amavisd at 127.0.0.1:10024. To be able to specify the port number,
  a new 'mailer' needs to be defined, let's call it 'amavis', with similar
  settings as the already defined 'esmtp', except with port number 10024.

- To make MTA-RX reject mail for nonexistent local users by itself
  (instead of generating a bounce later on), one may use the 'virtusertable',
  listing all known recipients, and rejecting the rest, e.g.:

    jim@example.com         %1%3
    joe@example.com         %1%3
    postmaster@example.com  %1
    @example.com            error:5.7.0:550 No such user here

  VIRTUSER_DOMAIN(`example.com')dnl
  FEATURE(`virtusertable', `hash /etc/mail/virtusertable')dnl

  You may use the righthand side of the map to specify local user
  (e.g. %1%3, without domain name) in which case MAIL_HUB will be used,
  or specify an explicit domain name that is not in the {w} class,
  in which case the SMART_HOST will get consulted.


PERFORMANCE NOTES

- Mail handling is i/o-intensive. For better performance one may place
  the two mail queue areas (/var/spool/mqueue and /var/spool/mqueue-rx),
  and the /var/amavis work directory ($TEMPBASE) on three separate disks.

- One of the important arguments for choosing the dual-MTA setup is to be
  able to keep the number of content filtering processes under control,
  and not at the mercy of current mail inflow. Don't blow this advantage
  by setting the number of amavisd-new processes and MTA-RX queue runners
  too high!

  Throughput optimum is somewhere between 3 and 10 with fast daemonized
  av-scanner (or no av scanner) (with or without SpamAssassin),
  and between 2 and 3 with most command line scanners (regardless of SA).
  If the host is low on memory and when using SpamAssassin, even 2 may be
  a lot.

  Start conservatively, e.g. at 2 or 3, and if everything works normally
  and higher throughput is needed, try a bit more. Anything above the point
  where throughput function levels off is just a waste of memory and
  gains nothing!

  The optimum may be higher if high-latency external SpamAssassn tests
  are enabled (e.g. Razor, RBL), Still, never go beyond available memory.
  For example with SpamAssassin 20..25 processes on a 256 MB machine
  is where througput begins to drop rapidly on a way to swapping tar pit.
