Dual-instance sendmail with amavisd-new
---------------------------------------
  Mark Martinec,  2003-05-06
  (based on initial research by Ricardo Stella)
  last updated on: 2004-04-16

The most recent version of this document can be found at:
  http://www.ijs.si/software/amavisd/README.sendmail-dual

==========================================================================

The setup is very similar to the one described in README.sendmail
(by Rainer Link) in section 'Scanning incoming/outgoing and relayed mail',
except that it uses SMTP protocol over inet socket (instead of pipes
to commands) to transfer files between MTA and amavisd-new and back,
and that it uses a permanently running second sendmail instance
in 'queue only' delivery mode, instead of bringing it up every time
a new checked mail comes from amavisd.

==========================================================================

Comparing the setup described in this document with the sendmail milter
setup, as described in README.milter:

milter - reasons in favour:
- can REJECT on the original SMTP session, instead of generating a bounce
  (sending a non-delivery notification _after_ the mail has been enqueued);

- only one sendmail daemon need be running, only one config file needed,
  no additional queue area needed  (although starting with sendmail 8.12
  more than one queue area is already a norm: clientmqueue, queue groups;
  and MSP already uses a different .cf file);


dual-MTA - reasons in favour:
- Full amavisd-new functionality is available, including adding spam and
  virus information header fields, adding address extensions and removing
  certain recipients from delivery while delivering the same message to
  the rest (*_lovers). Also a message can be split if different recipients
  need different header edits. All this is not available when using
  amavis-milter helper program.

- Content scanning need not be performed at the time of mail reception.
  This allows better control on CPU-intensive content filtering: mail
  checking can be streamlined and performed at optimum throughput setting
  (number of content checker processes) so as not to overwhelm host resources,
  instead of leaving it at the mercy of the current number of incoming
  SMTP sessions, where available crude controls are mostly based on system
  load. Typically the number of incoming SMTP sessions (tiny processes)
  is desired to be many times above the number of content filtering
  processes (heavy resource consumers).

- No helper programs needed, MTA communicates with amavisd-new directly
  via SMTP, saves on creating one directory and one file for each message,
  and deleting it (at the cost of one additional transfer);

- Receiving sendmail daemon (MTA-RX) need not run as root (using option
  RunAsUser) since it does not need to run any local delivery agents (LDA)
  or to access user .forward files. This avoids external SMTP clients
  talking directly to a process running as root.

==========================================================================


The following setup is described in this document:

      ............................   ............................
      : sendmail instance MTA-RX :   : sendmail instance MTA-TX :
      :                          :   :                          :
 25 -----> \ (mqueue-rx)         :   : (mqueue)          / -------> forward
587 ----->  > -queue-            :   : -queue-   ->-----+       :
 ^    :    /     | MAIL_HUB,     :   :   |               \ -------> local
 |    :          v SMART_HOST    :   :   ^                      :  delivery
msp   ...........|................   ....|.......................
                 |                       ^ loopback interface
                 v                       | port 10025
 loopback interf.| port 10024            |
            .....|.......................|.............
            : $inet_socket_port=10024    |            :
            :                            |            :
            : $forward_method='smtp:127.0.0.1:10025'  :
            : $notify_method ='smtp:127.0.0.1:10025'  :
            :                                         :
            :    amavisd-new                          :
            ...........................................


The setup is based on the recent sendmail (8.12.9 or later) with its set of
m4 configuration macros. Because of several security problems with earlier
versions of sendmail it is advised to stick to the most recent version,
although the functionality needed for this setup has long been available.
If a particular macro or feature is not available with some older version,
it is usually possible to achieve the same or similar by manually writing
a new 'mailer' specification and/or tweaking the .cf file.


We'll prepare two sendmail daemon instances (processes), let's call them
MTA-RX (receiving, accepting) and MTA-TX (transmitting, delivering).

For convenience we keep the name of the configuration file and the
queue (spool) area at default names for one mailer instance, and choose
non-default names for the other. Let's choose the MTA-TX to keep default
names, and supply non-default names to MTA-RX explicitly. This will make
admin utilities like mailq, newaliases, hoststat and purgestat operate
on the outgoing mailer instance unless explicitly told otherwise.
It can just as well be the other way around. 

MTA-RX (receiving mailer) will be responsible for accepting mail from
the Internet or from internal hosts on port 25, optionally accepting local
message submissions on tcp port 587 (rfc2476), and for message submissions
via sendmail program. It will forward all mail (both for local and for
nonlocal recipients) via SMTP protocol (or LMTP) to 127.0.0.1 (a loopback
interface) on tcp port 10024, where amavisd daemon will be listening. 
- its queue:       /var/spool/mqueue-rx
- its config file: /etc/mail/sendmail-rx.cf, /etc/mail/submit.cf
- the source (.mc) of the configuration file: thishost-rx.mc
  (where 'thishost' is often by convention the name of the system (uname))

MTA-TX (transmitting mailer) will be responsible for accepting checked
mail and notifications from amavisd-new via SMTP on the loopback interface
(127.0.0.1) at tcp port 10025, and will forward all mail to its final
destinations, either for local delivery, or delivering outgoing mail
to the Internet or to other internal mailers.
- its queue:         /var/spool/mqueue
- its config file:   /etc/mail/sendmail.cf
- the source (.mc) of the configuration file: thishost-tx.mc

In-between the two MTAs an amavisd daemon will accept mail via SMTP (or LMTP)
protocol on tcp port 10024, check it, and forward checked mail and
notifications via SMTP to MTA-TX. 

If you already have an existing sendmail installation, you already
have a queue directory /var/spool/mqueue and the configuration file(s)
(.mc source and the compiled .cf file). Most of the existing settings
in your .mc file can be reused, and are to be moved to the new files
thishost-rx.mc or thishost-tx.mc, or (some of them) to both.
The settings pertaining to receiving mail, including recource limits,
should go to thishost-rx.mc; settings pertaining to delivering mail
(locally or to other mailers) should go to thishost-tx.mc, and general
settings should go to both.

The MTA-TX should have none or hardly any resource limits, or at least
have them larger that MTA-RX. Large message, common errors in mail, and
mail rush-ins should be stopped or limited at their entry to the system.
Accepting them first, but choking later can lead to trouble or at least
to wasted resources.

The file names thishost-rx.mc and thishost-tx.mc are arbitrary, they only
serve as source (to the m4 macro processor) for producing .cf files,
which control sendmail's behaviour. Sendmail never uses .mc files directly.


MTA-TX already got its queue directory during sendmail installation.

For MTA-RX a new queue directory needs to be created where incoming
mail can be collected. Use the same ownership and protection as used
for /var/spool/mqueue, e.g:
  # mkdir /var/spool/mqueue-rx
  # chown root:wheel /var/spool/mqueue-rx
  # chmod 700 /var/spool/mqueue-rx

SECURITY NOTE:
  starting with sendmail 8.12 it is possible to start sendmail daemon
  as root and let it drop privileges (become user specified by RunAsUser)
  after binding to port 25. This is normally used by MSP, and it can
  just as well be used by MTA-RX, since it has no need to access user
  mailboxes and .forward files. To use this feature, specify user and
  group in the macro confRUN_AS_USER (file thishost-rx.mc), and set the
  ownership of mqueue-rx to this user and group:
  # chown smmsp:smmsp /var/spool/mqueue-rx
  # chmod 770 /var/spool/mqueue-rx


More complex queue setup is possible if needed, like separating
sendmail work area and core dump area from actual queues.
For details about queue groups see sendmail documentation.


Create file thishost-rx.mc:
---cut-here------------------------------
dnl To be used for MTA-RX, the first MTA instance (receiving mail)

dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.

dnl Specify here also access controls, relayable domains, anti-spam measures
dnl including milter settings if needed, mail submission settings, client
dnl authentication, resource controls, maximum mail size and header size,
dnl confMIN_FREE_BLOCKS, and other settings needed for receiving mail. 
dnl
dnl NOTE:
dnl   confMIN_FREE_BLOCKS at MTA-RX should be kept higher than the same
dnl   setting at MTA-TX, to quench down clients when disk space is low,
dnl   and not to stop processing the already received mail.
dnl
dnl In particular, here are some settings to be considered:
dnl   ( see also http://www.sendmail.org/m4/anti_spam.html )
dnl
dnl FEATURE(`access_db',`hash -T<TMPF> /etc/mail/access.db')
dnl VIRTUSER_DOMAIN(`sub1.example.com')dnl  list valid users here
dnl VIRTUSER_DOMAIN(`sub2.example.com')dnl  list valid users here
dnl FEATURE(`virtusertable', `hash /etc/mail/virtusertable')
dnl define(`confUSERDB_SPEC', `/etc/mail/userdb.db')
dnl FEATURE(`blacklist_recipients')
dnl INPUT_MAIL_FILTER(...)
dnl define(`confPRIVACY_FLAGS', `noexpn,novrfy,authwarnings')  nobodyreturn ?
dnl define(`confDONT_PROBE_INTERFACES')
dnl undefine(`USE_CW_FILE')dnl  cancel use_cw_file feature, no class {w} extras
dnl MASQUERADE_AS(...) FEATURE(`allmasquerade') FEATURE(`masquerade_envelope')
dnl define(`confTO_IDENT', `0')dnl  Disable IDENT
dnl define(`confMAX_MESSAGE_SIZE',`10485760')
dnl define(`confMAX_MIME_HEADER_LENGTH', `256/128')
dnl define(`confNO_RCPT_ACTION', `add-to-undisclosed')
dnl FEATURE(`nocanonify', ...)
dnl define(`confBIND_OPTS', ...)
dnl define(`confTO_RESOLVER_*... )
dnl define(`confDELAY_LA,    8)
dnl define(`confREFUSE_LA', 12)
dnl define(`confMAX_DAEMON_CHILDREN',20)
dnl define(`confMIN_FREE_BLOCKS', `10000')
dnl define(`confDEF_USER_ID', ...)

define(`confRUN_AS_USER',`smmsp:smmsp')dnl  Drop privileges (see SECURITY NOTE)

define(`confPID_FILE', `/var/run/sendmail-rx.pid')dnl  Non-default pid file
define(`STATUS_FILE', `/etc/mail/stat-rx')dnl    Non-default stat file
define(`QUEUE_DIR', `/var/spool/mqueue-rx')dnl   Non-default queue area
define(`confQUEUE_SORT_ORDER',`Modification')dnl Modif or Random are reasonable

dnl Match the number of queue runners (R=) to the number of amavisd-new child
dnl processes ($max_servers). 2 to 7 OK, 10 is plenty, 20 is too many
QUEUE_GROUP(`mqueue', `P=/var/spool/mqueue-rx, R=2, F=f')dnl

dnl Direct all mail to be forwarded to amavisd-new at 127.0.0.1:10024
FEATURE(stickyhost)dnl  Keep envelope addr "u@local.host" when fwd to MAIL_HUB
define(`MAIL_HUB',  `esmtp:[127.0.0.1]')dnl  Forward all local mail to amavisd
define(`SMART_HOST',`esmtp:[127.0.0.1]')dnl  Forward all other mail to amavisd

define(`confDELIVERY_MODE',`q')dnl     Delivery mode: queue only (a must,
dnl  ... otherwise the advantage of this setup of being able to specify
dnl  ... the number of queue runners is lost)
define(`ESMTP_MAILER_ARGS',`TCP $h 10024')dnl  To tcp port 10024 instead of 25
MODIFY_MAILER_FLAGS(`ESMTP', `+z')dnl  Speak LMTP (this is optional)
define(`SMTP_MAILER_MAXMSGS',`10')dnl  Max no. of msgs in a single connection
define(`confTO_DATAFINAL',`20m')dnl    20 minute timeout for content checking
DAEMON_OPTIONS(`Name=MTA-RX')dnl       Daemon name used in logged messages

dnl Disable local delivery, as all local mail will go to MAIL_HUB
undefine(`ALIAS_FILE')dnl     No aliases file, all local mail goes to MAIL_HUB
define(`confFORWARD_PATH')dnl Empty search path for .forward files
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
undefine(`DECNET_RELAY')dnl

MAILER(smtp)
---end-----------------------------------



Create file thishost-tx.mc:
---cut-here------------------------------
dnl To be used for MTA-TX, the second MTA instance
dnl (delivering outgoing and local mail)

dnl Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.

dnl Specify here also the required outgoing mail processing and
dnl local delivery settings such as mailertables, needed mailers, aliases,
dnl local delivery mailer settings, smrsh, delivery mode, queue groups, ...
dnl Don't use milters here - for all common purposes they belong to MTA-RX.

define(`confREFUSE_LA',999)dnl  Disable the feature, limiting belongs to MTA-RX
define(`confMAX_DAEMON_CHILDREN',0)dnl  Disable, limiting belongs to MTA-RX
FEATURE(`no_default_msa')dnl  No need for another MSA, MTA-RX already has one
DAEMON_OPTIONS(`Addr=127.0.0.1, Port=10025, Name=MTA-TX')dnl Listen on lo:10025
define(`confSMTP_LOGIN_MSG', `$w.tx.$m Sendmail $v/$Z; $b')dnl
define(`confTO_IDENT', `0')dnl  Disable IDENT

MAILER(smtp)
MAILER(local)
---end-----------------------------------


Now macro-expand .mc files into .cf files:
(adjust the path if needed to where your cf/m4/cf.m4 file resides)

# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-rx.mc >/etc/mail/sendmail-rx.cf
# m4 /usr/share/sendmail/cf/m4/cf.m4 thishost-tx.mc >/etc/mail/sendmail.cf

Start MTA-RX and MTA-TX daemons:
# /usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf -L sm-mta-rx -bd -qp
# /usr/sbin/sendmail                            -L sm-mta-tx -bd -q15m

Start queue runner for the MSP client queue as usual, if using it:
# /usr/sbin/sendmail -Ac -L sm-msp-queue -q10m


Start amavisd-new:
# amavisd

Test if MTA-RX is listening:
# telnet localhost 25
QUIT

Test if MTA-RX is listening on MSA port 587 (a newer sendmail invention)
# telnet localhost 587
QUIT

Test if MTA-TX is listening:
# telnet localhost 10025
QUIT

Test if amavisd is listening:
# telnet localhost 10024
QUIT



For convenience some shell aliases may be defined:

alias mailq-rx='mailq -C/etc/mail/sendmail-rx.cf'
alias mailq-tx='mailq'
alias sendmail-rx='/usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf'
alias sendmail-tx='/usr/sbin/sendmail'


All done!


NOTES

- In amavisd.conf file follow the 'POSTFIX or EXIM V4 or dual MTA setup',
  which is also the default.

- The $final_*_destiny should not specify D_REJECT.
  The D_BOUNCE (or D_PASS or D_DISCARD) is preferred.

- To make MTA-RX content-check only some mail but not all, one may use
  mailertables instead of MAIL_HUB and SMART_HOST. For example setting
  some recipient domains to be passed to MTA-TX at 127.0.0.1:10025
  directly (e.g. via mailer 'esmtp'), while sending all the rest to
  amavisd at 127.0.0.1:10024. To be able to specify the port number,
  a new 'mailer' needs to be defined, let's call it 'amavis', with similar
  settings as the already defined 'esmtp', except with port number 10024.

- depending on how local addresses are translated by MTA-RX, the
  @local_domains list (in amavisd.conf) needs to be adjusted accordingly.
  Check the amavisd-new log what recipient addresses it sees for local
  recipients. The '[127.0.0.1]' may need to be added to the @local_domains.

- To make MTA-RX reject mail for nonexistent local users by itself
  (instead of generating a bounce later on), one may use the 'virtusertable'
  in thishost-rx.mc, listing all known recipients, and rejecting the rest,
  e.g.:

  VIRTUSER_DOMAIN(`example.com')dnl
  FEATURE(`virtusertable', `hash /etc/mail/virtusertable')dnl

    jim@example.com         %1%3
    joe@example.com         %1%3
    postmaster@example.com  %1
    @example.com            error:5.7.0:550 No such user here

  You may use the righthand side of the map to specify local user
  (e.g. %1%3, or just jim, without domain name) in which case MAIL_HUB will
  be used, or specify an explicit domain name that is not in the {w} class,
  in which case the SMART_HOST will get consulted.

  Perhaps what Stephane Lentz writes is even better:

    Dictionary attacks and messages to retired accounts can be bounced with
    sendmail: just replicate your aliases or write some SLocal_check_rcpt
    rule-set that checks addresses of your domain against a map of valid users
    (valid_addresses.db).  I hope some standard FEATURE will be provided
    with sendmail - something like FEATURE(checkdomainaddresses) and
    CHECKDOMAINADDRESSES(mydomain.com). 


PERFORMANCE NOTES

- Mail handling is I/O-intensive. For better performance one may place
  the two mail queue areas (/var/spool/mqueue and /var/spool/mqueue-rx),
  and the /var/amavis work directory ($TEMPBASE) on three separate disks.
  The /var/amavis/tmp may be a tmpfs or a RAMdisk or delayed-sync fs.

- One of the important arguments for choosing the dual-MTA setup is to be
  able to keep the number of content filtering processes under control,
  and not at the mercy of current mail inflow. Don't blow this advantage
  by setting the number of amavisd-new processes and MTA-RX queue runners
  too high!

  Throughput optimum is somewhere between 3 and 10 with fast daemonized
  av-scanner (or no av scanner) (with or without SpamAssassin),
  and between 2 and 3 with many command line scanners (regardless of SA).
  If the host is low on memory and when spam checking (SpamAssassin)
  is used, even 2 may be a lot.

  Start conservatively, e.g. at 2 or 3, and if everything works normally
  and higher throughput is needed, try a bit more. Anything above the point
  where throughput function levels off is just a waste of memory and
  gains nothing!

  The optimum may be higher if high-latency external SpamAssassin tests
  are enabled (e.g. Razor, RBL), Still, never go beyond available memory.
  For example with SpamAssassin enabled, the 20..25 processes on a 256 MB
  machine is where throughput begins to drop rapidly on a way to a swapping
  tar pit.
