                                                           November 2, 2004
amavisd-new-2.2.0 release notes


SECURITY:

- modified MIME entity traversal to include MIME container parts
  (e.g. multipart/*) as pseudo parts. Such pseudo-parts do not carry
  any body contents but do show up (with their MIME content type only)
  in the tree structure as seen by banned names checks. This makes it
  possible to specify more complex banned rules based on the placement
  of leaf nodes within or outside of mime multipart containers.

  This also re-enables the possibility to check such MIME wrappers for
  banned MIME Content-Types (most notably for the message/partial and
  message/external-body), which was lost with the change of internal
  representation of mail parts in version 2.0 (amavisd-new-20040701);

- preserve original zip archive for virus scanners to see, if the archive
  contains any zero-length members (Archive::Zip module chooses not to
  extract members with declared zero size, even if the size does not
  correspond to the actual size); based on a patch by Dirk Datzert;

- add tests to mime_traverse to verify that the files produced by MIME::Parser
  really do exist and are readable; (and sent a patch adding I/O checks
  for MIME::Tools to its maintainer; please use MIME::Tools 5.414 or later
  from CPAN to avoid possibility of full /tmp partition to cause infected
  or spam mail to pass through);

- changed recommendation in INSTALL to choose the latest version of
  MIME-Tools from CPAN - the 5.415 at the time of this writing;

- do_unrar: recognize encrypted entire archives (not just their individual
  members), and flag mail as undecipherable;

- recognize file(1) report/^MS Windows\b.*\bDLL\b/ as short type 'dll';
  add 'dll' to example patterns in amavisd.conf and amavisd.conf-sample;
  add name extension '.cpl' to the list of basic banned names;


INCOMPATIBILITY:

- incompatible change: the default value for $recipient_delimiter is
  now undef and no longer '+'; adding address extensions must now be
  explicitly enabled;

- minor change in the default X-Virus-Scanned: header field, see below;


OTHER CHANGES SINCE 2.1.2:

- rewritten LDAP modules, by Michael Hall;

- improvements in handling of double errors (errors that occur while
  handling consequences of some previous error); be permissive on failures
  in DESTROY methods; prevent some cases for child process not being able
  to sign off from the nanny database;

- enforce $max_requests more strictly, dropping SMTP session after task
  count is exceeded by one; this is in response to the new smtp session
  caching behaviour in Postfix, which is now much more persistent in keeping
  the session open on a busy mailer; although dropping session at the server
  side is discouraged by RFC 2821, this change was recommended on the
  postfix-users mailing list;

- added a site-wide mail size limit $smtpd_message_size_limit, and a
  per-recipient mail size limit lookups @message_size_limit_maps along with
  SQL and LDAP fields, making it possible to reject mail based on its size.
  The list of lookup tables maps recipients to mail size in bytes, e.g.:

    @message_size_limit_maps = ({'user1@example.com' => 20_000_000,
                                 'user2@example.com' => 15*1024*1024,
                                 'user3@example.com' => 0, # uses global limit
                                 '.example.com'      => 10*1024*1024 });
    $smtpd_message_size_limit = 20*1024*1024;  # global limit if nonzero

  A value of 0 or undef disables the check and is a default. A per-recipient
  limit is bound on the high side by the $smtpd_message_size_limit, and on the
  low side by 64kB, which is a minimal allowed size limit imposed by RFC 2821.

  This limiting really belongs to MTA and is only partially supported here
  (no admin notifications, no quarantine, no final_*_destiny configurability).
  It is mostly provided here to be able to place some sanity limit on runaway
  or malicious clients, or if someone insists on using amavisd-new in a
  pre-queue filtering setup; suggested by Tuomo Soini;

- internal: add new object $msginfo->mail_text_fn to hold the file name
  of the original mail, decouple the location of file email.txt from the
  temporary directory, which was implied until now. This is presently needed
  for the Courier interface. Add optional attribute 'mail_file' to the
  AM.PDP protocol, see README.protocol;

- in banned parts descriptor strings which are used in banned name checking,
  provide a 'T=empty' short type for empty mail parts, including for empty
  MIME parts (instead of omitting T=... altogether). This can be used in
  banned rules to test for empty parts, generally or restricted to empty
  MIME parts; suggested by Ricardo Stella and Stephane Lentz;

- a banned lookup result (which is interpreted as boolean for most purposes:
  zero or empty for false, anything else for true) may give a result 'DISCARD'
  (which is true as well), which will disable DSN if it turns out the mail
  was blocked by such banned rule (akin to viruses_that_fake_sender and
  spam_dsn_cutoff_level). Here is an example rule in $banned_namepath_re:
    [ qr'(?# BLOCK EMPTY MIME PART APPLICATION/OCTET-STREAM )
         ^ (.*\t)? M=application/octet-stream \t(.*\t)* T=empty (\t.*)? $'xmi
      => 'DISCARD' ],
  Suggested by Ricardo Stella and Stephane Lentz;

- fix 'Insecure dependency in unlink' in sub files_to_scan which could
  happen when some decoder left non-regular files in the directory;
  reported by Andrzej Kukula;

- bug fix: only insert LDAP and SQL lookups objects into lists of maps at the
  first task of a child process, later tasks should not insert duplicates;

- new subroutine do_ar and new configuration variable $ar to handle
  standard Unix archives, including Debian binary packages;

- recognize a Unix archive and give it a short type .a;
- recognize a Unix relocatable binary and give it a short type .o;

- do not penalize SMTP status "554 5.1.1 Error (DATA): no valid recipients",
  the situation arises regularly when pipelining is in effect and all
  recipients happen to be rejected;

- protect spaces and newlines when logging broken Message-ID and
  Resent-Message-ID header fields (macros %m and %r) to facilitate log
  parsing; protect newlines in logged Subject header field (macro %j);
  parsing difficulty pointed out by Chris Lee;

  The present solution uses =XX (hex) encoding and is a quick and dirty
  fix. A cleaner solution to avoid double sanitation of special characters
  is needed, but would involve a deeper reorganization;

- updated example list of "banned extensions - long" in amavisd.conf
  and amavisd.conf-sample;

- change the default lock file name from "$MYHOME/amavisd-$$.lock" to
  "$MYHOME/amavisd.lock", to avoid inventing a new name at every restart
  and leaving old files behind; pointed out by Dale Walsh and Martin Orr;

- updated av entry for nvcc (Norman Virus Control v5 / Linux) to include
  statuses 10,11, and 2,14 to the status lists according to documentation;
  password protected or corrupted archive (status 11) was not recognized
  as non-infected status; thanks to Michael Ramke of Norman Data Defense
  Systems GmbH;

- updated DrWebD entry to recognize and ignore flag DERR_SKIPPED;

- support Mail::ClamAV 0.12 and 0.13 or later, which is incompatible with
  0.11 due to change of constant names in the underlying ClamAV library;

- added 'check-jpeg' example entry to the @av_scanners list and provide the
  associated module JpegTester.pm; it offers a fully-fledged check for jpeg
  comment field buffer overflow attempts; should serve mainly as an example
  for adding similar quick responses to new threats;

- added 'check-jpeg-simple' example entry to the @av_scanners list (only in
  amavisd.conf-sample); it offers a quick check (and not very exact one:
  checks only the first 32kB) for jpeg comment field buffer overflow attempts;
  should serve mainly as an example for adding similar quick responses
  to new threats;

- relax too restrictive sanity check on temporary directory name
  when accepting requests from a helper program or via AM.PDP protocol
  (e.g. with sendmail milter setup); reported by Babu Kanagala;

- relax allowed set of characters when receiving XFORWARD attribute values,
  it turns out that characters like '=' and '+' are allowed;

- when using "bsmtp:" delivery method suppress X-Envelope-From and
  X-Envelope-To header fields, as the addresses are already available
  in the envelope;

- when using the "bsmtp:" quarantine method the *_quarantine_to was
  completely ignored, which made it impossible to turn off quarantining
  selectively for certain users by specifying an empty or undef value.
  Since 2.2.0 an empty *_quarantine_to turns off quarantine for a recipient
  regardless of the quarantine method. A nonempty string in *_quarantine_to
  (the exact value is still ignored) must now be used even with "bsmtp:"
  to enable quarantining. Inconsistency discovered by Sean Doherty;

- suppress leading $QUARANTINEDIR string from the value of macro %q,
  thus hiding the absolute file path from notifications;

- add configuration variable $local_client_bind_address (and equivalent
  policy bank key), to allow for explicitly binding local socket address
  to a specific interface in SMTP client; suggested by Wouter de Jong;

- keep whatever (if any) file results from gunzip and family (do_uncompress)
  even if the decompressor's exit status is nonzero; reason: gzip returns
  status 2 when decompressing file with trailing garbage; reported by
  Tobias Reckhard;

- collect declared original file name from gzipped (and friends) files
  if reported by file(1), making them available to banned name checking;

- avoid unnecessarily checking white/black lists if spam checks will not
  be used (e.g. infected mail);

- use qquote_rfc2821_local to properly quote e-mail addresses in the
  most visible log entries;

- if there is more than a single (or less then one) occurrence of %k in the
  SQL SELECT clause template, multiplicate the set of query keys accordingly,
  making possible more complicated custom SELECT clauses;

- don't forget to load amavis policy delegation protocol support code
  if AM.PDP is explicitly requested in $protocol, even if not listening
  on a Unix socket; 

- add 'queue_id' attribute to the AM.PDP protocol; equivalent to a change
  in the Courier support code by Martin Orr;

- include the declared (faked) sender address in the virus recipient
  notification template, in addition to the originator IP address;

- add macro %Q and method Amavis::In::Message::queue_id, holding a
  MTA queue ID if available (in Courier and milter/AM.PDP setup);
  by Michael Musikhin (through Martin Orr);

- add macro %y to show elapsed processing time; suggested by Ed Walker;

- sanitize newlines and spaces (and some other characters) when moving
  syntactically invalid Message-ID and Resent-Message-ID to macros %m and %r
  for logging purposes; resulting wrapped main log entry reported by CRivera;

- bring up syslog priority to LOG_NOTICE when debug or debug_oneshot
  is in effect;

- make a product name, version ID and version date available as separate
  variables to avoid the need to parse $myversion for the purpose of
  customizing e.g. the setting of $X_HEADER_LINE;  based on suggestion
  by Dale Walsh; the re-introduction of a date ($myversion_date) also
  suggested by Stephane Lentz.

  Added variables: $myproduct_name, $myversion_id, $myversion_id_numeric,
  and $myversion_date, which serve to assemble the $myversion. Modified the
  default templates of $smtpd_greeting_banner and $smtpd_quit_banner to
  take advantage of the new variables. Changed $X_HEADER_LINE default
  from "by amavisd-new at $mydomain" to "$myproduct_name at $mydomain" and
  added an example of a $X_HEADER_LINE setting with version number included
  to the amavisd.conf-sample;

- added SQL fields 'virus_admin' and 'spam_admin' to lookup lists
  @virus_admin_maps and @spam_admin_maps; patch by Robert Collier;

- add a log message 'SPAM-KILL, ...' (at log level 3) for not-passed mail,
  to complement the 'SPAM-TAG' log message for passed mail;

- add Mail::SpamAssassin::Plugin::SPF to a list of modules that SA fails
  to load at init time;

- prevent sending the same SMTP response more than once, if the first
  attempt failed due to disconnected SMTP session;

- fix a double-@ formatting buglet in the log message
  "adding address extension _spam to user@@domain", reported by Vincenzo;

- add kill('TERM',$$) to the arsenal of attempts to get rid of unwanted
  forked process;

- includes rpm spec file, including the init script, contributed by
  Marius Andreiana, based on previous work by Dag Wieers;

- document the localization template directory contents
  (in file amavisd.conf-sample) when read_l10n_templates is used;
  thanks to Joël Brogniart;

- includes file Macintosh.tar.gz, which contains auto-startup scripts
  and installation instructions for Mac OS X, contributed by Dale Walsh
  of the Dale Enterprise L.L.C.


---------------------------------------------------------------------------
                                                          September 6, 2004
amavisd-new-2.1.2 release notes

- fixed (hard)black- and white-listing on static lookup tables
  which failed to match any sender; reported by Derck Floor;

- use $hdrfrom_notify_recip address in the From: field for recipient
  notifications, instead of $hdrfrom_notify_admin; inconsistency
  pointed out by Ekkehard Burkon;

- the 'neutral' sender notification template was joining the Subject
  and the Message-ID header fields into one longer Subject when it was
  reporting some nondelivery other than the 'invalid characters in header'.
  Likewise the first body line of this same DSN was eaten up:
    "This nondelivery report was generated by the amavisd-new program"
  (the problem was introduced in amavisd-new-20030616 and never reported);

- in amavisd-agent, amavisd-nanny, amavisd: extend the signal and error
  handling in code sections holding bdb locks from just ignoring the SIGINT,
  to controlled catching and re-signaling several signals and error conditions;
  problem reported by Tom Mulder;

- suppress duplicate names from the list of virus names in macro %V;
  by Gregor Weiss;

- fine-adjusted log level of some log messages;

- discard leading and trailing whitespace from the macro %t (Received trace);

- extend the search for IP in the Received trace from 4 to a maximum of
  6 entries;

- ignore private IPv6 addresses (RFC 3513: link-local, site-local, multicast)
  when searching through Received trace for the origin of mail;

- place mail header field X-Envelope-From in front of the X-Envelope-To in
  quarantined mail; also changed case of X-Quarantine-id into X-Quarantine-Id
  for consistency with other header fields;

- provide new macro %e which evaluates to our best guess of the originator
  IP address collected from the Received trace, complementing similar macros
  %t, %a and %g; suggested by Gregor Weiss;

- add the result of macro %e to the default 0-level log entry;

- provide new macros %u and %U to evaluate to a timestamp of the message
  reception similar to an existing macro %d (RFC 2822 local date-time);
  the (%u) as Unix time (seconds since 1970-01-01T00:00Z as a decimal integer,
  suggested by Gregor Weiss), and (%U) as ISO 8601 (EN 28601) UTC date-time;

- avoid some empty lines in default DSN templates and fix some inconsistencies
  in their formatting;

- internal: collect existing common code for time formatting as new
  subroutines iso8601_timestamp and iso8601_utc_timestamp; collect existing
  common code to find IP address in the Received trace as a new subroutine
  best_try_originator_ip;

- bump up the version number in $myversion - the 2.1.1 still presented
  itself as 2.1.0;

- add a note about a data structure difference between @score_sender_maps and
  $per_recip_blacklist_sender_lookup_tables (amavisd.conf-sample, amavisd)


---------------------------------------------------------------------------
                                                            August 24, 2004
amavisd-new-2.1.1 (amavisd-new-20040824) release notes

- unconditionally initialize @banned_filename_maps to (\$banned_filename_re),
  otherwise $banned_filename_re is ignored by default (unless amavisd.conf
  explicitly assigns to @banned_filename_maps); a patch by Thomas Jarosch;

- fixed inappropriate log entry in SQL whitelisting:
     wbl: (SQL) recip <xxxx@yyyy.com> whitelisted sender <...>,
     unexpected wb field value: "1";
  reported by Carlos Horowicz;

- added missing import of &ca to the amavisd-new-courier.patch; by Martin Orr;

- produce a warning when there is an unknown field in the policy bank
  to be loaded;

- with delivery method 'bsmtp:' prepend a directory $QUARANTINEDIR to the
  file path if not explicitly specified, to behave like the 'local:' delivery
  method, making it possible to hide full path from the X-Quarantine-Id
  and notifications; a patch by Thomas Jarosch;

- pre-load SA 3.0.0 module Mail::SpamAssassin::Plugin::Hashcash
  to make it available in the chroot jail;

- pre-load modules Mail::SpamAssassin::SpamCopURI and URI::*
  for SA older than 3.0.0;

- enhancement to amavisd-nanny: when terminating a process and SIGTERM
  produces no result for some time, try SIGKILL; contributed by Philip Engdahl;


---------------------------------------------------------------------------
                                                            August 15, 2004
amavisd-new-2.1.0 (amavisd-new-20040815) release notes

The 2.1.0 release is mostly a maintenance release over 2.0, with
only a handful of smaller features added. Based on a manual code audit
the number of smaller internal code changes is rather extensive,
some changes dealing with long-standing known deficiencies, minor bugs,
documentation problems and typos. Only a few fixes are for new bugs
introduced in 2.0.

The files amavisd.conf, amavisd.conf-default and amavisd.conf-sample
have been extensively reworked, with the hope to suit better the
new installations, while possibly causing some head-scratching for
existing installations when looking at a diff. The file amavisd.conf
is the one that should serve as a sound base for the initial config file,
while keeping an eye on the list of all variables and their defaults
in amavisd.conf-default. The amavisd.conf-sample is being phased out
of active maintenance, and should serve mostly as a set of examples and
the source of documentation until better documentation is available.

Two nice features are available:

- the use of BerkeleyDB and libdb is now optional;
  controlled by variables $enable_db and $enable_global_cache;

- a new program 'amavisd-nanny', with the accompanying instrumentation
  in amavisd, displays the general health of all amavisd child processes,
  reports crashed ones and attempts to kill long overdue processes;
  It is still experimental and minimalistic, problem reporting is currently
  only to stdout.
 

Other changes:

- incompatible change since 2.0: the use of BerkeleyDB is now off by default;

  The use of BerkeleyDB and libdb is made optional, controlled by
  variables $enable_db and $enable_global_cache, both false by default.

  $enable_db:
    enables the use of BerkeleyDB/libdb
    (for SNMP counters database and nanny, and optionally for cache);

  $enable_global_cache:
    enables the use of libdb-based cache when $enable_db is also true;

    If either the $enable_global_cache or $enable_db are false,
    cache of mail body MD5 digests is kept in child-local memory
    as in pre-2.0 versions, and is therefore local and short-lived,
    with lower expected cache hit rate;

- incompatible change: DSPAM 3.0 or better is needed (if $dspam is enabled),
  no longer works with 2.x;

- incompatible change: changed name of the (hardly ever needed) configuration
  variable auth_supported_out to auth_required_out, to better reflect its
  semantics (should be true if MTA server to which amavisd is sending
  notifications and forwarding mail requires authentication (AUTH smtp
  command));

- a new small program included: amavisd-nanny is a program to show the status
  and keep an eye on the health of amavisd child processes (experimental);

- fixed a bug in lookup_acl where a "." did not act as a catchall;
  thanks to JP;

- fixed a problem in SQL lookup which could return undef even when
  not all the matched records had NULL in the field;

- fixed compatibility with old 'amavis' helper program
  ('delivery_care_of' defaulted to 'client', instead of depending on
  the presence of ldaargs), reported by Charlie Schluting and Christer;

- fixed long standing problem in do_ascii, which could return without calling
  Convert::UUlib::CleanUp, occasionally spilling state into subsequent
  mail checks within the same process;

- fixed macros %D, %O and %N when log_recip_templ is being expanded;
  a patch by Ed Walker;

- fixed recognition of separators in a nested call during macro expansion;

- pre-load missing modules Net::LDAP, Net::LDAP::Schema, Net::LDAP::Search,
  and Net::LDAP::RootDSE; suggested by Paul Jacobson;

- fix locale-related bug in rfc2822 date generation, where we were restoring
  the saved LC_TIME value to LC_CTYPE (!);
  a patch by Henrique de Moraes Holschuh / Debian support team;
  
- protect from signals while bdb cursor holds a lock;

- new subroutine inherit_header_edits() and slight code restructuring
  makes possible for spam_scan() and other code before the final delivery
  to start submitting common header edits into $msginfo->header_edits,
  avoiding the need for passing them through global variables;

- now a loglevel-based automatic syslog priority assignment can no longer
  lower a message syslog priority below the syslog priority specified
  in the $SYSLOG_LEVEL, it can only increase it; the violation of the
  least-surprise principle pointed out by Andy Dills;

- a small optimization in logging: a new subroutine ll allows to save time
  in preparing complex log entries when we know their log level exceeds
  the current log level and won't be logged;

- in default macro templates $log_templ and $log_recip_templ:
  * placed 'spam' condition before 'bad header' for consistency
    with program behavior;
  * added reporting of tag/tag2/kill levels in $log_recip_templ
    (experimental: macro names may change in future versions);
  * rewritten templates using negation (i.e. [?x|1] ) to avoid unsightly
    selector nesting;

- MIME decoding is now allowed to exceed the decoding quota, avoiding
  the problem when a small quota settings might not allow even a plain
  mail through;

- override LC_TIME to "C" on every log message, to work around issues
  with Unix::Syslog, which would log stuff with the date stamp localized,
  which syslog would dislike and add its own, and the resulting mess
  is not recognized by amavis log processors;
  a patch by Henrique de Moraes Holschuh / Debian support;

- changed dspam command line options to work with dspam 3.0 (no longer
  with 2.x), a patch by Ron Ohmer, Nabil Sefrioui, and Reech;

- dspam header fields are now inserted into passed mail if all recipients
  are local;

- supply better defaults for $hdrfrom_notify_sender, $hdrfrom_notify_recip,
  $hdrfrom_notify_admin and $hdrfrom_notify_spamadmin, similar to defaults
  from amavisd-new-20030616-p10;

- when parsing output from the 'AVG grisoft' virus scanner,
  don't include CR in virus name; reported by Vernon A. Fort;

- new file 'amavisd-new-qmqpqq.patch' provided by Martin Solciansky,
  (similar to fixes by Christopher Odenbach) making it work again
  with the current version of amavisd-new;

- use lstat instead of stat, and test for soft links wherever appropriate;

- remember inode and device number when creating temporary directory
  and temporary file, and test for change before removing them;

- enhanced security: certain tainted values are allowed to enter deeper
  into program, untainting them only where and when really necessary;

- avoid a taint problem in Mail::ClamAV;

- added AV entry for CAI eTrust Antivirus; by Stephane Lentz
  (requires a suid shell wrapper around inocmd32);

- added status 9 to the set of infected statuses for the
  drweb command line scanner (DrWeb Antivirus);

- use our subroutine q_encode to Q-encode header fields from the
  notification templates, instead of MIME::Words::encode_mimeword
  (a similar fix in 2.0 applied only to encoding of modified headers
  in passing mail);

- add attribute 'x-spam-type=original' to the Content-Type header field
  (the SpamAssassin's code to recognize an original email) when defanging
  spam, facilitating reporting spam via SA to other spam fighting tools;
  a patch by Brian May;

- add a note to amavisd.conf that $sa_auto_whitelist has no effect on SA
  since 3.0.0 - SA has now a configuration file option 'use_auto_whitelist';

- turn off timer in post_process_request_hook() to avoid periodically
  recreating child processes on an idle machine;

- added @mynetworks_maps and enhanced lookup_ip_acl() to take a list
  of lookup tables: currently members can be an array ref (as before),
  or a hash ref (new) or a plain constant (new);

- generalized @debug_sender_acl into @debug_sender_maps along the lines of
  other lookup tables, and make it part of policy banks;

- added @warnvirusrecip_maps, @warnbannedrecip_maps, @warnbadhrecip_maps;

- added @spam_subject_tag_maps and @spam_subject_tag2_maps,
  to allow per-recipient spam tags string; suggested by Ed Walker;

  Note, there is an inconsistency in names of legacy variables and the
  new @*_maps, in an attempt to rectify an unfortunate choice of name
  for seldomly used variable $sa_spam_subject_tag1:
    @spam_subject_tag_maps  = (\$sa_spam_subject_tag1);  # exotic
    @spam_subject_tag2_maps = (\$sa_spam_subject_tag);   # in common use

- added configuration variable $auth_reauthenticate_forwarded, which
  directs amavisd to apply its own credentials ($amavis_auth_user and
  $amavis_auth_pass) to unauthenticated forwarded (passed) mail, besides
  using them for submitting original messages (notifications and quarantine).
  This is similar to how mailing list managers are allowed to work (rfc2554).
  Note that the Perl module Net::SMTP in its current version is unable to
  specify the 'submitter' in its 'MAIL FROM' command, this should be rectified;

---------------------------------------------------------------------------
                                                               July 1, 2004
amavisd-new-20040701 / amavisd-new-2.0 release notes


MAJOR NEW FEATURES (since amavisd-new-20030616-p10):

- security improvements: no shell required in chroot jail, checks performed
  to see if dropping privileges was successful, can drop privileges before
  reading config file;
- the 'amavisd reload' command reimplemented, it now works even
  if running chroot-ed;
- new feature: policy banks hold sets of configuration variables that may
  be switched with another predefined set based on incoming port number or
  original SMTP client IP, avoiding the need to run more than one instance
  of amavisd daemon;
- new feature: @score_sender_maps is a soft variant of black- and whitelisting;
- extended semantics of SQL field wblist.wb for soft black/white-listing;
- redesigned mail structure representation allows better control
  over 'banned' names and types;
- MIME defanging can wrap the entire original mail in a MIME container;
- more flexible control on lookups: configurable list of lookup tables
  observes the specified order of tables and permits arbitrary number
  of tables of any available type;
- level-0 logging either by-message or by-recipient;
- syslog priorities are now dynamically derived from the log level;
- constantly updates a small database of SNMP-style counters, providing
  real-time measurements for status monitoring and statistics reporting,
  reducing the need for analyzing a log file; includes a sample/demo
  program 'amavisd-agent';
- new 'policy delegation protocol' between helper program and the daemon
  can pass more information to the daemon and allows the daemon to
  pass more instructions back to MTA (useful for sendmail milter setup);
- persistent cache of recent virus and spam checks, common to all child
  processes, improves the cache hit rate;
- support for IPv6 address formats;
- provisional/experimental support for DSPAM spam checker;
- support for ClamAV virus scanner via Perl module Mail::ClamAV;
- cleaned amavisd.conf :
    amavisd.conf         configuration file with the essentials
    amavisd.conf-default lists all configuration variables with their defaults
    amavisd.conf-sample  traditional-style commented amavisd.conf with examples

----

INCOMPATIBLE CHANGES since amavisd-new-20030616 (any patch level)

- requires Perl module BerkeleyDB with libdb version 3.1 or later
  (tested with db 4.1); This requirement will be made optional
  at a later date (in amavisd-new-2.1.0);

- a directory at $db_home (default /var/amavis/db) must be manually created
  to store cache and snmp DB files. It should be writable by user running
  amavisd. The db files within are removed and re-created at each restart
  to avoid having to deal with db recovery (but need not be, as far as the
  program logic is concerned);

- sending signal HUP in order to restart amavisd no longer works (previously
  it only worked in non-chrooted environment and relied on guessing amavisd
  absolute path); please use 'amavisd reload', or 'amavisd stop' and restart;

  If the HUP method is really still needed, please replace the line
    commandline => [],  # disable
  by:
    commandline => ['/usr/local/sbin/amavisd','-c',$config_file],
  in file amavisd, adjusting the path if necessary.

- due to changed names of temporary files, the old 'DrWebD' av entry needs
  to be replaced with the current one;

- changed $final_virus_destiny default from D_BOUNCE to D_DISCARD
- changed $final_spam_destiny  default from D_REJECT to D_BOUNCE

- changed defaults for variables $virus_quarantine_to, $banned_quarantine_to,
  $bad_header_quarantine_to and $spam_quarantine_to from undef (no quarantine)
  to values 'virus-quarantine', 'banned-quarantine', 'bad-header-quarantine'
  and 'spam-quarantine' respectively.  Set them to undef or '' (empty string)
  to disable quarantine;

- add address extension at tag2 level, not at kill level as before;
  suggested by Jacob Elder and others;

- because of the reorganization of lookup tables, a new way
  of quickly disabling virus or spam checks in amavisd.conf is used.
  Instead of:
    # @bypass_virus_checks_acl= qw( . ); # uncomment to DISABLE anti-virus code
    # @bypass_spam_checks_acl = qw( . ); # uncomment to DISABLE anti-spam code
  the new recipe is:
    # @bypass_virus_checks_maps = (1);   # uncomment to DISABLE anti-virus code
    # @bypass_spam_checks_maps  = (1);   # uncomment to DISABLE anti-spam code

- @virus_admin_maps and @spam_admin_maps now take as lookup keys recipient
  addresses, not sender address as before. The new semantics was often
  requested, the old semantics was not useful because modern viruses and
  spam fake sender address, so the choice was made to incompatibly change
  semantics and use the same config variable names, rather than introduce
  new names and leave behind useless variables;

  An implication is that with multiple-recipient mail it is now possible
  to have more than one admin notification generated when recipients have
  different admin addresses associated. Still, each distinct admin address
  receives only one admin notification;

- removed old compatibility variable $mailto. Use $virus_admin and $spam_admin
  variables instead, as suggested in amavisd-new-20021116 release notes,
  or the more recent inventions @virus_admin_maps and @spam_admin_maps,
  or their equivalents in policy banks;

- removed support for old configuration variable $mailfrom. Use variables
  $mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip,
  and $mailfrom_notify_spamadmin as introduced in amavisd-new-20020630
  for the purpose;

- removed ancient variable @local_domains, use @local_domains_acl instead,
  or the more general @local_domains_maps

- removed old compatibility code which allowed for a couple of traditional
  variables to treat value "no" as false. The use of "no" for false has been
  deprecated since amavisd-new-20021116.

- revoke an old compatibility measure where a missing (undef) tag2 level
  would fall back to the kill level value;

- lookup_hash incompatible change (but hardly anyone will notice):
  a key presence in the Perl hash but with undefined hash value
  used to be interpreted as true, but now it is treated as undef,
  causing search to continue with remaining lookup tables (if any).
  This is more general and more in spirit with other lookup mechanisms;

- changed a default to initialize SAVI-Perl every time a child process is
  started, no longer at master process startup time only. This is to avoid
  the need to restart amavisd every time the Sophos IDE database is changed.
  One can revert to the previous behavior by uncommenting a call to
  Amavis::AV::sophos_savi_init in subroutine fetch_modules_extra;


SECURITY

- no longer invoke shell to call gzip for compressing quarantined files
  or to call the sendmail command for submitting messages; the most important
  consequence is that a shell is no longer needed in a chroot environment
  and should preferably be removed;

- not to forget what was introduced in -p10: inserted a security check
  for a missing Net::Server patch, and abort if vulnerable;

- new command line options '-u user' and '-g group' are available.
  These are pretty much equivalent to doing a su(1) to the specified user
  first (in which case the use of these options is redundant).

  By doing 'su' or by specifying a command-line option '-u username'
  one can prevent a potential security risk on misconfigured sites where
  amavisd.conf is writable by UID running amavisd (e.g. not owned by root).
  If a (non-root) username or UID is specified, privileges are now dropped
  _before_ opening and evaluating a configuration file. The consequence
  is that the configuration variables $daemon_user and $daemon_group
  (in amavisd.conf) can not have an after-effect (a warning is issued
  if different).

  If -u is not specified, the behaviour is as before, i.e. the config file
  is read and evaluated under the current UID (as root unless 'su' was done),
  and the values of $daemon_user and $daemon_group from the config file
  are passed to Net::Server, which changes UID during its startup after
  chroot-ing (if requested).
  
  If chroot is desired, the -u must not be used: the root privilege is
  required to do chroot, and the config file must be read _before_ doing
  chroot. A case of Catch-22. Be doubly careful of who can modify the
  configuration file.

  Another consequence of specifying -u is that any external files (e.g.
  templates, lookup hashes) as possibly read from amavisd.conf, are now
  accessed as unprivileged user and no longer as root. The same goes for
  opening the log file when not logging via syslog.


MAJOR NEW FEATURES

- policy banks hold sets of configuration variables controlling most of
  per-message settings, including: static lookup tables, IP interface
  access rules, forwarding address, log level, templates, administrator
  addresses, spam trigger levels, quarantine rules, lists of anti-virus
  scanner entries (or just a subset), banned names rules, defang settings,
  etc. The whole set of these settings may be replaced with another
  predefined set based on incoming port number, making it possible for
  one amavisd daemon to cope with more diverse needs of served user
  communities which could so far only be implemented by running more than
  one instance of the amavisd daemon, each with its own configuration file;

  This mechanism brings new potentials for the future: in principle policy
  banks could be swapped not only based on port number or SMTP client
  IP address, but on any characteristics pertaining to a mail message as
  a whole (not specific to each of its recipients), or to characteristics
  of a connection from a mailer (e.g. the interface address or protocol);

  Until a better mechanism is available, a policy bank named 'MYNETS' has
  special semantics: this policy bank gets loaded whenever MTA supplies a
  SMTP client IP address (Postfix XFORWARD extension or a new AM.PDP protocol)
  and that address matches the @mynetworks list.

  A hash %$interface_policy is a current mechanism of assigning a
  policy bank to an incoming TCP port number (port must be in the list
  @$inet_socket_port). Whenever the connection from MTA is received,
  first a built-in policy bank with an empty name - the $policy_bank{''}
  gets loaded, which bringings in all the global/legacy settings.
  Then it is overlaid by whatever configuration settings are in the bank
  named in the $interface_policy{$port} if any, and finally the bank 'MYNETS'
  is overlaid if it exists and the SMTP client IP address is known
  (by XFORWARD SMTP extension command from MTA) and it matches @mynetworks.
  See amavisd.conf-sample for examples.

  When a new policy bank is overlaid over an existing set of configuration
  variables, the variables not present in the new policy bank retain their
  value.

  The built-in policy bank (with empty name) is predefined, and includes
  references to most other variables (the dynamic config variables),
  which are accessed only indirectly through the currently installed
  policy bank. Overlaying a policy bank with another policy bank may
  bring in references to entirely different variables, possibly unnamed.

  Configuration variables are referenced from a built-in policy bank (which
  is implemented as a perl hash, i.e. an associative array) by keys of the
  same name, e.g. { log_level => \$log_level, inet_acl => \@inet_acl, ...}.
  For scalars one level of indirection is allowed, e.g.
  a policy bank { log_level => \$log_level }; $log_level=2;
  is equivalent to { log_level => $log_level } or { log_level => 2 },
  but in the first example with an indirect reference, the $log_level
  may be assigned to even _after_ the policy bank has already been formed.

  One word of caution: the syntax of entries within a policy bank hash
  is slightly different from assignments to configuration variables.
  This is because entries within policy bank are not asssignments, but
  key=>value pairs as in any Perl hash. And these pairs are delimited by
  commas, unlike statements, which are delimited by semicolons.
  Value is separated from its key by '=>' (or by a comma), whereas the
  assignment operator is '='. Keys of a policy bank are without leading $
  or @ or %, unlike variable names. Values of a hash can only be scalars
  (e.g. strings or refs to arrays).

  Compare:
  - value of a policy bank is a reference to a Perl hash, e.g.:
    { log_level => 3,
      forward_method => 'smtp:[127.0.0.1]:10025',
      spam_admin_maps => ["spamalert\@$mydomain"],
    }
  - normal assignments look like:
      $log_level = 3;
      $forward_method = 'smtp:[127.0.0.1]:10025';
      @spam_admin_maps = ("spamalert\@$mydomain");

  And a final note: Perl can detect and report typing mistakes in variable
  names, but mistyped key is just some unused hash entry lurking in a hash,
  never used and never reported as mistyped/useless.

- @score_sender_maps is a soft variant of black- and white-listing,
  contributing positive or negative score points based on sender
  e-mail address. Btw, the 'score' in the name '@score_sender_maps' is meant
  as a verb, recipient is scoring a sender (= to grade, to determine the
  merit of);

  Whitelisting is becoming deprecated because it is often and easily abused
  (but blacklisting can still be useful);

  # ENVELOPE SENDER SOFT-WHITELISTING / SOFT-BLACKLISTING
  # Instead of hard black- or whitelisting, a softer approach is to add
  # score points (penalties) to the SA score for mail from certain senders.
  # Positive points lean towards blacklisting, negative towards whitelisting.
  # This is much like adding SA rules or using its white/blacklisting, except
  # that here only envelope sender addresses are considered (not addresses
  # in a mail header), and that score points can be assigned per-recipient
  # (or globally), and the assigned penalties are customarily much lower
  # than the default SA white/blacklisting score.
  #
  # The table structure is similar to $per_recip_blacklist_sender_lookup_tables
  # i.e. the first level key is recipient, pointing to by-sender lookup tables.
  # The essential difference is that scores from _all_ matching by-recipient
  # lookups (not just the first that matches) are summed to give the final
  # score boost. That means that both the site and domain administrators,
  # as well as the recipient can have a say on the final score.

- the MIME defanging (defang: to make harmless or less powerful)
  wraps the entire original mail in a MIME container of type
  'Content-type: multipart/mixed', where the first part is a text/plain
  with a short explanation, and the second part is a complete original
  mail, enclosed in a 'Content-type: message/rfc822' MIME part.
  Defanging is only done when enabled (selectively by malware type)
  and the malware is allowed to pass (*_lovers or *_destiny=D_PASS)

  Conventional mail header fields are retained, and header fields
  Resent-{From,Date,Message-ID} are added. A header field X-Amavis-Modified
  is inserted to indicate that the mail body has been modified.

  It is an experimental feature (disabled by default, except for mail bombs),
  and not very efficient for large mail.

  MIME defanging is unconditionally done for mail bombs, i.e. when the
  X-Amavis-Hold header field is added. The text in the first MIME part
  describes the reason, e.g.:

      WARNING, possible mail bomb, NOT CHECKED FOR VIRUSES:
      Exceeded storage quota 29089500 bytes by do_unzip; ...

  When MIME defanging is enabled for passed spam, the first MIME part
  contains the full SpamAssassin report.

  The MIME defanging feature is not available in the sendmail milter setup.

- new parameter: @mynetworks
  It is an IP access list which determines if the original SMTP client
  IP address belongs to our internal networks. It is much like the Postfix
  parameter 'mynetworks' in semantics and similar in syntax, and its value
  should normally match its Postfix counterpart. It only affects the value
  of a macro %l (=sender-is-local), and the loading of policy 'MYNETS'
  (if present). Note that '-o smtp_send_xforward_command=yes' (or its lmtp
  counterpart) must be enabled in the Postfix service that feeds amavisd,
  otherwise client IP address is not available to amavisd-new and new
  features based on @mynetworks do not work (the %l macro works as before
  and bases its decision on sender e-mail address matching local_domains);


LOGGING

- syslog priorities are now dynamically derived from the log level of each
  log message (the first argument of sub do_log). The priority as specified
  in the $SYSLOG_LEVEL configuration variable is ignored in 2.0 (no longer
  in 2.1), but the 'facility' is not ignored. This makes it possible
  to influence the log verbosity by syslog.conf settings. Here is an
  example of a useful syslogd.conf setting (some levels may be left out;
  assumes the $log_level is high enough, e.g. 2, to produce any low-priority
  log messages at all):

    mail.err      /var/log/messages
    mail.notice   /var/log/amavisd.log
    mail.info     /var/log/amavisd-info.log
    mail.debug    /var/log/amavisd-debug.log

  (On linux do not forget to prefix filenames in syslog.conf with
   a '-' to disable sync, which has much impact on syslog performance!)

  At the moment the mapping of message log levels to syslog priorities
  is hardwired:
    level <= -3: LOG_CRIT
    level <= -2: LOG_ERR
    level <= -1: LOG_WARNING
    level <=  0: LOG_NOTICE
    level <=  1: LOG_INFO  (in version 2.0)
    level <=  2: LOG_INFO  (since 2.1)
    else:        LOG_DEBUG

  Not to be confused with the $log_level configuration variable setting,
  which still works as before, suppressing generation of all log messages
  with log levels above $log_level.

  For efficiency reasons one should not specify unnecessarily high $log_level
  and then discard low syslog priority messages in the syslogd.

- added $log_recip_templ variable, which is similar to $log_templ,
  but gets called for every recipient (the $log_templ is evaluated only once
  per message). Normally one or the other log template should be disabled
  by assigning undef or an empty string to the corresponding variable;

- the default $log_templ no longer shows quarantine ID if quarantining is
  disabled;

- added a macro %. (a dot), which might be useful in the $log_recip_templ;
  Its value is empty when $log_templ is expanded, and is a recipient counter
  (starting by 1) when $log_recip_templ template is expanded. Based on
  this macro one can provide a single template for both the $log_templ
  and the $log_recip_templ if needed, or perhaps let the log entry for
  the first recipient be more verbose that the rest;

- added a macro %T which expands to a list of triggered SA tests, but only
  when $log_templ and $log_recip_templ are expanded. In notifications
  the %T is still a list of To: addresses. An overlaid semantics, but we
  are running out of letters and a macro expander rewrite would be needed;

- new macros %k, %1, %2, %O, please see README.customize


LOOKUPS

- new configuration variables make it more flexible to specify arbitrary
  list of lookup tables. Legacy configuration variables are still available
  and are referenced from the default values of @*_maps lists. If these lists
  are redefined, legacy variables are not used.

  The new variables (lists) are:

    @local_domains_maps
    @bypass_virus_checks_maps @bypass_spam_checks_maps
    @bypass_banned_checks_maps @bypass_header_checks_maps
    @virus_lovers_maps @spam_lovers_maps
    @banned_files_lovers_maps @bad_header_lovers_maps
    @virus_admin_maps @spam_admin_maps
    @virus_quarantine_to_maps
    @banned_quarantine_to_maps @bad_header_quarantine_to_maps
    @spam_quarantine_to_maps @spam_quarantine_bysender_to_maps
    @banned_filename_maps
    @spam_tag_level_maps @spam_tag2_level_maps @spam_kill_level_maps
    @spam_dsn_cutoff_level_maps
    @spam_modifies_subj_maps
    @whitelist_sender_maps @blacklist_sender_maps @score_sender_maps
    @addr_extension_virus_maps  @addr_extension_spam_maps
    @addr_extension_banned_maps @addr_extension_bad_header_maps

    @keep_decoded_original_maps
    @map_full_type_to_short_type_maps
    @viruses_that_fake_sender_maps

- for more informative logging of lookup operations, a new object type
  Amavis::Lookup::Label can be inserted to lists of lookup tables
  for the purpose of labeling the main purpose of the list;

- all lookup* subroutines can now return matching key when called in a
  list context;

- lookups can now return a list of all matching entries (not just the first
  match); used for the new soft- white/blacklists (@score_sender_maps);

- sub lookup() now allows for one level of list elements dereferencing,
  which makes possible the construction of the argument list and later
  still be able to modify its members (e.g. creation of regexp lookup table
  objects in the configuration file); It facilitates transition from old
  hard-wired lists of lookup tables to new @*_maps list variables which
  permits specifying an arbitrary number of lookup tables and to specify
  their search order;

- simplify and unify calls to lookup() by collecting arguments (references
  to lookup tables) in lists, e.g. @local_domains_maps, @virus_lovers_maps,
  @virus_admin_maps. These array variables default to lists of legacy
  variables, which are never directly used by the program. Either the
  individual legacy variables may be assigned to, or the entire list
  replaced, in which case the legacy variables no longer have any effect.

- lookup_acl: respect $localpart_is_case_sensitive setting;

- lookup_hash and lookup_sql: rewritten lookup_hash and factored out the
  common code from lookup_hash and lookup_sql to make_query_keys();

- lookup_hash bug fix: avoid splitting address literal as if it were
  a domain name; (a bug with key '.' not being tried for address literals
  fixed thanks to Uwe S. Fuerst);


SQL, LDAP LOOKUPS

- extended semantics of SQL field wblist.wb, which can hold a score value
  boost, which is interpreted as soft black/white-listing (the same semantics
  as the value in @score_sender_maps);

- recognize SQL server error 2013/"Lost connection to" and treat it
  the same as 2006/"MySQL server has gone away"; by Max Kalika;

- full domain stripping: @.sub.example.com @.example.com @.com @.

- lookup_hash and lookup_sql: limit the list of subdomain
  search keys to 10 levels as a sanity measure; e.g. for address
  user@14.13.12.11.10.9.8.7.6.5.4.3.2.com the subdomains keys search list
  starts at .9.8.7.6.5.4.3.2.com; (domain names are limited by standard
  to 127 levels);

- prepare SELECT statements on demand;

- recognize all-zero and all-null boolean fields as false;

- recognize new (optional) fields in the table 'policy':
    spam_dsn_cutoff_level
    virus_quarantine_to, banned_quarantine_to, bad_header_quarantine_to
    addr_extension_virus, addr_extension_spam,
    addr_extension_banned, addr_extension_bad_header

  (the addition of virus_quarantine_to was suggested by Harald Kapper);

- consider the SQL user id a string (no longer required to be numeric);
  thanks to Max Kalika / Gentoo support;

- LDAP white/black list support by Jacques Supcik (similar to contribution
  from Scott Dier and Eric Dorland, which I forgot about, sorry);

- added amavisSpamQuarantineTo to the LDAP schema;
  new version of LDAP schema; by Jacques Supcik:

  In the previous schema, the tag levels have been defined as integer.
  This is too restrictive, and have now been changed to strings
  (there is no float type in LDAP);

- added LDAP attributes for completeness: amavisBadHeaderLover,
  amavisBypassBannedChecks, amavisBypassHeaderChecks, amavisVirusQuarantineTo,
  amavisBannedQuarantineTo, amavisBadHeaderQuarantineTo; by Jérôme Schell;


DECODING / DEARCHIVING

- provide optional ability to retain complete email message in its
  un-decoded form (alongside its decoded parts) for a virus scanner
  to see (enabled if $keep_decoded_original_re matched string 'MAIL');
  suggested by Tomasz Papszun; (partly backported to amavisd-new-20030616-p8);

- rewrite code that generates new file names (Amavis::Unpackers::Part->new),
  and rewrite code dealing with banned names. Keep information about each
  part organized as a tree, matching the descendence of each part, new
  package/object Amavis::Unpackers::Part to collect such information;

- make Amavis::Unpackers an optional module: the interface code to external
  decoder/unpackers/dearchivers does not get compiled and does not consume
  virtual memory if $bypass_decode_parts is true; (previously it just
  didn't get called, but was sitting in memory nevertheless);

- decode RPM archives by converting them to cpio, if rpm2cpio and cpio
  are available;

- do_tnef: extract $tnef->message if it exists, not just $tnef->attachments;

- support extracting MS cabinet files (CAB) by calling cabextract, if enabled
  and found. Beware: Lars Hecking warns that cabextract 0.6 is quite buggy
  and the author has been notified.


VIRUS SCANNING

- support for ClamAV virus scanner via Perl module Mail::ClamAV,
  based on code by Roberto Pereira da Rosa;

- don't call virus scanners if there are no files in the directory to be
  scanned (e.g. mail with an empty body); some virus scanners don't like
  to be given an empty directory (e.g. Symantec newer savsecls);
  reported by Marco Bicca;

- rewritten/unified/generalized subroutines ask_daemon and sophos_savi
  based on the new subroutine ask_av;

- scan parts directory for file names exactly once regardless of the number
  of virus scanners and their arguments (containing '{}/*' or not);

- supply full original mail to virus scanners in case of MIME parse errors
  (in addition to any possibly decoded parts);

- when collecting file names to be virus scanned, prepare a hash which maps
  base file names to Amavis::Unpackers::Part objects, and makes it available
  to virus scanner interface routines, which may benefit from the additional
  information about the file to be scanned.

  In particular, the new interface to Mail::ClamAV now turns on the option
  CL_MAIL, and the interface to SAVI turns on the option MIME, when entire
  mail is passed to AV scanner for checking. This enables ClamAV and SAVI
  to attempt MIME decoding the file by itself.

  TODO: The same option (--mime) would need to be specified when calling
  'clamscan' and supplying a non-decoded mail for checking; pointed out
  by Riccardo Ghiglianovich and Michael Boelen;


OTHER EXTERNALLY VISIBLE CHANGES

- The 'amavisd reload' command is now implemented differently:
    old: signals SIGHUP to a running amavisd process and exits immediately;
         the running amavisd process (under control of Net::Server) when it
         receives a SIGHUP starts its own copy with same arguments and exits;
    new: signals SIGTERM to a running amavisd process, waits for it to finish,
         then continues (same as 'amavisd start') to become a new daemon;
  The new method works even when chrooted, and is more reliable when the
  existing process is slow to terminate, as it actively waits for the previous
  daemon to finish before proceeding to promote itself to become a new daemon.

- a simple demo program 'amavisd-agent' is provided, allowing
  for continuous inspection of SNMP counters; a path to the
  /var/amavis/db is hardwired in the program, modify it if necessary.

- server-side support for optional Postfix SMTP/LMTP command XFORWARD:
  information about the original SMTP client IP, its DNS name, HELO name
  and protocol used is now made available to the amavisd program for
  logging and other purposes. The same information can also be obtained
  from the 'Amavis policy delegation protocol (AM.PDP)' if the helper
  program supports it (useful for sendmail milter setup);

- client-side support for optional Postfix SMTP command XFORWARD:
  if MTA announces in its SMTP EHLO response that it supports XFORWARD,
  amavisd will provide additional information about the original SMTP client
  if the information is available (either from XFORWARD on the receiving
  side, or by the 'Amavis policy delegation protocol';

- server side support for the new amavis helper protocol which allows
  for header modifications, removal of recipient addresses (e.g. non- spam
  lovers) or rewrite of recipient addresses (e.g. adding address extensions),
  and specification of full SMTP response;

- modified search logic for matching mail parts against $banned_filename_re;

  The old search order for names did not result in what one might
  expect when pattern list elements with a value of false were used
  in $banned_filename_re. Namely, all three components were searched
  independently (Content-Type, declared name, and file(1) type)
  and a logical or was used. Because searches for each mail part were
  independent, it was not possible to specify for example that anything
  within a zip would be allowed. If any of these searches returned true,
  mail was blocked.

  To make this useful, a complete rewrite of mail unpacking was needed
  and all information be made available in one place after the unpacking
  is over, so that checking for banned names can be done all at once.

  The search order is now much the same as used in rsync and its server,
  see man rsync, section 'EXCLUDE PATTERNS'. The new comments in
  amavisd.conf-sample explain the new situation.

- replaced $relayhost_is_client by a more flexible specification:

  To make it possible for several hosts to share one content checking daemon,
  the IP address and/or the port number in $forward_method and $notify_method
  may be specified as an asterisk. An asterisk in the colon-separated
  second field (host) will be replaced by the SMTP client peer address
  (i.e. the MTA host). An asterisk in the third field (tcp port) will be
  replaced by the incoming SMTP/LMTP session port number plus one. This
  obsoletes the previously used less flexible configuration parameter
  $relayhost_is_client. An example:

    $forward_method = 'smtp:*:*'; $notify_method = 'smtp:[127.0.0.1]:10025';

  The same functionality can also by achieved by using a bigger hammer,
  the policy banks. These may completely replace the global settings
  for $forward_method and $notify_method, based on incoming port number;

- turn address extension variables (the so called "plus addressing")
  into recipient-based lookup tables, including the SQL lookups. For example:
    @addr_extension_virus_maps  = ('virus');     # defaults to empty
    @addr_extension_spam_maps   = ('spam');      # defaults to empty
    @addr_extension_banned_maps = ('banned');    # defaults to empty
    @addr_extension_bad_header_maps = ('badh');  # defaults to empty
  or perhaps:
    @addr_extension_virus_maps = (
      {'sub.example.com'=>'infected', '.example.com'=>'malware'}, 'virus' );
  suggested the Gentoo modification, Jacques Supcik, and others;

- log and report hits and tag/tag2/kill levels rounded to three decimal
  places (trailing zeroes trimmed), no longer rounded to one decimal place;

- added @spam_dsn_cutoff_level_maps, making it possible to specify
  different DSN cutoff levels for different recipient domains or users.
  In multi-recipient messages where recipients can specify different values,
  the maximum value is used for deciding whether DSN should be suppressed;
  suggested by Ales Casar;

- configuration variable $gets_addr_in_quoted_form is no longer used;
  knowledge about address form (quoted or not) is now implicit in the
  receiving protocol;

- if tag level turns out to be undef, it will not be shown in X-Spam-*
  header fields, and will be interpreted as having a value lower than any
  spam score when deciding whether to insert X-Spam-* header fields or not;

- added macros %a and %g:
    * %a  original SMTP session client IP address (empty if unknown)
    * %g  original SMTP session client DNS name (empty if unknown)
  (like macros %I and %M that were once proposed by Dibo in his 2002-07 patch)
  This information may be available from Postfix when XFORWARD protocol
  extension to SMTP is enabled, and can be made available by helper program
  (e.g from sendmail milter) when the new AM.CL protocol is used;

- added macro %p, expanding to a current policy bank name (or empty
  if a built-in policy bank is still in place);

- added macro %r, expanding to the contents of the first Resent-Message-ID
  header field, or empty if no such field exists. Include reporting the
  Resent-Message-ID in the log and in the sender notification;
  suggested by Oliver Gorwits;

- new configuration variable $addr_extension_bad_header for completeness;

- added $bad_header_quarantine_to, @bad_header_quarantine_to_maps,
  $bad_header_quarantine_method, and $warnbadhrecip for completeness;
  suggested by Robin Lynn Frank;

- MIME::Parser errors now contribute to bad-header checks, so that the
  header checking is now conceptually extended to MIME sub headers
  (Postfix similarly considers MIME subheaders part of mail headers);
  MIME::Parser 6.1xx or later is recommended.

- allow $*_quarantine_method to be undef as a quick way of disabling
  some quarantine (it also can be disabled as previously, by using
  method 'local:' and following its rules);

- persistent cache of recent virus and spam checks, global to all child
  processes, can improve the cache hit rate. Uses BerkeleyDB database (hash
  and queue) and its interlocking mechanisms (Berkeley DB Concurrent Data
  Store) for the purpose. The V3.1 or better is required, V4 is recommended.

- include version information in the 'Usage' text;

- rewritten lookup_ip_acl() and added ip_to_vec() to allow for IPv6
  address syntax as specified in rfc3513 to be used in IP lookups;

- @inet_acl now defaults to ('127.0.0.1', '::1'), i.e. it adds the
  IPv6 loopback address to the list;

- new configuration variable $sa_spam_level_char (defaults to '*')
  allows specifying another character for X-Spam-Level bar. Empty or
  undefined value disables inserting the X-Spam-Level header field;

- added configuration variable $sa_spam_report_header to enable/disable
  inserting the X-Spam-Report header; patch by Craig Sanders;

- added $banned_quarantine_to configuration setting to make possible the
  quarantining of banned mail to a different place from viruses;

- don't insert virus-, banned- and bad headers- related headers for passed
  mail to recipients with corresponding bypass_*_checks, making them believe
  the mail was not spam-checked (as they are not expecting such headers
  anyway); This was already done in version amavisd-new-20030616-p6
  but only for spam-related headers;

- for choosing address extensions use the same criteria as for adding
  header fields, e.g. pretend to not know the result of a certain test
  (virus, spam, ...) when recipient chooses to bypass such test,
  even if the result of the test is known;

- added variable $sa_spam_subject_tag1 (undef by default).
  If $sa_spam_subject_tag will not be inserted (at tag2 level), and
  $sa_spam_subject_tag1 is nonempty, this string (e.g. '***possible SPAM*** ')
  will be inserted into the Subject header field for spam levels above
  tag level; suggested by Immo Goltz;

- added separate configuration variables $banned_files_quarantine_method
  and $bad_header_quarantine_method. Quarantining of banned files and
  bad headers were previously controlled by $virus_quarantine_method;

- rewritten read_hash, it is now possible to specify key value (right-hand
  side) for each key. If value is not specified, a '1' is assumed as before;

- SMTP server support for rfc2554 authentications (PLAIN and LOGIN only)
  allows client authentication to be relayed to the MTA when message is
  forwarded. Might be useful if amavisd-new is used in a Postfix SMTP proxy
  setup, but is not needed for other setups. Disabled by default,
  see variables $auth_required_inp and @auth_mech_avail.

- SMTP client support for rfc2554 authentications (any authentication method
  as supported by Net::SMTP and Authen::SASL Perl modules). Authentication
  of forwarded mail (PLAIN and LOGIN only) is carried over from the incoming
  mail, authentication to be used when submitting notifications is controlled
  by configuration variables $amavis_auth_user and $amavis_auth_pass.
  Disabled by default, see variable $auth_supported_out.

- provisional/experimental support for DSPAM spam checker (pre 3.0):
  if configuration variable $dspam is nonempty and represents a path to
  a 'dspam' program, a message is passed to dspam and its inserted headers
  of the form X-DSPAM-* are axtracted and then made available for
  SpamAssassin rules to check and score if desired.

  All messages are currently presented to dspam as the same user, affecting
  how its database is being built. False negatives and false positives
  (based on SA assessment) are fed back into DSPAM as a simple form
  of auto-learning. Works reasonably well, but do not expect miracles.
  See subroutine spam_scan.

  Here is how DSPAM can be installed to be able to be used by amavisd-new:

    dspam 2.x:
    ./configure --enable-alternative-bayesian \
      --with-userdir=/var/amavis/dspam --enable-signature-headers \
      --without-local-delivery-agent --without-quarantine-agent

    dspam 3.0.0:
    ./configure \
      --with-dspam-home=/var/amavis/dspam --enable-signature-headers \
      --without-delivery-agent --without-quarantine-agent

    make install
    chmod u-s,a+rx /usr/local/bin/dspam
    chown vscan:vscan /var/amavis/dspam

  and the following can be inserted into the SA config file (local.cf)
  to make it recognize and incorporate DSPAM's assessment:

    header DSPAM_SPAM X-DSPAM-Result =~ /^Spam$/
    describe DSPAM_SPAM DSPAM claims it is spam
    score DSPAM_SPAM 0.5

    header DSPAM_HAM X-DSPAM-Result =~ /^Innocent$/
    describe DSPAM_HAM DSPAM claims it is ham
    score DSPAM_HAM -0.1

  Eventually DSPAM support should be removed from amavisd-new, as soon as
  SA will be able to call it on its own.


INTERNAL CHANGES

- reformatted the whole program, reducing indentation from 4 to 2
  and replacing TABs with spaces (with some dubious help from perltidy,
  plus lots of manual adjustments);

- completely rewritten code to handle both the old and the new amavis helper
  protocol, as well as Postfix 'TCP client/server table lookup protocol'
  as specified in the Postfix documentation: tcp_table(5);
  (process_policy_request, prepare_policy_query, check_amcl_policy)

- type_short may now be a list of short types. Typical use is to classify
  a MS executable as both an 'exe' and as 'exe-ms', which makes more specific
  banned rules possible without unnecessary complication in regexps;

- parts now carry attributes, which can be inspected for banned checks;
  current attributes are U for undecodable, and C for (en)crypted;

- opened another can of Perl worms (taint bugs): turn on Perl pragma
  "use re 'taint'" in all modules, and selectively turn it off where needed.
  It replaces cumbersome manual preservation of taintedness when regexp
  saved ranges are used without intention to untaint. Because of Perl bugs,
  strategically placed local($1,$2,...) are needed, otherwise previous
  taint flag in $1, $2, ... can be brought on to new variables, which can
  all of a sudden become tainted out of nowhere;

- catch and report throws (die) in pre_loop_hook() to properly report
  problems during initialization;

- introduced subroutine exit_status_str and unify reporting of
  subprocess status;

- enhanced sub best_try_originator to ignore IP addresses from private,
  local and dedicated IANA networks (rfc3330) and look for the first
  public address in the 'Received' path;

- examine first four (chronologically) Received header fields (instead of
  first two) when looking for an originator IP address, and ignore those
  with private IP addresses;

- moved code dealing with body digest cache to a new package
  Amavis::Cache to facilitate transition to shared or persistent cache;

- new explicit cache expiration time controls (time to live in seconds):
    $virus_check_negative_ttl $virus_check_positive_ttl
    $spam_check_negative_ttl $spam_check_positive_ttl
    $cache_entry_ttl

- discard cache db ($db_home) and recreate it during restart;

- more informative changes of child process $0, which may show in the ps(1)
  output;

- store tempdir of a current message to the Amavis::In::Message object;

- gather some statistics about idle time;

- reorder and adjust mapping from file(1) results to file type classes;

- optimization: instead of invoking file(1) utility program for each
  part to be analyzed, now call it once for each round of currently
  available parts, giving it the list of all available parts as arguments;

- shorten the names of parts from part-..... to p..., to be able to stash
  more files names into a command line, e.g. when calling file(1) or
  external virus checker which can not deal with directories;

- use regexp lookup table mechanism (table $map_full_type_to_short_type)
  to match long types (output of file(1) utility) to short types (.exe,
  .jpg, .doc, ...).  The default table can be replaced by a customized
  table in amavisd.conf;

- replace $(?!\n) with \z in regular expressions throughout;
  replace most of the remaining $ with \z in regular expressions.
  The regular expression primitive \z is available since perl 5.005.

- TODO: disregard $MAXFILES during initial MIME unpacking;
  reported by Stephane Lentz and Robert LeBlanc;




Some un-edited notes on the new banned rules mechanism:
(I wrapped your log lines, and replaced \\ by \ for clarity:

| Feb 24 19:07:29 hauptpostamt amavis[29847]: (29847-04-5) p.path BANNED:
| 
|  "P=p002,M=application/octet-stream,T=zip,N=document.zip |
|   P=p003,T=exe,T=exe-ms,N=document.htm   .scr",

part p003 is of type (file(1)) MS executable, with suggested
name "document.htm   .scr" (lots of spaces in the name)

its parent resides on temp file p002 (i.e. p003 was extracted from it),
which is of type (T) zip archive, with suggested (MIME) name
(N) "document.zip", and has a MIME type (M) "application/octet-stream".

such a component p003 lying within such p002 is considered banned
by the following regexp rule (one rule within the $banned_namepath_re list):

|     matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* \.
|                    (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"

which says that any component at any level must not have a name (N)
matching a pattern:
  any number of characters,
  a dot,
  any number of non-dot and non-slash characters
  a dot,
  and ending with: exe or vbs ...
(basically: double extension ending with listed extensions)

The complications such as using [^\t\n]* instead of .* are there
to keep regexp contained within fields and ancestors/descendents.


There is one detail to remember when comparing logged p.path log entries
and the actual matching rules:

- for the sake of readability the logged entry has \n (newlines) converted
  into ' | '. The \n is a separator between components in the tree
  from the root (the mail itself, hidden) to the the leaf component
  which can not be further expanded (i.e. not an archive)

- for the sake of readability the logged entry has \t (a tab) converted
  into comma, separating information fields such as P=...  M=..  T=..  M=..

So the above logged string:
  P=p002,M=appl...,T=zip,N=document.zip | P=p003,T=exe,T=exe-ms,N=document.htm.scr
is actually a single string:
  P=p002\tM=appl...\tT=zip\tN=document.zip\nP=p003\tT=exe\tT=exe-ms\tN=document.htm.scr

and a Perl regexp is applied directly to it.

The raw string is rather unsightly, but the \n and \t were chosen
to minimize clash with valid characters within file names.

If a \n or \t is present in a name of the components, such character
is converted into a space to avoid clashing with separators.


| Feb 24 19:11:58 hauptpostamt amavis[31505]: (31505-01-5) p.path BANNED:
|   "P=p002,M=application/octet-stream,T=zip,N=paypal.zip |
|    P=p003,T=exe,T=exe-ms,N=paypal.scr",

a MS executable named "paypal.scr" within a zip archive "paypal.zip"

|      matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \.
|                    (exe|vbs|pif|scr|bat|com) (\t.*)? $)"

block component at any level with a name (N) terminating 
by dot followed by any of the listed extensions.



| Feb 24 19:18:25 hauptpostamt amavis[32159]: (32159-01-2) p.path BANNED:
|   "P=p002,M=application/octet-stream,T=zip,N=text.zip |
|    P=p003,T=exe,T=exe-ms,N=text.txt     .exe",

a MS executable named "text.txt     .exe" (with lots of spaces in the name)
within a zip archive named "text.zip"

|      matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n
|                    (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"

blocked by the double-extension rule.


| Feb 24 19:30:15 hauptpostamt amavis[1690]: (01690-02-8) p.path BANNED:
|   "P=p002,M=application/octet-stream,T=zip,N=jokes.zip |
|    P=p003,T=exe,T=exe-ms,N=jokes.doc   .exe",
|      matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n
|                    (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"

same thing
---------------------------


---------------------------------------------------------------------------
                                                              June 29, 2004
Patch: amavisd-new-20030616-p10

- insert a security check to test for a missing Net::Server patch, and abort
  if vulnerable;

- provide and use our own subroutine q_encode to do the Q-encoding when
  editing an existing invalid header field with non-encoded 8-bit characters,
  e.g. when inserting ***SPAM*** or ***UNCHECKED*** into Subject.
  The MIME::Words::encode_mimeword() does not encode spaces and does not limit
  encoded words to 75 characters, which violates the RFC 2047 and breaks mail
  readers; reported by Sebastian Hagedorn and Gregor Hoffleit;

- fixed a bug in inserting the tag_level header field, which was missing
  if sender was blacklisted and tag_level was greater or equal to 0;
  thanks to Joerg Thaler;

- amavis-milter.c
  * log envelope sender address at the same log level (DBG_INFO) as
    recipient addresses;
  * remove the log message "(mlfi_eom) header already present", it was
    inappropriate, the call to smfi_chgheader succeeds even if no such
    header was already present;
  * relax permissions on created directory and temporary file to allow
    group read access (needed if virus scanner runs under a different
    user id but within the same group); by Adam C. Migus;
  * add queue id (when available) to most log messages;

- do not preserve evidence just because a message gets an X-Amavis-Hold
  header field;

- when logging directly to a file and started as root, change UID of a
  log file to $daemon_user to avoid restart problems; based on a patch
  by Carsten Hoeger and Gregor Weiss;

- added configuration variable $first_infected_stops_scan
  to stop anti-virus scanning when the first scanner detects a virus;
  the default is false, all scanners in a group are called (as usual);

- in the AV entry for clamscan added the option '--tempdir=$TEMPBASE';

- in the AV entry for 'Norman Virus Control v5 / Linux'
  changed the command name 'nvccmd' into 'nvcc'; correction by
  Michael Ramke of the Norman Data Defense Systems GmbH;

- insert debug reports into sub ip_addr_to_name to help recognize DNS problems;

- revoke deleting an existing 'X-Scanned-By' header field, which was
  introduced in -p8; removing it gets in a way when more than one content
  filter is being chained;

- load a missing SpamAssassin v3.0 module, needed when running in chroot;

- contributed (for now in the form of an optional patch): support for
  the incoming qmqpqq protocol over a TCP socket - to be used with qmail.
  Apply the provided patch 'amavisd-new-qmqpqq.patch', it updates files
  amavisd.conf and amavisd in the current directory; by Martin Solciansky;

- documentation updates;

---------------------------------------------------------------------------
                                                              April 2, 2004
Patch: amavisd-new-20030616-p9

- avoid choking on undefined $banned_filename_re; thanks to Ales Casar,
  Sebastian Hagedorn and E. Falk;

- if Subject mail header field got $undecipherable_subject_tag inserted, it
  would also receive the spam tag $sa_spam_subject_tag unconditionally;
  fixed, thanks to Francis Stevens;

- updates to the @av_scanners list in amavisd.conf:
  * DrWebD now works with a new DrWeb daemon 4.31,
    thanks to information provided by Krzysztof Cegielski, DrWeb Polska;
  * updated BitDefender bdc, thanks to Alfredo Milani Comparetti;
  * clamscan: use documented option --no-summary instead of the
    undocumented/old(?) option --disable-summary; by Georgy Salnikov;
  * updated Kaspersky aveclient and F-Secure fsav, thanks to Tomi Hakala;
  * recognize that KasperskyLab antivirus in demo mode turns on the bit 0x10
    in status; avp and avpdc now use the same two sets of status codes;
    avp: statuses 3 and 6 moved to infected (to match avpdc);
    statuses 2 and 5 considered infected; all suggested by Georgy Salnikov;

- when original undecoded mail is to be kept for virus scanners (requested
  by patterns /^MAIL$/ or /^MAIL-UNDECIPHERABLE$/ in $banned_filename_re),
  the preserved file is now named parts/part-00000 instead of parts/email.txt
  to preserve the size of the name, upon which some virus scanners depend
  (e.g. DrWebD);

  A note to Courier users: due to the way a file name in Courier is created
  and passed to amavisd-new, it is currently not possible to use the triggers
  /^MAIL$/ and /^MAIL-UNDECIPHERABLE$/ in $keep_decoded_original_re, or a
  failure to create a hard link occurs; Thanks to Bowie Bailey for helping to
  troubleshoot the problem; (this limitation is lifted in amavisd-new-2.2.0)

- remove option -c when calling gzip, bzip2, compress, lzop and unfreeze
  (as has long been done in the development version); the option -c is not
  needed when no file argument is present, and some implementations of gzip
  and compress may choke on it;

- make loading of Perl module Carp::Heavy optional, versions of Carp that
  came with Perl 5.005 did not have it; reported by Jefferson Pizzolatti;

- load module Mail::SpamAssassin::BayesStore::DBM, required by SA 3.0
  running in chroot jail;

- look for program 'gcpio' ( $cpio = ['gcpio','cpio'] ), as on OpenBSD the
  plain cpio does not support required options (the --no-absolute-filenames
  is essential), but GNU cpio does; thanks to Manfred Gloiber;

- relax parsing of file(1) output to allow tab as well as space to follow
  a file name; the file(1) on Solaris uses tab instead of space;
  suggested by Glen Harris;

- make LHA understand self-extracting archives (SFX); patch by Georgy Salnikov:

  Although lha unpacks any non-SFX archive independently if its extension,
  it unpacks the SFX lha archives only if they have the extension .exe.
  Now the file is symlinked to $part.exe for checking by lha: this will now
  work for SFX, and will still work for non-SFX. Also, do_lha will return 0
  if $exec and the part cannot be de-archived, the same as in do_unrar.

  The executable file formats are checked for being zip/rar/lha SFX archives.
  However, if the SFX archive is not a zip archive, do_unrar always returns
  success, so that if the archive is also not a rar, it will be never checked
  by lha. Now do_unrar returns 0 if $exec and the corresponding part can not
  be un-RARed;

- do_executable and do_unarj: added checking for SFX arj; by Georgy Salnikov:
  (commented out the call to do_unarj in do_executable until more experience
  is gained on how well unarj survives certain mail contents; Mark);

- do_unarj: let arj/unarj work on file named part*.arj;

- when calling IO::File::open() use '+>' instead of 'w+' to avoid
  Perl taint bug ($mode turns tainted) (bug still present in 5.8.2)
  triggered by expression in IO::Handle::_open_mode_string();

- attack the Perl 5.8.0/5.8.1/5.8.3 taint bug (once variables $1,$2,etc
  get tainted they start spreading taintedness to other variables):
  * insert local($1,$2,$3,...) in blocks of code which call external
    modules which trigger the bug (Mail::SpamAssasin, MIME::Parser, ...)
  * insert local($1,$2,$3,...) in blocks of code which depend on these
    variables to be clean, and which demonstrated through bug reports
    and experience with various version of Perl that these variables
    were not always taint-clean;
  The last taint incident triggered by SA 3.0.0 (svn) reported by Luc de Louw;

- recognize status EX_NOUSER when forwarding via pipe to sendmail (old setup);
  previously it was treated as a temporary failure; patch by The Mindflayer;

- turn error message 'error reading from client socket' back into
  'client broke the connection without a QUIT' for consistency with P7;

- when %virus_admin lookup table is used, prefer $msginfo->sender_source
  (unmangled sender domain) over $msginfo->sender; suggested by Pawel
  Golaszewski;

- nicety: in virus recipient notifications now supply the To: header field
  with the true recipient address instead of "undisclosed-recipients:;"
  in case of single-recipient mail; suggested by several people;

- amavisd would insert differently capitalized header fields (either
  X-AMaViS-Alert or X-Amavis-Alert), depending on the reason being reported;
  now use the X-Amavis-Alert throughout; reported by Carsten Hoeger;

- fetch Perl auto-loaded modules auto::POSIX::setgid and auto::POSIX::setuid
  if they exist; they are needed by older versions of Perl when running in
  chroot. A manual change was needed until now (documented in README.chroot),
  which should no longer be needed; a FreeBSD problem report 64636.

- helper program amavis-milter.c (used in the sendmail milter setup):

  * amavis-milter.c only included the client IP and client host name on the
    first mail transaction of a multiple-transaction SMTP session, but not
    in subsequent transactions; pointed out by Stephane Lentz. The solution
    was once already provided back in July 2002 by Radoslav Dibarbora - Dibo,
    and forgotten; credits where credits are due:
      http://marc.theaimsgroup.com/?l=amavis-tech&m=102569946910555

  * make a clear distinction between message data and connection data,
    which required code reshuffling and revealed previous unlean solutions;

  * add error checking and reporting to mkdir/rmdir/open/unlink/write
    system calls; previously an error could pass by unnoticed or just
    caused a tempfail without an explanation;

  * change final milter status ACCEPT into CONTINUE to allow further
    milters in the milter chain to examine the mail;

  * code cleanup;

  * regenerated helper-progs/configure with Autoconf 2.57 to make it
    capable of detecting mkdtemp and still be able to find sendmail
    libmilter files; thanks to Sebastian Hagedorn for a problem report,
    for testing, and for revealing a bug in the pre-released version;

  * use different syslog priority for different internal message
    log levels; by Sebastian Hagedorn;

  * adjust log levels of messages, set default verbosity 1 (DBG_WARN);

- documentation updates, including the updated comments in amavisd.conf
  regarding whitelisted senders, to reflect the change indicated in the
  amavisd-new-20030616-p4 release notes;


---------------------------------------------------------------------------
                                                              March 9, 2004
Patch: amavisd-new-20030616-p8

- be compatible with SpamAssassin version 2.70 and 3.0 as well as 2.6x;
  SA changed its API, replacing Mail::SpamAssassin::NoMailAudit
  with $spamassassin_obj->parse and belonging objects;

- as a stop-gap solution to the W32/Bagle-{F,...} detection problem
  (password-protected zip archives), three new measures are available:

  * ability to present the full non-decoded original message to virus
    scanners was partly back-ported from the development version.
    Enabled by adding qr'^MAIL$' or qr'^MAIL-UNDECIPHERABLE$' to the
    list in $keep_decoded_original_re, as illustrated in amavisd.conf;
    similar to a patch by Ted Cabeen. The following keys are used for a
    lookup into $keep_decoded_original_re:

    'MAIL'
      always provide a full original message to virus scanners
      (besides its successfully decoded components);

    'MAIL-UNDECIPHERABLE'
      same as for 'MAIL', but only if it contains undecipherable components
      such as password protected archive members, unsupported compression
      methods or encrypted parts (e.g. with PGP). Don't put too much trust
      into this, as some more exotic file formats may not be understood
      and not flagged as undecipherable;

  * a key 'UNDECIPHERABLE' is matched against $banned_filename_re when mail
    contains any undecipherable components, and if lookup returns true,
    mail will be banned. For example:
  
    $banned_filename_re = new_RE(
      qr'^UNDECIPHERABLE$',   # contains any undecipherable components
    ... );

  * a string can be prepended to Subject (for local recipients only)
    if mail could not be decoded or checked entirely, e.g. due to
    password-protected archives or non-decodable mail bombs:

    $undecipherable_subject_tag = '***UNCHECKED*** ';  # undef disables it

  NOTE: this solution is a quick-fix response to popular demand.
  Although the same or similar functionality will probably remain
  in future versions, the syntax and exact semantics may be refined.

- bring do_unarj in line with the rest of de-archivers: provide the
  same degree of mail bomb protection that was available in do_unrar;
  retain original archive for inspection by a virus scanner if it contains
  any members that can not be extracted (e.g. password protected members
  or unsupported compression schemes). (The .arj size checking deficiency
  was mentioned in the AMaViS Security Announcement (ASA) 2004-01-19);

- don't call a virus scanner if there are no files (no mail parts) to scan;
  it caused problems with certain scanners like aveclient, which expect
  a list of file names as arguments and complain if the list is empty;
  reported by Daniel Luttermann;

- a much needed feature: can specify $sa_dsn_cutoff_level in amavisd.conf
  to suppress sending a DSN (delivery status notification) when spam level
  is above this value, effectively turning D_BOUNCE into D_DISCARD for
  this message; undef disables the feature and is a default. A good
  first approximation value is 10, or with some risk go down to 8.
  The parameter has no effect if DSNs are already disabled (e.g. when
  $final_spam_destiny is D_DISCARD or D_REJECT);

- do_unrar: double check the archive size (against summary line as well);

- derive the value of macro %l from sender_source instead of from a declared
  sender address, so as to not rise false alarms when sender address is known
  to be faked; patch by Joerg Friedrich;

- reduce the number of retries to connect to a daemonized virus scanner
  from 3 to 2, so that a fallback to backup scanners occurs sooner;

- updated $banned_filename_re example in amavisd.conf, modify it to will;

- updates to the @av_scanners list in amavisd.conf:

  * KasperskyLab AVP - aveclient: *IMPORTANT* updated entry for the exit
    status only reflects the result of the last file scanned, we must use
    regular expression to detect viruses; fix provided by Andreas Triller;

  * Trend Micro FileScanner - vscan: added option -za, otherwise
    some broken archive may sneak-in a virus; suggested by Stephane Lentz;

  * Sophos sweep: added options -cab -tnef --no-reset-atime;

  * Dr.Web command line scanner: updated options, recognize exit status
    when using evaluation license;

  * ClamAV clamd: change socket location to a more usual value, adjust to will;

- amavisd.conf: added new virus names to the $viruses_that_fake_sender_re
  list, and uncommented the [qr/.*/=>1] line, so that by default any unknown
  virus will be treated as a sender-faking virus; adjust to will;

- do not send bad header notifications in response to messages
  from mailing lists;

- added header check for folded header field lines made up entirely of
  whitespace (a 'header space gap' violation to rfc2822); a check is only
  made when other header checks (bad character in header) are enabled;

- distinguish an empty string from undef in $mailfrom_to_quarantine,
  making it possible to specify a null return path when quarantining
  to a mailbox;

- helper-progs/amavis-milter.c (sendmail) enhancement based on work
  by Stephane Lentz: 

  * the name of a temporary directory is derived from the sendmail queue ID,
    making it easier to match sendmail and amavisd-new log entries;
    the queue ID is also a part of the quarantine file name;

  * a phantom 'Received:' header field is prepended on the temporary file
    to preserve the information on the original SMTP client IP address,
    host name and queue ID. This trace header field does not propagate to
    recipients (is not inserted into the original message), but is available
    in the quarantined messages and is visible to SpamAssassin.
    Adding a milter macro {b} on ENVFROM is advised to preserve the MTA
    timestamp in the log, although not mandatory (falls back to current time):

      define(`confMILTER_MACROS_ENVFROM',
              confMILTER_MACROS_ENVFROM``, {b}'')dnl

  If the new amavis-milter.c gives you trouble, switch the soft link to the
  previous version in the same directory. Keeping macro {b} does not hurt.

- catch possible errors in pre_loop_hook and report them properly;

- better check for I/O errors on SMTP input socket;

- add the following modules to the list of pre-fetched modules:
    Carp::Heavy, IO::Handle, IO::Socket::UNIX, IO::Socket::INET.
  The absence of Carp::Heavy could mute error report or a backtrace
  when running chroot-ed; the rest are for completeness only;

- added a macro %z which expands to the original mail size (in bytes);
  it could be useful in the $log_templ; thanks to Nick Leverton;

- make SpamAssassin timeout value configurable: variable $sa_timeout;
  by Henrique M. Holschuh (Debian);

- 'Received:' header field cosmetics: use 'unknown' in case the HELO argument
  was empty, and put-in a 'unix socket' if message was received from a helper
  program; by Henrique M. Holschuh (Debian);

- remove more pre-existing X-Spam* header fields from other scanners:
  X-Spam-Tests, X-Scanned-By; by Henrique M. Holschuh (Debian);

- helper-progs/amavis.c updates (old sendmail setup): report errors,
  log to syslog, change default dir to something obvious for bug reporting;
  by Henrique M. Holschuh (Debian);

- updated documentation;

---------------------------------------------------------------------------
                                                            January 5, 2004
Patch: amavisd-new-20030616-p7

- do_unzip and do_unrar: retain an archive if any of its components is
  password protected or encrypted (plus unpack what can be unpacked,
  as before). This gives virus checkers a chance to examine the original
  unpacked archive as a whole (e.g. scanning for variants of W32/Mimail),
  matching it in non-decoded form against virus patterns even if containing
  password-protected components. As a consequence, some virus scanners
  may now log their complaint when encountering such protected archives
  which previously didn't reach them. Such log entries should be considered
  informational only;

- add module Net::DNS::RR::AAAA to the list of Perl modules to be fetched
  before chroot takes place; thanks to Per olof Ljungmark;

- preload Perl modules DBD::*, based on @lookup_sql_dsn;
  required when using SQL lookups from a chroot jail;

- updated example dsn in @lookup_sql_dsn (file amavisd.conf) to use the
  new DBD::mysql syntax, and to show how to force accessing SQL server
  via inet socket which makes it easier to use from a chroot jail;

- disregard cached spam results for mail with small body.
  The most pronounced undesirable effect was in mail with
  an empty body where spam score was derived from header only;

- change the default value for local_domains_sql lookup for the catchall key
  '@.' under conditions: when using SQL lookups and user record with key '@.'
  is present in the database and a field 'local' is not present. Previously
  it surprisingly defaulted to true, now it falls back to static lookup table
  defaults, the same as if the record '@.' were not present in the table;

- fixed ugly text formatting in recipient notifications template;
  reported by Florian Effenberger;

- fixed parsing of 'Received:' header field for some unusual cases;

- updated amavisd-new-courier.patch to apply cleanly against -P6 (now P7);
  by Martin Orr;

- added av_scanners entry for 'AVG Anti-Virus', kindly provided by
  Grisoft s.r.o. from Czech Republic, http://www.grisoft.com/

- make Dr.Web Daemon av entry work with evaluation or regular license;
  thanks to Andrew I Baznikin;

- added av_scanners entry for new AVP client (aveclient) that is shipped
  whith avp 5.x.x.x.  Moved kavscanner to the @av_scanners_backup list
  (it is presumably slow and less reliable than aveclient), updated other
  AVP/Kaspersky entries; thanks to Nabil Sefrioui;

- added status codes 10 and 15 to the list of ok statuses for kavscanner
  (10=Password-protected archives, 15=Corrupted files);

- add an example dummy virus scanner 'all-clean' to the @av_scanners_backup
  list which always succeeds, always returning false (= status clean).
  Uncomment it if desired to avoid mail requeue when all other scanners fail,
  and to just pass the mail unchecked;

- disregard rfc2821 recommendation that 552 smtp response code
  should be treated as 452. It is unnecessary in amavisd-new setups,
  and it is wrong because 552 has other meanings assigned to it
  besides "too many recipients";

- allow IP address in $forward_method and $notify_method to be bracketed,
  which is needed for IPv6 addresses containing colons. Both formats
  are now allowed: 'smtp:127.0.0.1:10025' and 'smtp:[127.0.0.1]:10025',
  the later is now preferred;

- change IP address bracketing in log entries 'SEND via SMTP' and
  'FWD via SMTP' from [127.0.0.1:10025] into [127.0.0.1]:10025 to match
  the syntax in the connect log entry and to facilitate parsing of IPv6
  addresses from the log;

- update received_line() to generate valid Received header field even for
  IPv6 client addresses;

- modified helper program amavis.c to allow it to be run non-root and to
  set temp file mode for group accessibility. Change its default log level
  to DBG_WARN;

- fix an incorrect SELECT example in README.lookups; thanks to Nabil Sefrioui;

- documentation updates;


---------------------------------------------------------------------------
                                                          November 10, 2003
Patch: amavisd-new-20030616-p6

- change SQL lookup code to better handle SQL database server restarts.
  After a SQL server restart amavisd-new would previously TEMPFAIL (4xx)
  all messages until amavisd child process would run down, then resume
  normal operation with the new child birth. Now the SQL server reconnect
  is done when the next mail arrives, so only one mail with each amavisd
  child process TEMPFAILs during SQL server restart, and normal operation
  resumes faster;

- in flatten_and_tidy_dir and in do_ascii: sanitize protection of files and
  directories which may otherwise be made inaccessible to virus scanners;
  based on patch by Henrique de Moraes Holschuh (Debian amavisd-new support);
  problem reported by Aspa and by Tomasz Papszun;

- fix a potential security problem: don't let rmdir_recursively and
  rmdir_flat follow symbolic links; this might be exploited by attempting
  to delete some foreign file using privileges of the amavisd process
  (which should not be root);

- use cpio option -d when unpacking cpio archives;

- don't insert spam-related headers for passed mail to recipients with
  bypass_spam_checks, making them believe the mail was not spam-checked
  (as they are not expecting such headers anyway);

- text added to the banned mail sender notification template (at the end
  of file amavisd), explaining to the sender what happened and how to avoid
  the restriction; edit to will;

- amavis-milter.c program in the helper-progs subdirectory is now based
  on the most recent version 1.1.2.3.2.40 from the AMaViS CVS (maintained
  by Lars Hecking), but hacked a bit to make its options mostly compatible
  with the previous version. Start it with option -h to get current usage text.
  See helper-progs/README if using sendmail milter setup;

  Please revert to the old one if the .40 gives you trouble;
  both version are included in the helper-progs subdirectory.

- fix parsing of unrar info lines;

- consistency with other virus/banned logic: don't send recipient notification
  (reporting banned name) if mail contains both a virus and a banned name,
  but $warnvirusrecip is false; reported by Nathan G. Grennan and Urska Brinar;

- check for possible I/O errors when reading from SMTP socket,
  and distinguish error condition from normal TCP session teardown;

- do not redirect stderr to /dev/null when calling file(1). This way the
  diagnostics from file(1) will now at least be visible in the debug session;

- determine_file_types: make 'bzip compressed' pattern match older
  bzip format (v1) as well; thanks to Davaeron;

- sub run_command: explicitly close STDERR before reopening it; this way
  the reporting of possible problems in each operation would be separate,
  and it seems to avoid a rare problem (open STDERR '/dev/null' failing)
  reported by Sam Hart;

- don't attempt to do lookups in regexp table $viruses_that_fake_sender_re
  if it is undefined; reported by Chris Paul;

- remove existing X-Spam-Score along with other X-Spam* header fields
  if spam scanning is enabled;

- do not skip inserting 'X-Spam-Status: No, hits=- ... WHITELISTED'
  for whitelisted senders which caused SA to be bypassed - artificially
  assume score is -10 for the purpose of comparing it to tag_level;
  reported by Mike Vanecek;

- explicitly qualify wblist.rid field in $sql_select_white_black_list
  just in case someone has a field 'rid' in the other table;

- fix showing the value of LC_TYPE environment variable in the log,
  show LC_CTYPE as well;

- SAVI-perl: remove MIME option from the default set of options.
  Even with recent versions of Sophos SAVI there are cases where the
  library goes into a spin while trying to decode broken MIME message
  (same applies to Sophie - one may want to change its configuration);

- updated vfind entry in @av_scanners to work with the new version,
  changing '--vexit {}' to '--vexit {}/*'; thanks to Lowell Filak;

- updated kavdaemon entry in amavisd.conf; thanks to Michael Hall and
  Daniel Melanchthon;

- added option '-packed' to 'FRISK F-Prot Daemon' entry;
  thanks to Manfred Gloiber;

- NAI uvscan entry: commented pre/post actions in the entry to show 
  how to remove environment variable LD_PRELOAD after finishing,
  suggested by David Tilley; added option --mime, suggested by Max and
  Kevin W. Gagel; added note on how to treat password-protected files
  as viruses, by Seth Parker;

- added 'Dr.Web Daemon' entry which talks directly to Dr.Web daemon over
  its Unix socket, speeding up a single-file check more than 200 times;
  provided by Andrew I Baznikin;

- another entry for Symantec AntiVirus Scan Engine provided by Guido
  R. Rolon A.; I'm not sure which is which, check your documentation;

- added 'dumaru', 'parite', 'gibe' and some other virus names to
  $viruses_that_fake_sender_re;

- add examples to $viruses_that_fake_sender_re to show how to make
  a default result true, and only list exceptions;

- placed a comment in amavisd.conf pointing out the proper syntax
  for $hdrfrom_notify_* variables; thanks to Wouter de Jong;

- bump up the size of $sa_mail_body_size_limit to 150 kB in amavisd.conf

- documentation updates; fixed typos and spelling mistakes in the
  documentation files and in amavisd.conf comments;

- new documentation file README.protocol, specifying the new (and the old)
  protocol between helper programs and amavisd daemon, to be made available
  with the next major release. The description of the current (traditional)
  protocol was contributed by Stephane Lentz.


---------------------------------------------------------------------------
                                                            August 25, 2003
Patch: amavisd-new-20030616-p5

- fix 'Modification of non-creatable array value attempted' bug when 
  no 'Received' header field was present in an infected mail;
  reported by Paul Miner;

- caching of SQL lookups on white/blacklist was based on sender address only,
  instead of sender _and_ recipient. This could lead to white-/blacklisting
  of one recipient to affect other recipients of the same message;
  reported and debugged by Paul Gamble;

- added LDFLAGS to helper-progs/Makefile.in. It is needed at least on NetBSD.
  Patch by Julian C. Dunn (the NetBSD package maintainer for amavisd-new);

- more obvious logging of HOLD reason in sendmail/milter setup
  (reported by Pascal Martinez);

- $MAXLEVELS zero or undef should disable the limit according to docs,
  but was not honoured; reported by Rob Hutton;

- if notifications delivery encounters a temporary failure (4xx), propagate
  this status to the final result instead of only logging a warning;

- amavisd.conf:
  * VirusBuster entry changed to match newer version of the product;
    information from Marcus Schopen;
  * another entry in amavisd.conf for KasperskyLab kavscanner (v4.5?),
    contributed by Simone Marx;


---------------------------------------------------------------------------
                                                            August 12, 2003
Patch: amavisd-new-20030616-p4

- revert to using alarm() instead of Time::HiRes::alarm(). It is nonstandard
  to mix the two, and is causing problems on some operating systems
  (e.g. Solaris); thanks to Geoff Gibbs;

- rise log level for log entries on intentional mail drops in case a
  DSN (a bounce) should be sent, but will not be; such as on rejected
  bounces, viruses with forged names, and spam from mailing lists.
  The new log entries now say: '... Not sending DSN ...' and provide
  more information on the reason for dropping DSN;

- if sender is whitelisted, don't insert 'X-Spam-Flag: YES' header field,
  don't append spam address extension, and don't quarantine. This makes it
  less surprising, although previous behaviour was according to documentation.
  NOTE: the documentation still describes former behaviour, this needs
  to be fixed;

- report deaths of command line scanners and some external programs
  distinctly from normal exits with nonzero exit status;

- fix replacing * or {}/* in the pattern with actual file names, causing
  MkS_Vir (mks32), VFind, Dr.Web Daemon, and KasperskyLab aveclient to receive
  its file arguments glued together; based on patch by Rafael J. Wysocki;

- update 'Panda Antivirus for Linux' entry to work with new (and older)
  versions of pavcl; updated entry kindly provided by Panda Software;

- stop the timer after SMTP transaction is over to better behave in
  persistent SMTP/LMTP sessions; start the timer at the beginning of a
  SMTP transaction, in addition to restarting it when DATA mode is entered;

- sub mail_via_smtp_single: properly report SMTP response code when all
  recipients are rejected by MTA, instead of logging a 'mail_via_smtp:'
  without a value. The problem was commonly seen with the W32/Mimail-A
  virus which fakes an often invalid local sender address and gets rejected
  by Postfix outright; reported by Turgut Kalfaoglu;

- use SMTP response code 554 (instead of 550) for rejecting syntactically
  invalid header (according to rfc2476);

- add am_id to SMTP response code generated by one_response_for_all()
  to make it easier for MTA log to be correlated with amavisd-new log;
  some cosmetic improvements in the generated SMTP response text;

- added 'Return-Path:' in notifications to make it more obvious to see
  envelope address from reports;

- the 'Message-ID:' in neutral DSN notifications template was inadvertently
  pushed into the DSN body;

- report undefined spam score in X-Spam-Status header field as 'hits=-'
  instead of 'hits=0.0' which can be misleading;

- indicate blacklisting in X-Spam-Status header field of quarantined messages;

- add X-Envelope-From header field to quarantined messages;

- added virus names: tanatos, lentin, bridex  (alternative names
  for bugbear, yaha and braid) to the $viruses_that_fake_sender_re;
  thanks to Harrie Overdijk;

- set environment variables LINES and COLUMNS to sensible defaults
  to avoid some external program get puzzled about the terminal settings
  (e.g. older versions of pavcl from Panda Software);

- another attempt at fixing the Subject header field duplication. The patch
  amavisd-new-20030314-p2 fixed the case of entirely missing Subject header
  field, but did not fix the case of Subject header field present but with
  an empty text. Reported by Steven Cobb and Francois Rolland;

- rewritten 'Received' header fields parsing to better cope with valid,
  as well as with more common cases of broken syntax; used when trying
  to report originator IP address for believed-to-be-faked senders;

- more permissive parsing of SMTP addresses and options on MAIL FROM
  and RCPT TO commands;

- 'neutral' (=space) field in SQL black/whitelists now terminates the lookup
  search, avoiding fallback to static black/whitelists. It enables recipient
  to explicitly express its neutral stance towards the sender, overruling the
  site default;

- $sa_mail_body_size_limit now takes into account some portion of the
  mail header size so that huge mail headers that can cause slow SA calls
  are avoided (such degenerate cases were reported by Ralf Hildebrandt);

- taint fix in read_l10n_templates (as used by the Debian distribution),
  patch by Henrique de Moraes Holschuh;

- don't send recipient notifications to recipients that have
  bypass_virus_checks/bypass_banned_checks; suggested by Joe Breu;

- replace /bin/false with a more usual /usr/bin/false as a last resort exit;

- fetch modules 'Net::Ping' and 'bytes', which seem to be needed in certain
  chrooted setups;

- change log level from 0 to 1 for the log entry 'BAD HEADER from';
  reported by Thomas Lamy via Debian bug reports;
  same for 'unrar: all %d members are encrypted';

- fix typo in variable name: $spamassasin_obj -> $spamassassin_obj

- explicitly set pipes and sockets to binmode, as is a default since Perl 5.8.1

- documentation updates:
  * new file LDAP.schema, by Jacques Supcik, PhD
  * updated README.chroot to tell that /dev/urandom is needed
    in chroot jail (otherwise creation of MIME notifications fails);
    thanks to Lynn Duerksen and Jimmy Porter;
  * updated README.sendmail-dual, thanks to Robert LeBlanc and Stephane Lentz;
  * added URLs of external programs to INSTALL;
  * small updates to other doc files;


---------------------------------------------------------------------------
                                                              June 28, 2003
Patch: amavisd-new-20030616-p3

- avoid 'savemail: cannot save rejected email anywhere' sendmail panic when
  feeding mail via LMTP and using D_BOUNCE settings in dual-sendmail setup.


---------------------------------------------------------------------------
                                                              June 27, 2003
Patch: amavisd-new-20030616-p2

- when running chrooted, fix pre-loading of modules needed by SpamAssassin
  and Razor agents, which SpamAssassin forgets to pre-load by itself;
  reported by Neil Camara;

- updated KasperskyLab AVPDaemonClient entry to protect against
  carriage returns in collected virus names; thanks to Harrie Overdijk;

- updated 'Symantec CommandLineScanner' entry to look also for the new
  scanner name 'savsecls'; thanks to Guido Rolon;

- documented extended uses of read_hash() in the release notes;

- updated README.chroot;

---------------------------------------------------------------------------
                                                              June 24, 2003
Patch: amavisd-new-20030616-p1

- bug fix: allow stderr to be joined with stdout (>&1), instead of creating
  a file with a name "&1" when calling command-line scanners or doing
  LHA decoding (it got broken in 20030616); noticed by Henrik Larsson;

- bug fix: the X-Spam-Level header field would carry 64 asterisks instead
  of none when spam is configured to pass (mail tagging only) and when SA was
  not called (e.g. sender is whitelisted, ...) and tag_level or tag2_level
  is set to 0 or less; reported by Ralf Hildebrandt;

- bug fix: untaint e-mail addresses when forwarding via pipe;
  reported by Sam Tilders;

- modify code to match documentation: $relayhost_is_client should influence
  $notify_method too, not just $forward_method (adding an extra argument to
  mail_dispatch() was necessary to make this possible); reported by Zhu Yicun;

- SQL failure modes consistency: when initial connect to SQL failed, previous
  versions of amavisd-new would fall back to static defaults, disabling
  SQL lookups. Now initial connect failure is fatal, just like if connect
  failure occurred during operation - mail flow just stops during SQL servers
  outage, and is resumed when servers become reachable again;

- commented-out adding of 'X-Spam-Report' header field to retain behaviour
  from amavisd-new-20030314 - I find this header too long and intrusive
  for regular use. Uncomment it if adding X-Spam-Report is desired;

- support lzop (.lzo) and freeze (.F) compressors if available;

- use external program cpio (if available) to efficiently and safely handle
  the following archive formats: cpio binary, HPUX binary cpio, cpio crc,
    old ASCII cpio, new ASCII cpio, and HPUX old ASCII cpio, as well as
    POSIX.1 tar (also GNU tar) and old tar if allowed (see below)
  without reading whole archive members into memory;

- if your cpio(1L) can read tar format (as is common on FreeBSD and Linux),
  it is recommended to uncomment the following line (in file 'amavisd'):

    # /^\.tar$/  && defined $cpio && return do_cpio($part,$tempdir);

  which will cause tar archives to be safely and efficiently decoded
  by cpio(1L) instead of Archive::Tar, which loads the whole archive
  into memory;

- use separate file(1) 'classifications' ".uue" and ".hqx" (instead of ".asc")
  for uuencoded and binhex formats;

- merge 'H+BEDV AntiVir' and 'CentralCommand Vexira Antivirus'
  entries into a single entry - they are basically the same product
  (the Vexira entry didn't work for new version of Vexira);
  thanks to Vivek Khera;

- do_unrar: some switches to rar/unrar are not recognized by older versions
  of rar/unrar; issue a more informative log when this happens, and retry
  without newer switches for compatibility; thanks to Geoff Gibbs;

- crop backtrace on SA timeouts at 980 characters;

- preload Perl modules Net::DNS::* if spam scanning is enabled; they could not
  be loaded if running in a chroot jail, so SpamAssassin did not use Razor;

- documentation: collected documentation pieces pertaining to LDAP
  into README.lookups; fix SQL example table data in README.lookups;

( the patched MIME-tools by David F. Skoll is recommended over 5.411,
  as it better handles broken/bad MIME syntax:
  http://www.mimedefang.org/ -> Download section )


---------------------------------------------------------------------------
                                                              June 16, 2003
amavisd-new-20030616 release notes

This is mostly a maintenance release. Only a handful of new features
were added that were small or easy enough to provide.


- revised the incoming SMTP session abort procedure to properly shut down
  the remote SMTP client in case of unexpected trouble (e.g. disk full)
  during SMTP data reception;

- subroutine mail_via_smtp_single() overhaul to produce cleaner diagnostics
  and to properly abort outgoing SMTP session in case timeout occurs
  in SMTP 'DATA' mode (avoids recipients repeatedly receiving partial
  message in successive delivery attempts when the receiving MTA responds
  very slowly); problem reported by Scott Vintinner;

- fixed the generation of LMTP response code when D_BOUNCE was chosen,
  but DSN was not actually sent because sender was thought to be faked
  or mail came in from a mailing list. Since DSN was not really sent,
  the final response remained at 5xx (causing bounce by MTA) instead of
  being converted to 2xx as D_BOUNCE would suggests; reported by Peter Bates;

- security: avoid inadvertently untainting values (e.g. mail addresses)
  at several places throughout the program; 

- work around a Perl 5.8.0 taint bug where global variable $1 could become
  tainted, making further untainting attempts unsuccessful;

- work around another Perl 5.8.0 taint bug where Encode::encode fills up
  all available memory when given a tainted string with a non-encodable
  character. New subroutine safe_encode() provides a wrapper around
  the Encode::encode;

- treat 'Maximum number of files exceeded' the same as 'Exceeded storage
  quota', i.e. inserting the X-Amavis-Hold header field; noted by
  Christopher Odenbach;

- call exec with explicitly specified program path, preventing exec
  from even considering calling a shell;

- catch and better report the Perl's failure to fork on open;

- avoid inappropriately reporting 'Illegal seek' in run_av (and elsewhere)
  even though nothing went wrong; reported by William Yodlowsky;

- new subroutine flatten_and_tidy_dir() recursively descends into a directory
  containing potentially unsafe files with unpredictable names, soft links,
  etc., rename each regular nonempty file giving it a generated name, and
  discard all the rest. 

- do_unrar and do_unarj now let the archiver itself recursively unpack
  the archive and then use flatten_and_tidy_dir() to tidy up the result.
  With previous method it was not possible to extract archive members
  with names containing non-ASCII characters from rar archives due to
  a bug in unrar and rar programs. Problem reported by Pan Bambaryla.
  Protection from mail bombs in do_unrar is retained.

- do_unrar: use program rar or unrar, whichever is available. Both
  rar and unrar recognize the same options to extract. (amavisd.conf:
  $unrar=['rar','unrar'] ). Under some circumstances unrar falls in a
  loop while rar extracts the archive correctly.

- do_unrar: don't bail out on exit status 1 (warning) from unrar after
  collecting file names from the archive. Be more careful when extracting
  file names, such as archive members with names starting with '-' or spaces;
  Disable showing archive comment and file comments which can easily
  break member-name parsing. Avoid extracting files in subdirectories
  twice: once when directory itself is listed, and the second time when
  each file in a directory is listed. Licence note: unrar is free,
  rar is not ( see http://www.rarlab.com/ for fresh unrar sources ).

- do_unarj: prefer more versatile arj over unarj (amavisd.conf:
  $unarj=['arj','unarj'] ). Both programs support subcommand 'e' that we use;
  suggested by Guillem Jover (via Debian support);  ARJ since 2.78/3.10
  is Open Source (GPL license). When arj is available (recognizing several
  new archive types, archive versions and options), put these options
  to good use, recovering several archive members that are unaccessible
  by the demo program unarj, such as old member versions, memebers with
  equal names, etc.  Disable showing comments.

- add X-Virus-Scanned header only if a mail was actually scanned
  for viruses (i.e. av scan not bypassed for this set of recipients);
  pointed out by Phil Regnauld;

- updated entry for 'KasperskyLab AvpTeamDream' virus scanner;
  by Daniel Melanchthon;

- truncate instead of round the spam level when producing the bar
  in the X-Spam-Level header field for compatibility with SA.
  Clip the level bar at 64 characters instead of 60 (more would not
  be allowed by RFC 2822, although SA used to crop at 100);

- subroutine hdr() incorrectly fixed an illegal header field body
  which did not have a space after a line fold; noticed by Virna Gupta;
  (the buglet did not show unless one modified the program);

- replace several remaining regular expressions /...$/ with /...$(?!\n)/
  as a matter of principle (as was done in amavisd-new-20030314-p2 for the
  more urgent cases). The use of simple $ was almost always subtly
  semantically wrong, but fortunately in most cases without consequences;

- specify open mode explicitly in several 'open's, and protect special
  characters in file names from being 'too cleverly' interpreted by
  Perl open (still not all cases done);

- SpamAssassin changes umask to 0077 - restore our mask after the SA call;

- be a bit more careful regarding platform-independency, distinguishing
  between use of \012 and \015 against \r and \n (note the \n on Mac is CR);

- change log level of several less important av scanner log messages
  from 2 to 3;

- small documentation updates in amavisd,conf;

- updated README.lookups: the new substring capture and reference mechanism
  in regexp lookups explained. The SQL example modified to work under
  PostgreSQL, thanks to Phil Regnauld;

- updated documentation: README.sendmail-dual, README.postfix, README.chroot,
  new file: README.exim_v4_app2 - by Louis Erickson;


NEW FEATURES

- provide a command line option 'debug-sa', which is equivalent
  (but more convenient) then setting $sa_debug to true and starting
  as '# amavisd foreground';

- list version numbers of the more important Perl modules at startup;

- lookup_re(): RE pattern now allows for capturing of parenthesized substrings,
  which can then be referenced from the result string using the $1, $2, ...
  notation, as with the Perl m// operator. The number after the $ may be
  a multi-digit number. To avoid possible ambiguity the ${n} or $(n) form
  may be used. Substring numbering starts with 1. Nonexistent references
  evaluate to empty strings. If any substitution is done, the result inherits
  the taintedness of the key. Keep in mind that the $ character needs to be
  backslash-quoted in qq() strings (but not in q() strings). Example:
    $virus_quarantine_to = new_RE(
      [ qr'^(.*)@example\.com$'i => 'virus-${1}@example.com' ],
      [ qr'^(.*)(@[^@]*)?$'i     => 'virus-${1}${2}' ] );

- read_hash() now returns the hashref (first argument) also as a return
  value, making it easier to create new hashes (instead of only adding values
  to existing ones). Also the first argument (hashref) is now optional,
  and if missing a new hash is created. The following three cases are now
  equivalent:

  a) $sa_tag2_level_deflt = {};
     read_hash($sa_tag2_level_deflt, '/var/amavis/tag2_levels.dat');

  b) $sa_tag2_level_deflt = read_hash({}, '/var/amavis/tag2_levels.dat');

  c) $sa_tag2_level_deflt = read_hash('/var/amavis/tag2_levels.dat');

- extend the semantics of the configuration variable $inet_socket_port,
  which can now either be a scalar as before (i.e. a single port number),
  or a ref to a list of port numbers, e.g:
    $inet_socket_port=[10024,10026,10028]; # accept SMTP on all these TCP port

  When $relayhost_is_client is true, the semantics of $forward_method has
  changed slightly: instead of taking the port number from $forward_method,
  it is now calculated as being one higher than the port number on which
  the incoming SMTP connection came in (one from the $inet_socket_port list).

  This allows for multiple MTA pairs on the same host to share the same
  amavisd daemon without having to use multiple loopback interfaces.
  Useful for example if incoming and outgoing mail is handled by separate
  mailers on the same host.

- add field 'spam_quarantine_to' to SQL table 'policy' - a patch
  provided by Vivek Khera. For compatibility a missing field defaults
  to undef, causing lookup search to proceed with the next lookup map;

- add fields: spam_lover, banned_files_lover, bad_header_lover,
  bypass_banned_checks and bypass_header_checks to SQL table 'policy'
  for completeness;  For compatibility a missing field defaults to undef,
  causing lookup search to proceed with the next lookup map;

- add a 'SPAM-TAG, ...' log entry at log level 2, which is produced when
  inserting spam-related header fields for each cluster of recipients
  with same settings. It complements the 'SPAM, ...' log entry, which
  is triggered at kill level. Feel free to adjust their log levels
  if you think one or the other is redundant;

- insert a 'X-Spam-Report' header field at tag2 level when spam is passed.
  Feel free to comment out the ...append_header('X-Spam-Report'... line
  if this is undesired.

- fold-in the 'bad_headers.patch' to the main code
  (available in the distribution since amavisd-new-20030314-p1),
  but renamed *bad_headers_lovers* to *bad_header_lovers* and added
  $final_bad_header_destiny and *bypass_header_checks*.

  To refresh memory:

  Enables checking headers of each mail for invalid (non-encoded) 8-bit
  characters, and produces a bounce (non-delivery status notification,
  or a SMTP REJECT if desired) with the full explanation of the problem,
  with offending header fields trimmed, sanitized and included in the text.

  New setting $final_bad_header_destiny, defaults to D_PASS for backwards
  compatibility, but a value D_BOUNCE is suggested. Similarly to other
  *lovers* settings, a hash and an ACL lookup %bad_header_lovers and
  \@bad_header_lovers_acl are available, and a setting $warnbannedsender.

  Certain recipients may be exempt from the checking (*bad_header_lovers*).
  Similarly mail from mailing lists (Precedence: list or bulk), and mail
  with null reverse-path mail (e.g. bounces) is passed, even if violating
  the RFC 2822 header syntax. A log entry is produced nevertheless.

  Postfix users: this is similar to the Postfix strict_7bit_headers=yes
  functionality, but produces a much more informative problem report
  (non-delivery notification) to the sender. One difference is that
  amavisd-new header check takes into account only the RFC 2822 header,
  not MIME headers in the mail body. It is reasonably efficient to use
  amavisd-new as a header checker only, without any anti-virus or anti-spam
  checks, if desired.

- added configuration settings $warnbannedsender and $warnbannedrecip,
  separating this function from $warnvirussender and $warnvirusrecip,
  which previously applied to both virus and banned files;

- new macro %c which evaluates to spam level/hits (mnemonic: sCore)
  as provided by SpamAssassin; useful in $log_templ;

- new configuration settings *bypass_banned_checks* and *bypass_header_checks*;

- new parameter $spam_quarantine_bysender_to (in contrast to the usual
  $spam_quarantine_to) makes possible collecting quarantined spam
  by sender names or domains, instead of (or in addition to) the more
  usual by-recipient map;

- new configuration setting $remove_existing_spam_headers (defaults to true);

- bring-in the localization (l10n) support contribution from Debian Linux,
  by Henrique de Moraes Holschuh; see README.l10n, and comments at the
  call to read_l10n_templates() in file amavisd.conf;

- supply X-Envelope-From and X-Envelope-To header fields containing envelope
  addresses to the messaged passed to SpamAssassin for checking;

- there is now an official OID: 1.3.6.1.4.1.15312.2.2
  under which LDAP schema for amavisd-new LDAP lookups is to be defined
  (iso.org.dod.internet.private.enterprise. 'ijs.amavisd-new.ldap' );


---------------------------------------------------------------------------
                                                                May 6, 2003
Patch: amavisd-new-20030314-p2

- blacklisted sender only caused spam headers to be inserted,
  but did not cause rejecting the message and other 'evasive actions'
  due to over-optimization; reported by Lawrence Farr, Robin Elfrink
  and Eric Vollmer;

- fix improper evaluation of bypass_spam_checks* in multi-recipient mail
  where some recipients match 'bypass_spam_checks*' and others do not.
  Spam-check was bypassed if _any_ (instead of _all_) recipient matched
  'bypass_spam_checks*'; thanks to Joseph W. Breu;
  (it got broken in amavisd-new-20030314, the amavisd-new-20021227 was ok);

- adding address extensions was not done for *_lovers, only for
  *_destiny=D_PASS; reported by Steve Khoo. The "deal_with_spam" section
  needed to be overhauled for this reason. For consistency with the
  documentation and with the change in adding address extensions,
  quarantining is now done if mail is considered spam, regardless of
  it being delivered (e.g. to spam lovers), or not;

- if only spam checking is performed, but no virus scanning nor banned
  file names checking is enabled, and the spam is detected but configured
  to pass, and the Subject is configured to be edited, the Subject header
  field was duplicated because the existing Subject header field was not
  detected.

  Solution: make MIME entity available even if virus and banned files
  scanning is not performed. Header information is needed by
  add_forwarding_header_edits_per_recip() when deciding whether to add
  or replace the Subject header field in case spam is passed.

  (workaround for previous versions: don't leave $banned_filename_re
  undefined. If not needed, set it to an empty list: new_RE() );

- when quarantining to a mbox file, excape 'From ' with '>' only if it
  follows an empty line, as sendmail mail.local man page prescribes;

- added parameter $notify_xmailer_header to control the body of the header
  field X-Mailer as placed in the notification messages (e.g. bounces)
  generated by amavisd-new; by default this header is no longer inserted,
  as it was felt it revealed too much information. Choose from:
    $notify_xmailer_header = undef;  # no X-Mailer inserted (default)
    $notify_xmailer_header = '';     # X-Mailer: MIME-tools x.xxx (Entity x.xxx)
    $notify_xmailer_header = 'your-text-here';  # X-Mailer: your-text-here

- replace regular expressions /...$/ with /...$(?!\n)/ or with string
  equality operator in some places (many more to come in the next major
  release); the use of simple $ was almost always subtly semantically
  wrong, but in most cases without consequences;

- enable timeouts after SMTP connection has been established, to be able
  to handle cases where MTA is very slow in accepting checked mail,
  which could result in multiple deliveries if MTA on the input side
  times out;

- debugging aids: call stack backtrace is logged if
  Mail::SpamAssassin::NoMailAudit::check exceeds allowed time;

- LDAP enhancement: allow "hostname" to be a string or a reference to an
  array, in which case each entry will be tried in order until a connection
  is made; by Jacques Supcik;

- X-Spam-Status now says WHITELISTED if sender was whitelisted;
  contributed by Joseph W. Breu;

- if mail rejection was due to sender blacklisting, say so in the SMTP
  response; thanks to Robin Elfrink;

- updated file bad_headers.patch to apply cleanly against this version.
  The bad_headers.patch will be folded-in with the next major release,
  please try it out, comments welcome. For patch description see release
  notes for amavisd-new-20030314-p1;

- specify option 'ExactAddresses=>1' in a call to Net::SMTP::new()
  in anticipation for the next release of Net::SMTP;

- amavisd.conf: updated documentation / comments;

- amavisd.conf: removed option '-ni' in the DrWeb virus scanner entry;
  thanks to Mike Boev and the FreeBSD community
  ( http://www.freebsd.org/cgi/query-pr.cgi?pr=50893 )

- amavisd.conf: added entry for 'MkS_Vir daemon', thanks to Dariusz Grzegorski;

- amavisd.conf: updated entry for 'F-Secure Antivirus' to cope with fsav 4.50
  as well as with 4.1x; solution by Juhan Tamsalu;

- documentation: updated README.chroot; new README.sendmail describing
  dual-MTA sendmail setup, small updates to other doc files;


                                                             March 21, 2003
Patch: amavisd-new-20030314-p1

- fix passing of remote MTA reject code back to the input side
  (DSN was not sent if MTA rejected the message);

- fix broken logic which allowed a virus to pass if a recipient was a
  'banned file lover' and the same message also contained a 'banned file';
  (or the other way around: pass a banned file if containing a virus and sent
  to virus lovers); thanks to Ortwin Gentz, I'm glad somebody is looking
  over my shoulder;

- remove a log entry:  one_response_for_all: SHOULDN'T HAPPEN
  The test was inappropriate, but harmless;

- enhanced sub run_av() to fix a problem with MkS_Vir virus scanner,
  where a '*' in the template command was not expanded to actual
  file names; reported by Pan Bambaryla;

- trim virus scanner output (in sub run_av) to a manageable size
  of 900 characters so as not to clutter log entries and notifications
  too much; provoked by the Panda pavcl;

- move the check for 'Precedence:' mail header field from
  sub delivery_status_notification() to its caller for flexibility;

- replaced the text in notification templates:
    The message WAS delivered to:
  into:
    The message WILL BE delivered to:
  to reflect the truth. The message may still be rejected
  by the MTA when forwarding takes place.

- small documentations edits/updates;

- cosmetic: trim trailing whitespace in the program;

- cosmetic: make shorthand subroutine nf() locally scoped; by Jacques Supcik;

- make available an experimental patch (file: bad_headers.patch,
  not applied by default), which, when applied (with patch(1) utility)
  to amavisd-new, makes it check headers of each mail for invalid
  (non-encoded) 8-bit characters, and produces a bounce (non-delivery
  status notification, or a SMTP REJECT if desired) with the full
  explanation of the problem, with offending header fields trimmed,
  sanitized and included in the text.

  Certain recipients may be exempt from the checking (*bad_header_lovers*).
  Similarly mail from mailing lists (Precedence: list or bulk), and mail
  with null reverse-path mail (e.g. bounces) is passed, even if violating
  the RFC 2822 header syntax. A log entry is produced nevertheless.

  Postfix users: this is similar to the Postfix strict_7bit_headers=yes
  functionality, but produces a much more informative problem report
  (non-delivery notification) to the sender. One difference is that
  amavisd-new header check takes into account only the RFC 2822 header,
  not MIME headers in the mail body. It is reasonably efficient to use
  amavisd-new as a header checker only, without any anti-virus or anti-spam
  checks, if desired.

  The patch is experimental in the sense that it may change in future
  versions, but is fully functional when applied;

- Jacques Supcik cleaned up and generalized his LDAP lookups
  support code (which is off by default). In his words:

  The major change is in the definition of ldap lookups.
  In the config file, you have to enable ldap first:

  | $enable_ldap = 1;

  Then you can define defaults for your ldap queries:

  | $default_ldap = {
  |   hostname => 'localhost', tls => 0,
  |   base => 'ou=hosting,dc=example,dc=com', scope => 'sub',
  |   query_filter => '(&(objectClass=amavisAccount)(mail=%m))'
  | };

  And then the lookups themselves:

  | $virus_lovers_ldap = {res_attr => 'amavisVirusLover'};
  | $banned_files_lovers_ldap = {res_attr => 'amavisBannedFilesLover'};
  | ...

  With this method, you can define every parameter individually.
  You could have a different ldap server for each lookup! Like that,
  the configuration is closer to the one in postfix or courier.

  The hashes are converted into lookups objects in the
  Amavis::Lookup::LDAP class method.


---------------------------------------------------------------------------
                                                             March 14, 2003
amavisd-new-20030314 release notes


MAIN NEW FEATURES AT A GLANCE

- per-user white- and blacklisting, including a SQL lookup mechanism;
  refined semantics for white/blacklists;

- easier to run in chroot jail, see README.chroot;

- can specify two sets of virus scanners: the main list and a backup list,
  which gets consulted only if all scanners from the main list fail;

- split REJECT into separate destiny settings: REJECT and BOUNCE,
  giving you a choice of who is responsible for sending a non-delivery
  notifications to the sender; see comments in amavisd.conf;

- edited configure file amavisd.conf, providing more informative comments,
  cleaner examples and simplified common settings;

- made it work better with the Unicode-aware Perl 5.8.x (not against it)
  and distinguish between character data / byte data / octet data (binary),
  provide UTF-8 (Unicode) support for notification messages,
  while avoiding some possible nasty surprises in the UTF-8 locale
  (which is enabled by default in Red Hat 8.0);

- SQL SELECT clause settable from the config file;

- can store quarantined or forwarded messages as BSMTP files;



Changes since amavisd-new-20021227 patch level 2

BUG FIXES

- fixed caching problem when bypass_*_checks are used selectively: when only
  one of the virus/spam tests were performed but the other bypassed due to
  a per-recipient setting, the result of a non-performed test was inserted
  into cache as clean. This could cause tests to be bypassed for subsequent
  message with the same body but different recipients with different bypass*
  settings, but only within a lifetime of a child process (10 messages
  by default). Noticed by Jürgen Louis Fluk, thanks!

- changed caching of SQL lookups to match the documented behaviour,
  i.e. SQL lookup result is cached (for the benefit of field lookups)
  for a single message only, and no longer for the child lifetime.
  It is possible to revert to the previous behaviour by commenting-out
  one line (look for $sql->clear_cache). Pointed out by Paul Miner;

- see also further down about the field 'banned_files_lover'
  in the SQL table 'policy', and the UTF-8 (Unicode) workarounds.


SECURITY

- running amavisd-new in chroot jail is now easier to set up due to added
  configuration variable $MYHOME and its simplification effects on other
  settings in the config file. Relevant settings are better commented;

- avoid the need to have a shell in chroot jail (use of the new subroutine
  run_command() instead of Perl functions qx() and system() );

- new file README.chroot with guidelines on how to run amavisd-new
  in a chroot jail, to prevent possible vulnerabilities from affecting
  the rest of your system;

- added a security notice (web page and other places) that the Unix utility
  file(1) 3.39 and older contains an exploitable security vulnerability,
  which can cause system access with access rights of the user running
  amavisd-new daemon (not root, unless you violated the recommendations).
  Please upgrade file(1) to 3.41 or newer:

     ftp://ftp.astron.com/pub/file/file-3.41.tar.gz

- new section (Security considerations) on the web page;


CLEANUPS AND CHANGES IN amavisd.conf

- improve explanation texts (comments) in amavisd.conf,
  edit the file to improve legibility and structure, change some examples.
  The daemon is thought to be compatible with amavisd.conf from previous
  version, but it may be worthwhile to switch to the new file to avoid
  asking FAQ;

- introduce a configuration variable $MYHOME, which is never used directly
  by amavisd daemon, but is a handy way of setting default values for
  other configurable variables in the config file. Also makes default
  settings more chroot-friendly.

- introduce a configuration variable $mydomain, which is never used directly
  by amavisd daemon, but is a handy way of setting default values for
  other configurable variables when there is only a single domain (with
  subdomains) behind the mailer. This avoids the need to edit half-a-dozen
  configuration variables in simple setups;

- moved debugging-related settings to a separate section
  (Section VIII - Debugging);

- make SQL SELECT clause settable from the config file;

- renamed variable @local_domains to @local_domains_acl for consistency;
  The old name became a synonym for @local_domains_acl, so compatibility
  with existing config files is retained;

- comment out the example: read_hash(\%whitelist_sender, ...);
  it slipped into the amavisd-new-20021227 released code from my tests;

- comment out the setting $relayhost_is_client=1;  It is only useful
  for multi-access daemon setup, and potentially confusing for others;


VIRUS SCANNERS

- introduced a list of backup virus scanners: @av_scanners_backup,
  complementing the normal list of virus scanners.

  If no virus scanners from the @av_scanners list produce 'clean' nor
  'infected' status (e.g. they all fail to run or the list is empty),
  then all scanners in the @av_scanners_backup list are tried.
  When there are both daemonized and command-line scanners available,
  it is customary to place slower command-line scanners in the
  @av_scanners_backup list;

- removed the Mac and SafeMacDfHandling options from SAVI-Perl,
  as they are no longer supported by Sophos library since 2003-03;
  set new option NamespaceSupport to SOPHOS_DOS_AND_MAC_FILES;
  thanks to Paul B. Henson;

- changed a default to initialize SAVI-Perl only at startup time,
  and no longer every time the child process is started. This is because
  starting with SAVI 3.0 (March 2003) the initialization is slower.

  NOTE: it is now necessary to restart amavisd master process
  after SAVI database is updated, to make it notice new virus descriptions.

- in 'MkS_Vir for Linux' entry replaced options -e -c with -s,
  to make mks work, suggested by Pan Bambaryla;

- added status 8 ('corrupted') to the list of clean statuses for
  'KasperskyLab AVPDaemonClient', as is already the case for some other
  virus scanners. See amavisd-new web page (Security considerations)
  if the idea bothers you;

- RAV command line scanner no longer supported off-the-shelf,
  as interfacing to it seems to be violating their license terms.

  Btw, if RAV command line scanner does not work as user amavis,
  check permissions and ownerships of /usr/local/rav8/* or reinstall
  RAV as the user that will run amavisd; thanks to Oyku Gencay;

- RAV does not like '</dev/null' (segfault) (reported by Johnny Ljunggren),
  most others don't care, but possibly Panda needs it. Tried to remove it,
  but later put is back, as I don't like external command to have access
  to the amavisd socket over which SMTP or AM.CL session is running;

- H+BEDV AntiVir: changed option -allfiles to --allfiles;
  thanks to Dr Patrick Atlas, confirmed in the H+BEDV documentation;

- comment out the default entry for Sophos sweep, as the name 'sweep'
  clashes with the Debian sweep package (audio samples editor).
  If you uncomment the entry, make sure the correct 'sweep' is found
  in the specified path; thanks to Guido Guenther and Brian May;

- added av definition for BitDefender (based on amavis-ng code by
  Hilko Bengen); thanks to Stephane Lentz for testing and suggesting
  command line options;

- added av definition for Ikarus AntiVirus for Linux, based on
  amavisd-0.1 code by Lars Hecking;


PERL 5.8 UNICODE AND INTERNATIONALIZATION (I18N)

- make program aware of the type of data coming-in from external files,
  pipes and sockets - distinguish between binary data, byte data, and
  character data / plain text.

  This is necessary in order to be able to survive in the UTF-8 locale-aware
  Perl 5.8.0 environment (e.g. to avoid 'Malformed UTF-8 character'
  and 'Wide character in print' warnings/errors, among others).

- check character ranges in regular expressions throughout the program,
  and fix/rewrite them to avoid assumption that all characters are in the
  range 0..255, which is no longer true! Some nasty surprises could have
  been lurking in that corner.

  Note the Perl 5.8.0 terminology (see perluniintro and Encode man page):
  * character: a character in the range 0..(2**32-1) (or more)
    (what Perl's strings are made of);
  * byte: a character in the range 0..255
    (a special case of a Perl character);
  * octet: 8 bits of data, with ordinal values 0..255
    (term for bytes passed to or from a non-Perl context, e.g. a disk file)

- modified sanitize_str() and hdr() to cope with UTF-8 characters;

- modified string_to_mime_entity() and hdr() to encode RFC2047-permitted
  headers according to $hdr_encoding, and notification body according
  to $bdy_encoding settings (both default to iso-88591-1, aka Latin1).
  Notification template texts are supposed to be stored internally as
  native or UTF-8 Perl strings, which is handled (mostly) transparently
  by Perl 5.8.x;

- added option $hdr_encoding_qb to be able to explicitly specify
  MIME encoding ("Q" or "B") for $hdr_encoding MIME charset;

- rfc2822_timestamp: produce correct date format regardless of
  LC_TIME locale setting;

- read_text() subroutine (which may be used in amavisd.conf to read
  notification text templates) may be given a second argument, specifying
  character encoding used in the external file. If Perl is Unicode-aware
  (5.8.0), this argument is given to binmode to influence the reading
  of the file and its conversion into internal Perl string representation.

  When notification text is generated from the internal Perl string
  representations (UTF-8), the following settings influence the character
  encoding (charset in MIME terminology):

    $hdr_encoding = 'iso-8859-1';  # ... header field bodies

  The header is first encoded into $hdr_encoding, which is then encoded
  into 7-bit ASCII as required by RFC 2821 and defined by RFC 2047.
  The specified encoding (charset) will appear in the RFC 2047 Q-encoded
  strings in header fields where this is allowed, i.e. in Subject,
  Comments, and X-* header fields. In other header fields where RFC 2047
  encoding is not allowed, encoding into plain ASCII is used, where invalid
  characters are treated as Perl Encode module does it, i.e. by converting
  them to '?'.

    $bdy_encoding = 'iso-8859-1';  # ... notification body text

  Notification body text is encoded from the internal Perl representation
  according to $bdy_encoding. The encoding (charset) is also inserted
  in MIME header as Content-Type.charset.



OTHER

- make it work with Net::Server version 0.85, which changed the handling
  of the CHLD signal, this is breaking qx and system calls used by previous
  versions of amavisd-new;

- with Net::Server 0.85 you may see a message:
    Net::Server: Couldn't POSIX::setuid to ... []
  If the process UID remained 0 (root), the program will terminate,
  otherwise just consider the message harmless (or complain to the
  author of Net::Server);

- temporarily removed utility amavisdconf, it is not yet updated to cope
  with the new configuration variables;

- determine_file_types: 'script text executable' is now considered .txt,
  not .exe; thanks to Stephane Lentz and Michael Dengler;

- report correct system error if Convert::UUlib::decode fails
  (e.g. permission denied); thanks to Nick Murtagh for figuring out
  that UUDecode() in uunconc.c tries to decode into a temporary file
  in /root/tmp/whatever;

- avoid the need for a Unix utility file(1) to support option -b;
  suggested by Clifton Royston considering OpenBSD;

- new macro %b for customized notifications and log: it evaluates to
  the message digest of mail body (MD5, hex).  Used as a caching key,
  can serve to identify messages in quarantine with same body text.

- provide a new BSMTP output method that can be used as $forward_method,
  $notify_method, $virus_quarantine_method or $spam_quarantine_method.
  It is simple in a sense that it generates one file for each message,
  so it is not terribly efficient. The parameter after the colon is used
  as a file name prototype, where %i in the prototype name is replaced
  by current time, %n by amavis internal message id, and %b by message
  digest of the body (MD5,hex). For example:

    $forward_method = "bsmtp:$TEMPBASE/out-%i-%n.bsmtp";
    $spam_quarantine_method = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp";

  The bsmtp file has ".tmp" appended to a name during writing, and is
  renamed to the final name when it is done, to avoid being seen by possible
  fetcher prematurely. It is experimental in a sense that configuration
  and implementation details may change in the future, including possible
  changes in the old 'local:' method;

- besides the default (implied) quarantine method 'local:', it is now possible
  to specify 'bsmtp:...' as a quarantine method, and have different settings
  for virus and spam quarantines. New settings: $virus_quarantine_method
  and $spam_quarantine_method, e.g.:
    $virus_quarantine_method = "bsmtp:$QUARANTINEDIR/virus-%i-%n.bsmtp";
    $spam_quarantine_method  = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp";

- if ***SPAM*** is to be inserted into Subject header field,
  but the original mail does not contain the Subject field, insert one;
  suggested by Christopher Odenbach;

- per-recipient SQL field and static flag (spam_modifies_subj) for modifying
  Subject: header field when spam is passed;

- updated README.lookups, describing new SQL fields and tables;

- split kill_level setting into two independent settings: tag2_level,
  and kill_level. A quick reference:
    tag_level  controls adding the X-Spam-Status and X-Spam-Level headers,
    tag2_level controls adding 'X-Spam-Flag: YES', and editing Subject,
    kill_level controls 'evasive actions' (reject, quarantine, extensions);
  The default is tag2_level=kill_level, which retains compatibility.

- work around a bug in Perl 5.8.0 in UTF-8 environment, which could
  produce error "Bad RFC822 field name 'C0c0\x{0}\x{0}T-Type'" when building
  notifications (the Perl bug is triggered in Mail::Header::_tag_case
  by statement $tag =~ s/\b([a-z]+)/\L\u$1/gio; reported on several
  Red Hat 8.0 hosts, seemingly only on Intel platforms).

  Many thanks to Chris Kloiber and Steve Sloan for making available
  their test machines for debugging.

- replace all calls to qx by run_command, to avoid 'Malformed UTF-8 character'
  when reading from external utilities that do not honour locale settings,
  e.g. Unix file(1) utility, which may return non-printable characters with
  codes above 127. This also ensures better error reporting.
  run_command() moved into Util module, part of its code moved into a
  new subroutine run_command_copy().

- improved diagnostics when certain external programs fail to run,
  such as when fork or exec fails;

- new LDAP lookups (experimental), by Jacques Supcik, PhD,
  IP-Plus Internet Services - Swisscom Enterprise Solutions Ltd,
  Genfergasse 14, 3050 Bern, Switzerland (http://www.ip-plus.net/)
  Many thanks for the contribution!

- changed enhanced status code 5.7.0 to 5.7.1 when message is considered
  bad (virus, banned file, spam);

- when sending message via SMTP, don't go into DATA part if any RCPT TO
  returned 4xx code, since without implementing proper queueing we wouldn't
  be able to handle the situation unless message came in via LMTP;

- updated README.postfix to describe:
  * the now recommended Postfix 2.0 setup with pre-cleanup service,
    instead of the (post-)cleanup2;
  * how to specify a backup content filter(s) in case the primary fails;
  * the Postfix setting if you want X-Amavis-Hold header field to cause
    message to be placed on hold by Postfix, e.g. when amavisd-new encounters
    a mail bomb which can not be checked for viruses.

- unconditionally delete old headers X-Spam-Status, X-Spam-Level,
  X-Spam-Flag and X-Spam-Report, when anti-spam code is enabled.
  Previously deletion was done only when new headers were to be inserted.

- replace sloppy file accessibility tests (Perl functions -e, -f, -d, -r, ...)
  with more reliable method, which also produces more informative
  diagnostics;

- set $SIG{CHLD}='DEFAULT' in Net::Server hooks to make qx work
  in Net::Server::Single environment (useful for debugging),
  as well as with the version of 0.85 of Net::Server;

- adjust the log level in do_ascii() (reported by Pan Bambaryla),
  and in few other places;

- fix $log_templ - an unconditional comma was causing the log entry to be
  always issued, regardless of mail content; reported by Stephane Lentz;

- enhance reject/discard/pass destinies with reject/bounce/discard/pass,
  making it more flexible to specify what action to perform on detection
  of bad content; (thanks again to Chris Hastie for earlier discussion
  on the topic); see explanations in amavisd.conf;

- added hash lookup table %local_domains for those that have a great number
  of hosted domains and prefer to read a list from a separate file;

- a special shorthand provision when SQL lookups are used: when a match
  for recipient address or domain is found in SQL tables (regardless of
  field values), the recipient is considered local, regardless of static
  @local_comains_acl or %local_domains lookup tables. This simplifies
  life when a large number of dynamically changing domains is hosted.
  To overrule this behaviour, add an explicit boolean field 'local'
  to table 'users' (missing field defaults to true, meaning record presence
  implies locality).

- changed SQL table 'policy' field name 'banned_files_lover' to a name
  'banned_files_lovers' (added an 's') for consistency with variables
  %banned_files_lovers and @banned_files_lovers_acl, and updated
  README.lookups;

- SQL field treated as boolean now recognize '\0' character as false
  (not to be confused with NULL SQL field, which is still mapped to undef);

- when generating Received: header field, move receiving port number
  into the second comment field - some software were too picky
  about the syntax of the first comment field ('TCP-info' in rfc2821);

- consistency: apply late 'string sanitation'; meaning that strings
  such as virus names, banned attachment names, e-mail addresses, etc,
  are sanitized at the point where the string is leaving amavisd-new,
  not at the point where it enters is. This allows the quotation rules
  to depend on the output properties (log entry, notification text,
  SMTP response text, ...) / (ASCII, iso-8859-1, UTF-8, binary, ...);

- delegate responsibility to call sanitize_str to write_log,
  instead of having a caller do it;

- wrap long syslog entries at 980 characters (less than 1023 minus prefix);
  thanks to Ted Cabeen;

- skip spam-related actions if $spam_level is undefined (e.g. when
  anti-spam code is not enabled) so that missing $sa_kill_level_deflt
  setting in amavisd.conf won't cause a surprise. Thanks to Marcio Merlone.

- new version of amavis-milter.c based on amavis-milter.c 1.1.2.3.2.36
  (from amavisd-0.1);

- use the /m regexp modifier by default when parsing av-scanner output
  for clean and infected regular expressions, making it easier to specify
  patterns, and to make it like the expression which collects virus names;
  thanks to Daniel Hunziker for reporting a problem with Symantec
  CommandLine Scanner;

- refined the semantics of sender whitelisting/blacklisting,
  add per-recipient sender while/blacklisting mechanisms:

  ENVELOPE SENDER WHITELISTING / BLACKLISTING  - GLOBAL (RECIPIENT-INDEPENDENT)

  WHITELISTING: use envelope sender lookups to ENSURE DELIVERY from whitelisted
  senders even if the message is recognized as spam. Effectively, for the
  specified senders, message recipients temporarily become 'spam_lovers', with
  further processing being the same as otherwise specified for spam lovers.
  It does not turn off inserting spam-related headers, if they are enabled.

  BLACKLISTING: messages from specified senders are DECLARED SPAM.
  Effectively, for messages from blacklisted senders, spam level
  is artificially pushed high, and the normal spam processing applies,
  resulting in 'X-Spam-Flag: YES', high 'X-Spam-Level' bar and other usual
  reactions to spam, including possible rejection. If the message nevertheless
  still passes (e.g. for spam loving recipients), it is tagged as BLACKLISTED
  in the 'X-Spam-Status' header field, but the reported spam value and
  set of tests in this report header field (if available from SpamAssassin,
  which may not have been called) is not adjusted.

  A sender may be both white- and blacklisted at the same time,
  settings are independent. For example, being both white- and blacklisted,
  message is delivered to recipients, but is tagged as spam.

  If ALL recipients of the message either white- or blacklist the sender,
  spam scanning (calling the SpamAssassin) is bypassed, saving on time.

  ENVELOPE SENDER WHITELISTING / BLACKLISTING - PER-RECIPIENT

  The same semantics as for global white/blacklisting applies, but this
  time each recipient (or its domain, or subdomain, ...) can be given
  an individual lookup table for matching senders. The per-recipient lookups
  override the global lookups, which serve as a fallback default.

  One can specify a two-level lookup table: the key for the outer table is
  recipient, and the result should be an inner lookup table (hash or ACL
  or RE), where the key used will be the sender.

  Per-recipient white/blacklisting is also available as a SQL lookup.
  See configuration variable $sql_select_white_black_list and README.lookups .


INCOMPATIBILITIES WITH amavisd-new-20021227

- the previous configuration file amavisd.conf from version
  amavisd-new-20021227 is usable with amavisd-new-20030314.
  Nevertheless it may be a good idea to adopt the new amavisd.conf,
  as it describes new options, is cleaner and better commented;

- reordered hash lookups from:
    user+foo@sub.example.com, user@sub.example.com,
    sub.example.com, .sub.example.com, .example.com, .com, . ,
    user+foo@, user@
  into:
    user+foo@sub.example.com, user@sub.example.com,
    user+foo@, user@,
    sub.example.com, .sub.example.com, .example.com, .com, .
  to make user@ still useful even when catchall ( . ) is used,
  and to bring it closer to the recommended SQL order;

  Not likely to affect existing setups, as previously the combination
  of user@ and catchall was not useful;

- SQL lookups:

  * incompatible change (avoiding unexpected behaviour): when the requested
    SQL field name does not exist in the SQL table, return a sensible default
    value for boolean fields (according to field name and its type),
    instead of 1 as previously. Logs a warning when no default is available.
    Thanks to Ortwin Gentz for noticing the problem;

  * new fields spam_tag2_level, spam_modifies_subj and local, with useful
    defaults if absent, so it is not necessary to immediately update an
    existing database;

  * changed SQL table 'policy' field name 'banned_files_lover' to a name
    'banned_files_lovers' (added an 's') for consistency with variables
    %banned_files_lovers and @banned_files_lovers_acl;


---------------------------------------------------------------------------
amavisd-new-20021227-p2 (2003-01-10):

- sender notification were not sent for messages containing banned
  filenames but no virus; fixed (thanks to Philip Ross);

- added virus name 'Sobig' to the default $viruses_that_fake_sender_re
  (amavisd.conf);

- extended default $banned_filename_re to match also a longer
  (4 character) middle file extension (e.g. Movie_0074.mpeg.pif)
  (amavisd.conf);

- add explicit path to the program vscan in the 'Trend Micro FileScanner'
  entry (amavisd.conf), thanks to Matthew Hobbs;

- improvements in the setup of helper programs (files configure.in,
  Makefile.in, config.h, amavis.c and amavis-milter.c), by Radu Greab;

- add Sophos error code 527 ('Not supported in this SAVI implementation')
  to the list of (mostly)harmless codes in the SAVI-Perl interface routine
  (thanks to Paul B. Henson and Tim Winders for discussion);

- initialize SAVI-Perl module (if available) in each new child, instead of
  only once during daemon startup time. This avoids the need to reload
  amavisd for every change in the Sophos IDE database - at the expense
  of additional SAVI-Perl startup time every $max_requests messages;


amavisd-new-20021227-p1 (2002-12-30):
  fix the problem when using multiple virus scanners, where the
  last scanner doesn't find a virus, but some other scanner does,
  and amavisd-new (incorrectly) does not report a virus.
  Thanks to Okke Timm, Henrik Larsson and Eduardas Paulavicius;

---------------------------------------------------------------------------
                                                          December 27, 2002
amavisd-new-20021227 release notes


amavisd-new-20021227 is primarily a maintenance release, consolidating
amavisd-new-20021116 with its mandatory patches, updating documentation,
but also brings a couple of niceties.


Changes since amavisd-new-20021116 patch level 5

- The new configure/makefile in the helper_progs subdirectory is
  contributed by Stephane Lentz, based on MIMEDefang configure.in.
  It is needed to build helper programs for sendmail (milter and nonmilter)
  setups (but not needed for Postfix and Exim v4 setups, or any dual-MTA
  setup). Similar to the prior work done by Ramiro Morales - thanks to both!

- Updated documentation (README, INSTALL, RELEASE_NOTES, README.lookups,
  README.customize, README.postfix).

- new subdirectory test-messages, containing sample mail messages to test
  virus and spam detection - see README there

- For daemonized antivirus scanners (only) the semantics of few parameters
  in the @av_scanners list has been incompatibly changed and enhanced.
  The same internal mechanism now handles both Unix sockets and inet socket.
  More than one socket may now be specified, to be tried one after the other
  until one is successfully connected to (useful for 'FRISK F-Prot Daemon').
  Also the usual regexp pattern parameters in the @av_scanners are now
  observed by Sophie and Trophie interfaces as well, bringing them more
  in line with other scanners.

- The (?m) regexp flag in @av_scanners patterns is now turned-on
  by default, so it is no longer necessary to specify it in @av_scanners,
  and has been removed to shorten the expressions.

- The SAVI-Perl interface routine now turns on most Sophos engine options,
  enabling recognition of some more file formats.

- All active antivirus scanners are now called, the scan no longer stops
  when the first one finds a virus. This is on the average a very small
  price to pay (only payed in virus storms), but makes possible to see
  which virus scanners found the virus and which did not, and to get their
  timing in the logs. The list of virus scanners that noticed the virus
  is now included in the notification message to the virus administrator.
  A new macro %W is available for this purpose.

- More sensible logic when checking more than one file for those
  daemonized av scanners that need to be handed individual files instead
  of directory name.

- The 'Clam Antivirus-clamd' daemon is now supported, both via Unix
  socket, as well as via inet socket (but Unix socket should be used,
  due to lack of self-protection in this respect by clamd).

  clamd is a fast and open-source anti-virus scanner
  ( http://clamav.elektrapro.com/ ), and I recommend it to be used in
  combination with other commercial virus scanners as the last resort
  scanner, to avoid bouncing mail in case primary scanner(s) stumble
  across some unhandled situation, like corrupted or encrypted mail.

- New utility program 'amavisdconf' to check the values of configurable
  variables set by /etc/amavisd.conf or defaulted (modeled after
  Postfix utility 'postconf', but -e option is not yet supported).

  Usage:
    amavisdconf [-d | -n] [-c conf-file] [variable]

  Options:
   (none) show the value as will be used by amavisd;
    -d  show default value as in the absence of the config file;
    -n  only show variable if its value is different from the default;
    -c conf-file ... use the specified file instead of /etc/amavisd.conf

  If a variable is specified show only that variable, otherwise show all
  configurable variables. (Note: the leading $, @ or % must be included
  with the variable. Make sure to protect $ from shell evaluation,
  e.g. by enclosing variable name in single quotes, or prepending
  a backslash:
    $ amavisdconf -d '$forward_method'
    $ amavisdconf -n

  This is the first attempt at such a utility, some details would need
  to be polished (e.g. how to report code reference), but it is hoped
  to be usable already in its present form.

- Two new subroutines (read_hash and read_text) are now available
  for use from amavisd.conf file, with a primary intent to move larger
  texts out of the amavisd.conf file:

  * read_hash allows for lookup hashes (e.g. white/blacklist,
    static %spam_lovers table, ...) to be read from a file during startup;
    It takes quoted form of addresses, optionally enclosed in angle
    brackets, and converts them into unquoted (raw) form, as needed
    for the hash keys. Format: one address per line, everything from
    '#' to end of line is a comment, leading and trailing whitespace
    removed, empty (or comment-only) lines are ignored.

  * read_text allows for reading more complex strings (e.g. multi-line,
    or including special characters such as backslash) from a file.
    Useful to override the default template text for notification messages.

  See example usage in amavisd.conf.

- Three synonym (alias) subroutine names are available for use
  in amavisd.conf to shorten the expressions:

    new_RE       same as   Amavis::Lookup::RE->new
    ask_daemon   same as   Amavis::AV::ask_daemon
    sophos_savi  same as   Amavis::AV::sophos_savi

  amavisd.conf file is changed to use the short names, but full
  subroutine names are still valid.

- Provide protection mechanism for SMTP/LMTP server code to limit the
  number of recipients in a mail transaction to some sane value
  ($smtpd_recipient_limit, default 1000). Even if the number is exceeded,
  mail will still be normally delivered in more than one transaction
  (provided your MTA is rfc2821 compliant).

- SMTP client code now correctly handles the situation where remote
  SMTP server returns '452 Too many recipients'. When this happens the
  mail forwarding or submission is split into several SMTP transactions,
  and as long as the remote SMTP server is willing to accept at least
  one recipient per transaction, mail delivery is assured.

- Auto white-listing (AWL) by SpamAssassin may be turned on by
  setting $sa_auto_whitelist to true (contributed by Hamish Marson).
  This only works as one global database, the per-recipient AWL
  is presently not available.

- New configuration variables $hdrfrom_notify_sender, $hdrfrom_notify_admin,
  and $hdrfrom_notify_spamadmin, to remove the overloading (double semantics)
  from variables $mailfrom_notify_*. Now $mailfrom_notify_* control ONLY
  envelope addresses, while $hdrfrom_notify_* only controls From header
  in notifications. The default values for $hdrfrom_notify_* are chosen
  to preserve compatibility.

- The spam headers in quarantined message now reflect the lowest
  tag and kill levels when more than one recipient is given, instead
  of reporting the levels for the first specified recipient.

- Tested with SpamAssassin 2.50 and with its new Bayesian classification
  (used as a global database by amavisd-new). Given two sets of spam
  and non-spam (=ham) messages, one can build a database by using new
  SpamAssassin utilities sa-learn-spam and sa-learn-nonspam as user amavis,
  and the database built this way will be used by amavisd-new/SpamAssassin.

- Initialize SpamAssassin as $daemon_user (no longer as root),
  as a safety measure, and to ensure the files it creates are still
  accessible after the UID change.



Incompatible changes to 20021116:

- Different parameters in @av_scanners list for daemonized antivirus
  scanners (as described above).

- lookup_hash: an undefined hash value was previously treated as a
  special case, resulting in value true. This is no longer the case,
  which brings it in line with other lookup methods. It is not likely
  anyone will be affected, as this practice was only documented but
  not encouraged by any example.


---------------------------------------------------------------------------
Here is a brief description of patches that needed to be applied to
amavisd-new-20021116, and were available from the amavisd-new web page:


patch5 (2002-12-09):
- fix a simple test to decide if the anti-virus and anti-spam code is needed;
- use the /m regexp modifier by default when parsing av-scanner output
  for virus names - seems like (?m) in qr is ignored (have we found
  a Perl bug?);
- fix DSN to report original smtp response in case reporting MTA is remote
  (thanks to Chris Hastie for his keen eye in understanding DSN);
- fix reporting quarantine mailbox name(s) in case of per-recip quarantine
  or quarantine forwarding to MTA;
- macro %q (quarantine id) is now a list of quarantine mailbox names or
  addresses (may have more than one entry in case of per-user quarantines);
- macro %v (av-scanner output) is now a list of scanner output lines,
  to facilitate formatting of multi-line av-scanner output;
- fix spam admin notifications, which used the virus template instead of
  the spam template;
- fix problem of undefined entity causing spam notification failure
  in case other content checking (virus/banned) is not enabled;
- make SQL lookups (DBI access) optional - no DBI code is loaded if
  @lookup_sql_dsn of empty;
- support SAVI-Perl module 0.10 or later (direct access to the Sophos library
  engine) ( http://www.csupomona.edu/~henson/www/projects/SAVI-Perl/ );
- add macro %j that evaluates to message 'Subject:' header field;
- fix minor SMTP protocol inconsistency when multiple MAIL FROM
  commands are issued by the client;
- more informative timing report when more than one av-scanner is used;
- make sendmail initial submission happy when null reverse-path is specified
  (instead of the correct empty argument, which works but makes sendmail
  issue a warning, we now use <> in arguments to specify null paths),
  (thanks to Sebastian Hagedorn and Dale Perkel);
- replace log entry "spam from=<%s>, to=%s" with "SPAM, <%s> -> %s ...".
- dropped (redundant) historical %bypass_checks @bypass_checks_acl
  (use bypass_virus_checks and %bypass_spam_checks instead);

patch4 (2002-11-21):
  In amavisd:
- SQL lookup fix;
- finished per-recipient header edits and split-forwarding, where
  one multi-recipient message may be forwarded as several messages
  if different headers need to be inserted for different recipients
  (not with milter);
- the check for banned file contents now looks for both the short
  and the original full type classification as provide by 'file' utility
  (thanks to Wolfgang Lumpp for the idea);
- #!/usr/local/bin/perl => #!/usr/bin/perl
- cosmetics

  In amavisd.conf:
- suggested group name change $daemon_group 'sweep'=>'amavis';
  so that non-Sophos users are not puzzled where 'sweep' group comes from;
- added path and fixed args for KasperskyLab AVPDaemonClient
  (thanks to Mike Hall);
- added path for Trend Micro FileScanner vscan
  (thanks to Eduardas Paulavicius);
- added 'use strict;' and '1;' at the end; prevent read_config from failing
  if the config file does not return true (thanks to Vivek Khera);
- cosmetics, fixed typos in comments;

patch2 & patch3 (2002-11-18, 2002-11-19):
- fixes a problem with 'recip_done' object method (causing TEMPFAIL)
  when $warnvirusrecip is enabled (thanks to Chris Hastie, Dale Perkel,
  and Didi Rieder);
- pass null return path to sendmail in such a way to make it happy
  (thanks to Sebastian Hagedorn and Dale Perkel);
- added a safety precaution to prevent somebody shooting himself in the
  foot by specifying an empty $forward_method while using the SMTP in/out
  setup (e.g. Postfix), which could send mail down the bit bucket;

patch1 (2002-11-17):
- fixes a problem in milter setup where per-recipient response codes
  needed to be handled differently (the problem was causing a TEMPFAIL)
  (thanks to Stephane Lentz and Didi Rieder for the initial milter testing
  and problem reports);



---------------------------------------------------------------------------
                                                          November 16, 2002
amavisd-new-20021116 release notes

- provide a mechanism to load only required code sections
  (anti-virus scanning, anti-spam scanning, SMTP/LMTP server module,
  traditional amavis client & milter server module),
  resulting in reduced memory usage and less installation dependencies;

- introduce the per-recipient status handling and make possible some
  per-recipient functionality that was not possible before, such as
  handling the per-recipient spam thresholds. This required a major
  rewrite and cleanup of some sections, hopefully providing cleaner
  mechanism for possible future advances in this area.

  Consider for example a previously incorrectly handled situation
  where there is more than one recipient, and some (but not all) of them
  get a reject from the (outgoing) relay MTA - it is not possible for a
  single SMTP response or status code to describe the situation;

- generate proper (non-)delivery status notifications (DSN), compliant with
  rfc1892 (now rfc3462) and rfc1894 (now rfc3464). This form supersedes
  the simple virus/spam sender notifications, but also covers the area of
  genuine delivery failures, such as selective recipient rejects by the
  outgoing relay MTA, which could previously lead to multiple deliveries;

  NOTE: to avoid sender getting two non-delivery messages (one from
  MTA, and another from amavisd), do not set $warnvirussender and
  $final_virus_destiny=-1 (REJECT) at the same time (and equivalently
  for spam settings);

- enhanced mail system status codes (rfc1893 (now rfc3463), and rfc2034)
  are now included with all SMTP responses and DSN notifications;

- added checking for banned MIME types and names. If any mail part
  matches, the whole mail is rejected, much like the way viruses
  are handled. A list in object $banned_filename_re can be defined
  to provide a list of Perl regular expressions to be matched against
  each part's:

  * Content-Type value (both declared and effective mime-type),
    including the possible security risk content types
    message/partial and message/external-body, as specified by rfc2046;

  * declared (recommended) file names as specified by MIME subfields
    Content-Disposition.filename and Content-Type.name, both in their
    raw (encoded) form and in rfc2047-decoded form if applicable;

  * file content type as guessed by 'file' utility and classified into
    short type names such as .asc, .txt, .html, .doc, .jpg, .pdf,
    .zip, .exe, ... - see subroutine determine_file_types().
    This step is done only if $bypass_decode_parts is not set.

  NOTE: by default the $final_banned_destiny is set to 1 (pass),
  so detected banned file names only cause a header line to be added,
  quarantining, and added address extension - but the mail is delivered
  nevertheless. This default is set so that we can get initial experience
  without being too obtrusive. Change as you prefer.

- besides SMTP/ESMTP protocol, the server now also accepts LMTP protocol
  (rfc2033). This now explains why I had to implement PIPELINING,
  ENHANCEDSTATUSCODES and 8BITMIME SMTP extensions, as these are required
  by rfc2033.

  One advantage of using LMTP to feed content filter is that LMTP uses
  per-recipient status response, as opposed to 'one-size-fits-all'
  SMTP status response, which require that the SMTP client (e.g. content
  filter) is capable of generating DSN.

  Another advantage is specific to the Postfix setup, as Postfix
  LMTP client supports multiple transactions per session, saving on
  connection teardown/reconnect for every message being checked.

  To enable LMTP feed from Postfix, add 'max_use=10' to main.cf, and
  replace last argument 'smtp' with 'lmtp' in the master.cf line:
  'smtp-amavis unix - - n - 2 lmtp'. No changes are needed in amavisd-new,
  both protocols can coexist, the distinction is based on HELO/EHLO
  vs. LHLO command.

  Btw, older versions of Postfix lmtp client inappropriately lowercased
  the envelope addresses. This is fixed in Postfix Snapshot 1.1.11-20021015,
  and in the regular Postfix release 1.2 (when available). Lowercasing
  can be a problem for addresses where local part is case-sensitive,
  although such setups are rare.

- improved per-recipient SMTP response code handling when sending mail
  via SMTP. Previously one rejected recipient (by MTA) in a multi-recipient
  message caused the whole message to me rejected. This is now correctly
  handled with the new ability to send DSN;

- added command line option '-c config-file' so one can override the
  default location of the configuration file (/etc/amavisd.conf);

- explicitly set PATH and HOME environment variables
  (settings: $path and $helpers_home)

- added another form of lookups: Perl regular expression matching.
  See README.lookups for details. Corresponding new variables in amavisd.conf
  are: $virus_lovers_re, $spam_lovers_re, $bypass_virus_checks_re,
  $bypass_spam_checks_re, $local_domains_re, as well as $banned_filename_re,
  $viruses_that_fake_sender_re, $keep_decoded_original_re,
  $whitelist_sender_re, $blacklist_sender_re.

- besides whitelist_sender* lookup tables, there is now also
  a blacklist_sender* set of tables, which causes mail to be declared spam
  and to skip remaining spam checks. See examples in amavisd.conf.

- provide a configurable lookup table $keep_decoded_original_re
  of file types, for cases where unpacker is not very trustworthy.
  The lookup key is what 'file' utility returned. If the part contents
  match the lookup table, we keep both the original and the unpacked file
  for virus scanner to check;

- provide a configurable list of regular expressions
  $viruses_that_fake_sender_re, which recognizes viruses that usually
  fake envelope sender address. Don't send sender notification if a match
  is found. For syntax of the new regexp lookup tables see README.lookups;

- specifying per-recipient quarantine address or location is now possible
  by setting $virus_quarantine_to and $spam_quarantine_to be a ref to a
  hash lookup table. Thanks to Vivek Khera for the idea;

- no 'configure' and 'make' for the daemon; all configuration is done
  via amavisd.conf configuration file at the daemon startup time;
  Also the ./mta, ./av and ./notify configuration subdirectories
  are now gone;

- provided a sample init shell script amavisd_init.sh (edit to will,
  and move it to /etc/init.d/amavisd if you want). Based on the script
  from Wil Cooley;

- one-shot debugging mechanism: if envelope sender matches @debug_sender_acl
  lookup table, turn debugging fully up just for this one message and
  cause temp file and directories not to be cleaned for this message.
  This facilitates debugging a particular problem even in the presence
  of regular traffic;

- cleaner debug log entries for multi-transaction SMTP/LMTP protocol sessions;

- cleaner log entries - avoid misleading and incomplete text
  when quarantine is disabled; thanks to Michael Leone;

- avoid using (nonstandard) field width in the %e format specifier
  when calling strftime, which lead to bad date syntax on some systems;

- local time zone offset is now automatically computed, no more '-0000'
  in rfc2822 dates;

- put Message-ID field value in angle brackets as required by rfc2822
  when generating notifications;

- add a In-Reply-To field to notifications when original Message-ID is known;

- don't bounce a virus (or spam) back to a mailing lists even if
  $final_virus_destiny (or $final_spam_destiny) is set to REJECT;
  a patch by Brian May from the Debian support crew, thanks;

- because some external module may play games with STDIN and STDOUT
  (like SpamAssassin seems to do when local_tests_only=>0)
  run the input protocol directly on the Net::Server's socket,
  not on STDIN and STDOUT, which are aliases to the socket;

- a patch to amavis-milter.c by Didi Rieder to support REJECTing mail,
  by instructing sendmail to return non-delivery notification to sender;

- a modified amavis.c helper program to make possible to invoke
  local delivery agent from it, for those still using such a setup;

- a new macro %l is available for use in notification messages (via
  'expand'); it evaluates to true (1) if the sender matches @local_domain,
  and returns empty otherwise; by default it is now used in creating
  a 'Subject:' line, inserting word 'LOCAL' before the sender name
  when appropriate;

- new macros %D and %N expand to lists of recipients that got the mail
  delivered (%D), or not-delivered (%N).  Union of both sets gives %R,
  i.e. a list of all recipients as specified in the envelope;

- dropped macro %a

- improved parsing of e-mail addresses according to rfc2821
  (full address literals syntax, etc.);

- proper line folding for generated rfc2822 header fields;

- proper SMTP response wrapping (for very long responses such as
  the ones that include a trouble report) according to rfc2821;

- allow for obsolete rfc822 syntax of permitting whitespace before
  colon in header field name;

- added optional spam-sender nondelivery notifications, based on patch
  from Lazslo E. Miranda (lazslo@dcc.ufmg.br) and Fernando F. Morais
  (frota@cecom.ufmg.br);

- dynamically change process name (Perl variable $0) to reflect
  the process state; suggested by Chip Paswater. Not all operating
  systems make this process state visible by ps(1);

- determine location of external programs (or their absence) at
  startup time; in amavisd.conf one may specify absolute path or just
  rely on PATH. This mechanism is also used to determine absolute path
  of the daemon itself, making reload (after HUP) more predictable;

- explicitly specify lock file for serialization to be used by the
  Net::Server::PreForkSimple module, instead of relying on default
  provided by POSIX::tmpnam. The default approach has a possibility
  for a minor security problem, because the lockfile is created with
  open()..., so it will follow symlinks. Observed, and a patch
  provided, by Jarno Huuskonen;

- new defense against mail bombs: for the cumulative total of all
  decoded mail parts we set max storage size. The formula is:
    quota = max($MIN_EXPANSION_QUOTA,
                $mail_size*$MIN_EXPANSION_FACTOR,
                min($MAX_EXPANSION_QUOTA, $mail_size*$MAX_EXPANSION_FACTOR))
  In plain words (later condition overrules previous ones):
    allow MAX_EXPANSION_FACTOR times initial mail size,
    but not more than MAX_EXPANSION_QUOTA,
    but not less than MIN_EXPANSION_FACTOR times initial mail size,
    but never less than MIN_EXPANSION_QUOTA

- if the permitted quota is exceeded (or the defense triggered by other
  similar safeguards), the virus scanning is skipped to protect the
  virus scanner from tripping over the mail bomb, a header field
  is inserted:
     X-Amavis-Hold: ... reason ...
  and a log entry 'Placing on HOLD: reason' at level 0 is produced.
  Also the temporary directory is preserved.

  The 'X-Amavis-Hold: ...' header field can be used by your MTA to
  put the message 'on hold' (freeze). If MTA is not set up to catch
  messages with this tag (the default), they are passed normally to
  recipients. This is likely to be the best action under the circumstances.

- tested with razor-agents-2.20 and SpamAssassin-2.43 (Razor2 is now called
  by SpamAssassin, and no longer directly by amavisd-new). New configuration
  variable $helpers_home, which defaults to $TEMPBASE. Thanks for hints
  by Chris Hastie and John Stewart;

- to avoid CPU loop in SpamAssassin-2.43 and earlier, my patch needs
  to applied to SA - see amavisd-new web page;

- to avoid taint problem in Razor 2.20 (if SpamAssassin-2.43 is configured
  to call it), my patch needs to be applied to it - see amavisd-new web page;

- chroot available (but not well tested): $daemon_chroot_dir = '/var/amavis'

- provide a fail-over mechanism for SQL database connect - given a list
  of SQL servers/databases, pick the first that is available.
  Thanks to Ken McKittrick for making available the patch,
  and to Ben Ransford <ben@terakeet.net> for writing it.

- remove existing virus-related and spam-related headers (some of them
  optionally) if we'll be providing our own; suggested by Borut Mrak;

- avoid 'insecure dependency' in lookup_sql when calling DBI::execute
  (thanks to ric* at mpc.com.br)

- fixed macro %H to provide original header lines, not the ones
  stored internally by MIME::Entity, which may have been modified;
  (thanks to Chris Hastie for noticing the problem);

- no longer inserts X-Razor-id header field;


INCOMPATIBLE CHANGES:

- when specifying boolean values to variables in the amavisd.conf file,
  please specify 1 (or old style "yes") for true, and 0 or "" or undef
  for false. The old style "no" yields true for Perl, and is only still
  supported (converted to 0) for some traditional variables for
  compatibility with amavisd(-snapshot);

- removed variables $sendmail_wrapper*, and changed the syntax for
  specifying $forward_methor/$notify_method, with the intention to
  do all mail sending settings at one place. See amavisd.conf for examples;

- variable $mailto is now deprecated (but still works as a fallback default
  for compatibility with previous version). Use $virus_admin and $spam_admin
  lookup tables instead, they also offer a mechanism to specify per-domain
  administrator address;

- variable $warnadmin is no longer used. Use $virus_admin and $spam_admin
  lookup tables instead. Not specifying administrator address
  (e.g. leaving $virus_admin, $spam_admin and $mailto undefined)
  turns off admin warnings.

- dropped variable $LOGDIR, the variable $LOGFILE now specifies the full path;

- renamed: $warnsender -> $warnvirussender, $warnrecip -> $warnvirusrecip

- dropped variables $enable_relay, $sendmail_cf_orig, $QMAILDIR;
  they were never used in amavisd-new;

- dropped $MAX_ARCHIVE_NESTING, which is replaced by new storage limitations:
  $MIN_EXPANSION_QUOTA, $MIN_EXPANSION_FACTOR
  $MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR

- SQL database is now specified differently (@lookup_sql_dsn)


MTA-SPECIFIC: sendmail/milter

- $notify_method now specifies deferred delivery mode ('-odd') by default,
  when submitting notifications to sendmail. This is to avoid calling milter
  immediately during submission, which in turn calls amavisd-new, possibly
  leading to a deadlock situation when the number of amavisd-new child
  processes is small. Seems like this change is needed since Sendmail 8.12
  or so.

  The following recommendation is from mimedefang-filter man page:

  | You MUST run a client-submission queue processor if you use
  | Sendmail 8.12.  We recommend executing this command as part of the
  | Sendmail startup sequence:
  |     sendmail -Ac -qp1m


DECODERS-UNPACKERS:

- rewritten decode_parts() to allow for retaining source text if the
  unpacker sw is considered unreliable. This more cleanly resolves
  the problem reported on the amavis-user mailing list on 2002-06-06:
    "Amavisd passing through VBS/VBSWG.gen@MM"
  A nice side-product is that a directory search is avoided for
  each nested unpacking step;

- refined do_unzip to control and limit the size of decompressed
  members (among others it defends against the 42.zip-type bombs);

- rewritten external decompressors interface to the gzip/bzip2/compress
  family. Instead of using a 'system' call, they are now called through
  fh_copy, making possible to control and limit the size of the
  decompressed contents on the fly, avoiding denial-of-service attacks.
  Affected: b(un)zip2, g(un)zip, (un)compress;

- fh_copy now uses IO::Handle object to assure the forked process
  gets reclaimed even in case of aborted contents extraction;
  stdin gets redirected to /dev/null or to a specified input file
  for the exec'd process;

- Convert::UUlib is called again (the amavisd-new-20020630 removed its
  usage due to problems with improper decoding). This time the originals
  are kept, so that virus checker sees both the original and the
  attempted-decoded part;

- completely rewritten do_ascii and its usage of Convert::UUlib
  to fix:
  * coding error (bitwise op treated as logical op and improperly negated):
      ... if (!$uu->state || !FILE_OK || -z $newpart);
  * it never reset the state, so if a successfully decoded ASCII file
    contained another ASCII file, each decoding level would decode all
    previously decoded parts again, plus add new ones at this level;
    recursion would not stop until the hard limit, resulting in TEMPFAIL;
  * it never checked nor reported errors that should have been detected
    (I/O errors, out of memory, trouble accessing or creating files);
  * more informative log entry;

- save MIME preambles and epilogues (if nonempty) as extra (pseudo)parts
  to be scanned. This also mitigates the problem of syntactically-incorrect
  MIME mail as produced by some user agents, which (rightfully) gets
  treated as one long preamble by MIME-Tools, and previously went by
  unchecked;

- supports unpacking arc archives using 'nomarch' (by Russell Marks,
  http://rus.members.beeb.net/nomarch.html), (thanks to David D. Kilzer
  for the initial code). Using 'nomatch' fixes a nasty habit of arc
  which gratuitously appends a form-feed at the end of file when using
  the 'p' (pipe) option, which might mislead a virus scanner.
  Besides, 'nomarch' is GPL licensed;

- fixed a 'broken pipe' problem when calling unrar,
  thanks to Ricardo Campos Passanezi and Rainer Link;


VIRUS SCANNERS:

- rewritten interfacing for most command-line virus scanners.
  All settings for them is now done in amavisd.conf. New ones may be
  added without having to modify the daemon source. More complex scanners
  (e.g. daemonized scanners) still need to have a corresponding interface
  routine in the daemon;

- rewritten Sophie and Trophie interfaces to be more resilient
  to Sophie/Trophie daemon restarts during virus database reloads,
  avoiding an unnecessary retry (TEMPFAIL) - thanks to Cor Bosman
  for the suggestion and code, and to Dale Perkel for testing the
  Trophie interface;

- make sophos sweep tolerant to encrypted attachments: if all files are
  password protected, then the scanner failure is ignored and the message
  is allowed to pass. Based on patch by Radu Greab;

- updated nai uvscan interface to recognize the result
  'Found trojan or variant Exploit-CodeBase !!!', a patch by Anton Berezin;

- Clam Antivirus supported;
  
- fixed a problem in 'avp' scanner interface ($TEMPBASE not imported),
  thanks to Joshua E Warchol;

- avpdc (KasperskyLab AVPDaemonClient): recognize additional exit codes,
  based on patch from Christian Hammers;

- Panda new regexps, no TERM vt100 setting (thanks to Benjamin Zwittnig);

- Trend vscan exit code seems to be the number of infected parts.
  Updated the test to reflect that, based on observation from Stephane Lentz;

- MksVir scanner interface returns error code 2 if viruses are removed
  (if --clean option is passed) - a patch by Robert Litwiniec;

----------------------------------------------------------------------------
                                                               June 30, 2002
amavisd-new-20020630 release notes

Since it seems like several people are adapting amavisd-new
in details to their requirements, and certain improvement requests
have much in common, I'm making available the 20020630 release,
shortly before leaving on vacation. As this means my support
will be absent for the coming few weeks, and this release is
perhaps by few days premature, please consider it primarily a
development and new features release.

It is available at the usual location at:

  http://www.ijs.si/software/amavisd/
  (or ask Google about 'amavisd-new')

Having said that, it is still a fully functional and tested version,
and it is running in production at a couple of sites now. Also it is
my version of choice in view of dependability, having it running
at our site while I'm away. If you get into trouble, you can still go
back to amavisd-new-20020517 with which it is fully upwards compatible.

The main changes and features since amavisd-new-20020517 are:

- the code is thoroughly rearranged, interfaces cleaned, separated
  into namespaces (packages), several sections generalized (e.g. lookups,
  appending/editing header lines). The AV scanner and unpacking sections
  are still mostly the same and compatible with amavisd, so whatever
  improvements and new AV scanner support becomes available for amavisd,
  applies almost without a change to amavisd-new;

- SMTP on the input side (used with Postfix and Exim) now talks ESMTP
  (rfc2821) and not just rfc821, including some SMTP extensions:
  command pipelining (rfc2920), message size declarations (rfc1870), and
  8bit-MIME transport (rfc1652). The main reason for this was the change
  in recent Postfix versions which can now do MIME transformations to support
  7bit transports (implied by SMTP). To ensure the transparent 8bit path and
  avoid message transformation by MTA, amavisd-new needs to declare it does
  present an 8bit-clean path. A side benefit is a little speedup in passing
  chunks of mailing list addresses due to pipelining support;

- split certain previous amavisd.conf settings (variables)
  into several variables or lookup tables:

  * sender address for notifications:
    $mailfrom -> $mailfrom_notify_admin, $mailfrom_notify_sender,
                 $mailfrom_notify_recip, $mailfrom_notify_spamadmin
    (these may also be empty to specify null reverse path <>,
    which is most useful for sender notification);

  * administrator address for notifications:
    $mailto -> $virus_admin, $spam_admin (per-sender lookups)

  * %bypass_checks -> %bypass_virus_checks, %bypass_spam_checks

- supports SQL database lookups via Perl module DBI (interface to
  popular database types). Some examples are provided, if you need
  other SQL lookups just modify the code by analogy. SQL lookups
  are most useful for per-user settings of virus_lovers, bypass_virus_checks,
  bypass_spam_checks, spam thresholds etc. when the user base is large
  and subgroups can not be identified through their (sub)domains.
  Another use is for dynamically changing settings without having
  to restart amavisd-new;

- can optionally insert 'Received:' header if acting as a mail relay
  (not with milter); does loop detection as required by rfc2821 section 6.2;

- notification messages now contain 'Date:' and 'Message-id:' headers;

- quarantined viruses contain X-AMaViS-Alert header line with names
  of detected viruses;

- quarantined spam contains X-Spam-Status and X-Spam-Level header lines;

- optionally send spam admin notifications, which include the full
  SpamAssassin spam report and message header;

- when started as root, changes UID and GID to $daemon_user, $daemon_group;

- to facilitate startup scripts and debugging, supports few simple
  command line parameters:
    amavisd        ... standard run: changes uid/gid and daemonizes
    amavisd start  ... same thing
    amavisd debug  ... starts with full debug level, stays attached
    amavisd reload ... finds amavisd master process and sends it a HUP
    amavisd stop   ... finds amavisd master process and sends it a TERM

- some more (minor) configurable options:
    $daemonize, $pid_file,
    $replace_existing_extension, $localpart_is_case_sensitive

- no longer calls Convert::UUlib for uuencoded, xxencoded, and binhex
  attachments. The first two are handled by MIME::Parser, the remaining
  are likely to be handled by anti-virus scanners, especially when some
  virus would use such encoding. The use of Convert::UUlib is dropped
  because it was causing recent problems with garbling virus so that it
  could no longer be recognized by AV scanners, and because the underlying
  library does not seem very dependable. See the thread 'Amavisd passing
  through VBS/VBSWG.gen@MM' from the beginning of June 2002 in amavis-user
  mailing list archives;

- quarantine files now include internal amavis id in the file name,
  instead of the process number; previous naming scheme could stumble
  across a name contention on a busy system;

- only a recommendation: logging via syslog is now preferred to direct
  logging to a file. It serializes the logging, and avoids locking/unlocking
  and reopening a log file by amavisd* for every log entry.
  The syslog daemon does it more efficiently and reliably.

  NOTE for Linux users: make sure you prefix the file name in syslogd.conf
  with a '-' tag to disable fsync after every write; this is most necessary
  for heavy logging such as from MTA and/or amavisd-new.  The downside
  is that you may lose the last few log entries in case of machine crash.
  See man page of your syslogd for details.

- changed examples according to rfc2606 and recent complaints on the
  postfix-users list.


CAVEATS:

- header rewriting is only available in SMTP-in/SMTP-out setup,
  i.e. with Postfix and Exim, but not with sendmail milter setup;

- the SQL lookups are a very recent addition and not so well tested
  as the rest of the program. Also since SQL lookups are supposed
  to introduce per-user settings (e.g. spam thresholds), the code
  still does not accommodate it, and spam thresholds of the first
  recipient in a message affects the whole message;

- as Razor2 is still pretty much unstable, I left the spam_scan()
  routine much as it was in the May version. Contributions welcome;

- as I ran out of my time for this release, I didn't prepare a
  separate version without SpamAssassin and Razor 1.20 support,
  so you will need to strip it out if you do not need it.
  Similarly for the required DBI Perl module. I'm very sorry;

- the time stamp used in the 'Date:' header in notifications,
  and in an optional 'Received:' header, does not contain true time zone
  offset, but -0000 instead (standard meaning for unknown). I did not want
  to include a fat Perl package for handling time zones. For now just edit
  the subroutine rfc2822_timestamp() if you want to change that;

- amavisd.conf settings $daemon_user and $daemon_group are not
  automatically set by ./configure options. Please set them manually.

I would like to thank many people on the amavis-user list, on the
postfix-users list, and in private conversations, who contributed
valuable ideas and improvements, and offered much appreciated
encouragements.

---------------------------------------------------------------------------
                                                               May 17, 2002
Available at:
  http://www.ijs.si/software/amavisd/

amavisd-new-20020517 is primarily a response to popular demand for
Mail::SpamAssassin support. If amavisd-new-20020424 meets your needs,
there is no urgent need to upgrade. There are no incompatible changes
between these two versions, except some new (optional) amavisd.conf
variables, so you may keep old amavisd.conf file if you wish.

A new file README.exim is now provided, so that Exim 4.x is now a
supported and tested configuration, besides the usual Postfix and
sendmail/milter MTAs. Also included is a brand new qmail amavis client
by Lars Hecking (untested, please try it), plus his updated version of
amavis.c (no longer needed in recommended Postfix and Exim configurations,
and Sendmail milter and qmail configurations use a different client).

Several files are unchanged from the base amavisd CVS release 2002-05-13
and do not reflect the amavisd-new state: FAQ, HINTS, INSTALL, BUGS,
NEWS, TODO, ChangeLog, tests/, doc/. Please start with this file
README.amavisd-new-RELNOTES and follow it. More recent instructions
and last-minute changes are available from the web page.


Changes since amavisd-new-20020424:

- supports Mail::SpamAssassin and Vipul's Razor (1.20 required)
  for spam checking (but not for modifying mail body - only add headers
  and/or address extension, or reject/discard/quarantine spam).

  NOTE: spam checks are off in the default amavisd setup.
        Copy file ./amavis/amavisd.in.all to ./amavis/amavisd.in
        before running ./configure and make, to use the
        spam-check-enabled version!

  Amavisd calls Mail::SpamAssassin directly, avoiding the need to set up
  spamc/spamd or to chain filters. This is more efficient, one daemon
  less to worry about, although maybe less flexible for some taste.
  At the moment the SA per-user database is not used. Feel free to
  experiment with it and let me know what you came up with.

  The usual SA config files are observed, but remote tests are disabled
  by default (Razor is handled directly, RBL lookups can/should be
  performed early by MTA which knows what IP address mail came from,
  and that information is (mostly) lost afterwards). If you feel otherwise,
  change the hard-wired settings in the call:

    $spamassasin_obj = Mail::SpamAssassin->new(
      { dont_copy_prefs => 1, local_tests_only => 1 } )

  (also the Razor score contribution is hard-wired,
  modify: '$razor_spam_found ? 3 : 0' to will).

  Amavisd-new handles Razor checks directly to be able to exercise
  more control over it than would otherwise be available through SA:
  timing, signatures are needed for insertion into header, skips one-liner
  body checks which are common Razor false-positives. To avoid SA calling
  Razor again, either keep the default setting 'local_tests_only=>1',
  or set 'score RAZOR_CHECK 0' in the SA configuration.

  Added are 'whitelist_sender' hash and ACL lookups (see README.lookups),
  which approves spam from specified SMTP originator addresses - SpamAssassin
  can only check and whitelist rfc822 headers, not the envelope addresses,
  and I see no way of passing envelope addresses to it - which is a pity,
  as important information is lost.

  SpamAssassin checks are computationally quite expensive compared to other
  amavisd activities, and the time needed for SA check goes up significantly
  with the message size. SA check are skipped (but not the Razor check)
  if mail size exceeds 64k. My analysis shows that presently less than 1%
  of spam exceeds 64k characters, and this is probably well below the
  false-negative SA rate, so it is not worth wasting time to check large mail.

  
remaining changes:

- new file README.exim (thanks to Jochen Erwied, Patrice Fournier
  and Igor D'Astolfo);

- updated README.postfix to describe how to avoid running header_checks,
  body_checks and dns_lookups in Postfix twice;

- put back file README.customize, which was omitted from amavisd-new-20020224
  by mistake (but available with previous version and on the web);

- added introductory paragraph to README, the rest still needs to be updated;

- added missing last-step check for '.' in hash lookups to match examples
  and make it more useful;

- added timeouts to certain tasks to make it better suitable for unattended
  operation (less, but still subject to certain DoS attacks, similar to
  the official amavisd). The main reason for adding timeouts is that we
  don't want spam checking to slow down amavisd operation too much: if it
  takes too long, just skip it (assume not spam) and move on;

- change unmangle_sender() to believe sender address for Klez viruses;
  this is sometimes wrong, but seems like people prefer to sacrifice
  few false accusations in favour of some warranted sender notifications.
  Feel free to hack this routine at will (and publish good ideas),
  it is intended to be modified;

- use Perl module Errno instead of errno.ph to avoid using broken
  file errno.ph on some popular platforms;

- make possible to have per-sender-domain administrator e-mail address
  for admin notifications (hash lookup %mailto, sub warn_admin() )

- relax temporary file/dir protection to allow them to be readable
  by the group. This makes possible for anti-virus (daemon) checker
  to run under a different user (but in the same group). Doing so should
  be safer as it makes impossible for virus-checking daemon to clobber
  files. It is only supported in the all-SMTP configuration. To use
  it with traditional amavis clients (e.g. milter), you will have to
  modify their sources to change umask and mods for file/dir they create;

- avoid changing sender address <> to <""> under certain circumstances
  (not strictly wrong, but still a bad idea);
  
- replaced header 'X-Razor-Warning:' with SA-compatible 'X-Spam-Status:' ;
  the 'X-Razor-id:' is still provided to facilitate user spam reporting;

- added '--' between options and argument when calling $sendmail_wrapper
  to be triply sure we avoid problems with some mailers (note that
  $sendmail_wrapper is only still needed in the sendmail setup,
  Postfix and Exim do not need it in the recommended configuration);

- tidy the %local_delivery_aliases mechanism for local delivery / quarantine;

- allow MIME::Parser to decode uuencoded parts, if it feels
  it can (and should) do it;

- write warnings from MIME::Parser to the log (at log level 1 or higher),
  instead of discarding them.


The package is available at the usual location at:

  http://www.ijs.si/software/amavisd/

where also the most up-to-date version of FAQ
and certain other fresh documentation files live.


---------------------------------------------------------------------------
                                                             April 24, 2002
amavisd-new-20020424 is primarily a maintenance release
to summarize one week's worth of experience with amavisd-new-20020418
and to implement some good ideas from the amavis-user mailing list.
It also brings one or two new features.

It is available at:
  http://www.ijs.si/software/amavisd/

or more specifically, at:
  http://www.ijs.si/software/amavisd/amavisd-new-20020424.tar.gz


Changes since amavisd-new-20020418:

- removed Perl 'my' declaration from configurable variables which prevented
  them from being changed in amavisd.conf (thanks to Sebastian Hagedorn
  and Wouter de Jong for reporting)

- introduced child timeouts to prevent bad amavis client from monopolizing
  a child forever (thanks to Sebastian Hagedorn for reporting the problem)

- supported and documented (in README.postfix) configuration where multiple
  remote or local SMTP-in/SMTP-out MTAs (e.g. Postfix) can use the same
  amavisd server, by making it deliver checked mail back to the same
  IP address it came from (see variable $relayhost_is_client in amavisd.conf)
  (thanks to Wouter de Jong for the splendid idea which fitted naturally
  into the overall scheme)

- in certain log messages include the SMTP-in and SMTP-out MTA IP address

- new access list checking for IP addresses - used to limit SMTP access
  to authorized MTAs only: @inet_acl; access control is now enabled by default

- slightly more sophisticated hash-based access list lookups, modeled after
  Postfix map lookups. The sequence now goes through the following steps:
    - hash lookup for user+foo@do.ma.in
    - hash lookup for user@do.ma.in  (only if $recipient_delimiter is nonempty)
    - hash lookup for do.ma.in
    - hash lookup for .do.ma.in
    - hash lookup for .ma.in
    - hash lookup for .in
    - hash lookup user+foo@
    - hash lookup user@  (only if $recipient_delimiter is nonempty)

- updated README.lookups

- cleaner quarantine code; new variable $mailfrom_quarantine allows
  to choose either the original envelope sender, or admin-specified
  fixed sender address; include a special per-user quarantine example
  (look for 'trouble-user-quarantine' in the amavisd)

- fixed problem with localized system error messages - use numeric errno
  instead of strings in Sophie and Trophie clients (thanks to Igor D'Astolfo
  for reporting the problem with Italianized version of Linux and for a good
  suggestion)

- successfully tested with Exim
  (thanks to Igor D'Astolfo for testing and for reporting
  the <<doubly-quoted>> sender (but not recipient!) problem)

- wrong variable used in the spam section ($final_virus_destiny
  instead of $final_spam_destiny) (thanks to Wayne Smith for reporting
  and for testing Vipul's Razor 'plugin' patch)

- changed pattern match in the ./av/oav to support new version of
  OpenAntiVirus ScannerDaemon (thanks to Rainer Link)

- changed 'configure' to make it recognize Sophie 1.33rc1
  (thanks to Igor D'Astolfo and Lars Hecking)

- include acinclude.m4/acx_pthread.m4 macro
  (thanks to Rainer Link)

- updated README.postfix: includes instructions on how to avoid body_checks
  and header_checks for reinserted mail (contributed by Wayne Smith,
  works nicely, requires Postfix version 1.1.7-20020331 or later)

- clarified comments in amavisd.conf

- optionally keep existing X-Virus-Scanned: header lines, or remove them
  before adding our own header line - see $remove_existing_x_scanned_headers
  (requested by Darryl Harvey)

- avoid historical (misleading) parameter name $localhost_ip;
  use $relayhost and $relayhost_port instead, but take old variables into
  account for backwards compatibility with existing amavisd.conf files

- Here is an overall picture (sequence of events)
  of how pieces fit together:

    bypass_checks? ==> PASS
    no viruses?    ==> PASS
    log virus     if $log_templ is nonempty
    quarantine    if $virus_quarantine_to is nonempty
    notify admin  if $warnadmin
    notify sender if $warnsender
    notify recips if $warnrecip
    final_destiny==pass?  ==> PASS
    virus_lovers?         ==> PASS
    DISCARD or REJECT (depending on final_*_destiny)


---------------------------------------------------------------------------
                                                             April 18, 2002
This it to announce the second release of amavisd-new-20020418,
available at:

  http://www.ijs.si/software/amavisd-new-20020418.tar.gz

It is a version of amavisd (a daemonized AMaViS, which is an interface
between MTA and virus scanners), based on amavisd CVS from today (20020418)
(same configuration, amavis clients in C, instructions, AV client code),
while also being a successor of the initial release of amavisd-new-20020329,
which is a performance-enhanced pre-forked Net::Server -based amavisd
with SMTP-in/SMTP-out capability, written in Perl.

(I tried to make this release based on amavisd-snapshot-20020300, but failed,
as that version still has $errval semantics bug (the bitwise-operations
problem was discussed on the AMaViS-user list some time ago).
This was fixed in the CVS version and most AV clients were changed then.
As the CVS version contains AV client code for new AV scanners, this makes
it incompatible with config stuff from amavisd-snapshot-20020300.

If you have problems with ./configure or make, these problems would be
common to both versions. Also the unpackers and decompressors code is mostly
the same for all recent amavisd versions, so any problems in this arena
(like DoS mishandling) are most likely to be common for both the official
amavisd and the amavisd-new version.

Compared to the first release of amavisd-new-20020329, the second release
brings further significant performance improvements especially in the
SMTP-in/SMTP-out configuration, e.g. with Postfix, but also brings
some interesting new features and new configuration possibilities.

No important bug fixes were needed, so upgrade at your leisure if you are
running my initial version. The Postfix users would perhaps want to rush
a bit though, to put new performance improvements into use.

The summary of changes since the initial version:

- significant SMTP-in speedups (25% with fast AV scanner), file reuse
- pass reject reason to MTA on the input side
- more informative MTA log entries in the SMTP-in/SMTP-out setup
- amavis internal id (am_id) in log entries and passed to MTA in SMTP response
- ISP features: specify subgroups of users who want to receive viruses
- address extensions: e.g. user@domain -> user+virus@domain if virus detected
- can specify final_virus_destiny: reject, discard, pass
- quarantine new options: save to individual file, save to mailbox, pass to MTA
- new headers in quarantined viruses preserve envelope addresses
  and quarantine id (similar to the suggestion from Furio Ercolessi - see code)
- detailed timing breakdown report for each passed message
- anti-spam hooks and examples, example patch to integrate Vipul's Razor client
- body cache now always enabled - heavy speedup for mailing list bursts
- Sophie 1.33-ready
- rewritten Trophie client, based on new Sophie client code
- rewritten README.postfix, describing new setup possibilities
- new file README.lookups (to be used with virus_lovers and bypass_checks)
- new file README.customize (same as in the initial version)
- new file README.performance (unfinished)
- new amavisd.conf options, documented there
- code heavily commented, cleaned, generalized again
- does not accuse innocent users of sending viruses if we are suspicious
  of sender address (see FAQ below) (same as in the initial version)

CAVEATS:

- no test mode;

- only Postfix and sendmail milter are fully supported and tested;
  to integrate with other mailers one would need some understanding of their
  operation to set up properly; contributions are welcome;

(P.S. note: this is probably a non-issue, related to the file system problem:
  All amavisd versions seem to share one still unresolved problem, probably
  with amavis-milter.c client, its use of libmilter, or perhaps even in the
  libmilter code itself - see recent thread on the AMaViS-user list
  (subject: Leftover email.txt files with amavisd 'standard' also).
  The problem is most pronounced in burst of heavy traffic.
)

FAQ:

- Net::Server 0.82 triggers a Perl 5.005 bug (the problem is obvious:
  you get syntax errors). Either upgrade to Net::Server 0.83, or
  upgrade your Perl - 5.6.x should be ok.

- if you intend to play with customized notifications, it is wise
  to remove the '-t' option from $sendmail_wrapper_args in amavisd.conf.
  That way you are free to screw up notification mail headers any way you want,
  and the message would still be delivered to the correct recipients.
  Removing '-t' is now the recommended setting, but both variants should work.
  If you use SMTP-out method for notifications (e.g. Postfix recommended
  and default setting), this does not affect you.

- if you see virus notifications claiming the virus originator is <?>
  or <?@some.domain> and sender notifications are not sent, this is not a bug,
  but a feature - see comments at the subroutine unmangle_sender().
  The original idea comes from Furio Ercolessi: as some viruses tend to use
  forged or corrupted sender or 'From:' addresses, we try to determine
  the true virus sender, and if we can not do that, we avoid accusing
  innocent users of sending viruses.

- if you kill or HUP amavisd, temporary directories may be left undeleted;
  this is normal and mails are not lost;

- if amavisd does not restart after receiving HUP, a possible reason may be
  that amavisd can not be found in the path as set in the $ENV{PATH} variable
  (near the beginning of amavisd program). Another reason may be a syntax
  error if you changed the amavisd.conf file. Try to start it manually:
    $ su vscan -c amavisd
  If that does not make you wiser, set $DEBUG = "yes" and retry.
  This is also the recommended first-time start method.

- after changing $inet_socket_bind in amavisd.conf, you must stop amavisd
  and start it anew. The HUP method causes amavisd to stumble over its feet.

Below is my announcement notice for the initial version of amavisd-new,
just slightly edited to remove some mistakes.


---------------------------------------------------------------------------
Date: Sat, 30 Mar 2002 04:13:25 +0100
From: Mark Martinec <Mark.Martinec@ijs.si>
Subject: [AMaViS-user] ANNOUNCE: new amavisd - leaner and meaner
To: amavis-user@lists.sourceforge.net
Message-id: <01KFYI1DJ2O200AMKT@CATHY.IJS.SI>

I would like to announce an updated version of amavisd,
based on recent CVS code of amavisd (which is not far
from the February amavisd snapshot). It is a result of my
three weeks work on the code, caused by our needs for:
- better reliability;
- higher throughput (less overhead);
- versatility (e.g. separating amavis and MTA hosts, load sharing),

Available at:
  http://www.ijs.si/software/amavisd-new-20020329.tar.gz

This is now finally a version which I can recommend to friends :)
TODO: better DoS handling in unpackers.

Main features - in brief:

- pre-forked reusable children - saving on process creations;
- persistent connections to certain AV scanners, e.g. Sophie, saving on forks;
- both SMTP and pipe (sendmail wrapper) interfaces independently available
  and configurable on all three sides: input, output, notifications;
- sendmail Milter interface supported and tested; Postfix supported and
  thoroughly tested (Exim untested; classical sendmail untested, no qmail);
- customizable notification messages;
- compatibility with existing configurations;
- cleanups, generalizations, speedups, fixes, better code documentation;
- HUP signal causes restart with new configuration;
- ISP feature: certain recipients may be allowed to receive viruses
  (with alert header line added (not with milter), notifications are
  still generated);
- anti-spam hooks, caching-ready, Vipul's Razor interface in a form
  of a patch included - should make integration with SpamAssassin easier.

Install:

  Unpack the tar over the checked-out CVS version of amavisd branch
  (or February snapshot). It overlays some files, the rest is unchanged.
  For milter interface make sure you use amavis-milter.c from CVS,
  as the snapshot version contains a bug which can cause message loss
  if amavisd dies.

Details:

- child and socket handling is now delegated to Perl module Net::Server,
  which gives us pre-forked children which are able to do more than one
  mail-check during their lifetime, saving on process creations
  and giving better response time;

- Net::Server controls number of children, does signal handling,
  takes care of dead children, handles listening on multiple sockets
  (both Unix and TCP/UDP), delegating tasks and synchronizing 'accept's;

- as a consequence, certain AV-scanner interfaces (most notably Sophie,
  and soon to follow Trophie) can keep persistent connections to the
  AV checking daemon, saving on AV scanner process creations
  (forks on accept), and socket setups/teardowns;

- on the input side: both SMTP (TCP) and traditional amavis client
  protocol (Unix socket) are now accepted - even both at the same time
  (by default) to ease transition to SMTP interface;

- new input-side SMTP interface (SMTP server) is easier to set up
  (no need to worry about file ownerships, UIDs and GIDs),
  and more versatile (e.g. SMTP responses carry more information
  that sysexits.h-based status codes, SMTP can talk to remote host);
  It is based on Perl native I/O and it is quite fast - no OO overhead,
  _not_ SMTP::Server-based (which I do not consider production quality);

- a nice by-product of SMTP input interface is the extra information
  available in MTA logs, e.g. Postfix log:
  postfix/smtp[7656]: DA7B147FA6:
    to=<user@domain>, relay=localhost[127.0.0.1], delay=5,
    status=sent (250 Ok, discarded - VIRUS: EICAR-AV-Test)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- notifications messages can be delivered either via SMTP or
  as traditionally via pipe to sendmail wrapper / MTA pickup.
  This means that in the Postfix setup for example, one needs
  to worry only about one output interface (SMTP or pipe).
  This also saves unnecessary work of checking the just-generated
  notification messages for viruses;

- the output pipe interface is now more careful with status
  checking and forking;

- output SMTP interface error handling more compact;

- sender notifications can be sent to more than one contact address
  (not used at the moment, but the mechanism is available);

- sender address unmangling patch incorporated (but I'm willing
  to take it out if considered inappropriate for the base distribution);

- see file ./amavis/mta/postfix_init for comments describing
  how to select SMTP- or pipe-based output interface
  (for re-injection and notifications);

- customizable notification messages and log entry text
  (see README.customize);

- clean notion of when mail addresses are in their quoted and
  when in unquoted form (RFC2821). All internal handling uses unquoted
  form, addresses get quoted as required by the output interface,
  and quoting gets stripped away as required by each input interface;

- besides traditional choices where a mail can be forwarded/accepted,
  discarded, or temporary failure/retry indicated, there is now
  a fourth choice: REJECT. Depending on the MTA this requires a
  message rejection to be done by the input side MTA itself.
  This comes handy in cases when amavisd accepts a non-infected mail,
  but outgoing MTA does not want to take it back for final delivery,
  e.g. in case of some policy violation. Traditionally amavisd
  would indicate temporary failure on its input side, causing the
  message to be retried and re-scanned over and over again,
  without having a chance of ever being accepted;

- all existing virus scanners are still there (one little change
  in each: replaced a call to do_virus with return 1);
  Sophie client rewritten to take advantage of persistent connections;

- decoding sections are mostly unchanged. This area needs more work
  in the future;

- includes exit status codes from sysexits.ph
  instead or having them hard-wired;

- anti-spam code easier to integrate into amavisd due to some code
  rearrangements; a patch to integrate Vipul's Razor is included
  as an example, but it may be better to tie amavisd with SpamAssassin.
  Anti-spam code will NOT be integrated into amavisd, but the least
  we can do is making it easier for people to add their own code;

- cache-ready (example in the included Razor patch). This works by
  calculating a message digest (hash, signature) of the message body
  and keeping it in storage for a short while, e.g. for the lifetime
  of a child process (10 consecutive requests by default).
  If another message with the same body content arrives in the near future
  we can skip a virus check. This comes handy where mailing list traffic
  is frequent, especially if we have to deal with poorly done mailing
  list managers or heavy spam traffic;

  To support this concept the functions of MIME decoding and
  unpacking of archives are now separated.

- MIME parsing is now supplied with our own Filer subclass.
  This was necessary to avoid MIME parser complaints when it tried
  to reconstruct file name extensions from file names in strange
  character sets. Its work was completely unnecessary and harmful,
  since we want to supply our own file names and do not care
  for file name extensions.

- as a consequence of our own Filer subclass, we now avoid the first
  (or the only) directory traversal (reading) in the first decoding pass;
  Not much, but every little bit counts.

- when using sendmail wrapper it is no longer necessary to supply
  the sendmail -t option. If -t is not specified, sendmail wrapper
  receives addresses via command arguments (exec, no shell),
  which is more reliable than having to parse mail headers - which are
  now more error-prone due to user-customizable notifications.
  The -t option is still supported though, but not recommended.

- do_unzip no longer complains with multi-line backtrace when it dies;
  just a single message is issued, like with other decoders;

- HUP signal causes restart and re-reading of config file;

- many new comments, code unifications, supplied some missing
  error checks, code generalizations;


MTA support:

- thoroughly tested with Postfix in all combinations of input,
  output, and notifications interfaces; in production use;

- big thanks to Sebastian Hagedorn who helped to test the milter
  interface on his Solaris!

- Exim and traditional sendmail interface untested, but should
  work without much work.

- Qmail interface is still missing, like in current CVS amavisd.
  I believe the amavis client for qmail can be written as
  a few-dozen line Perl program. Anybody want to try?


A quick cookbook on how to set up Postfix / amavisd interface
using SMTP on both amavisd input and output side (including notifications).
Amavisd by default now accepts both Unix socket and SMTP on the loopback
interface, so the transition is easier: first install new amavisd,
then at some other time change Postfix configuration (if desired) to:

master.cf:
  # MTA -> amavisd
  smtp-amavisd      unix - - n - 2 smtp

  # amavisd -> MTA
  localhost:10025   inet n - n - - smtpd -o content_filter=

main.cf:
  # choose transport to amavisd
  content_filter = smtp-amavisd:localhost:10024

[see new file README.postfix for details]

The amavisd can now easily be located on a different host than MTA,
also Postfix load-balancing transport methods can be used (e.g. multiple
MX records). Although amavisd now talks SMTP and incorporates some rudimentary
defenses against malicious SMTP clients, do not expose its SMTP server
directly to the world - always front-end it with MTA. By default it
binds to the local interface only, but other access restrictions are
also available.

Happy amavising!  Experiences and comments are most welcome.
