LOOKUP MAPS (hash, SQL) AND ACCESS LISTS EXPLAINED
==================================================
   Updated: 2002-04, 2002-06, 2002-11, 2002-12,
            2003-03, 2003-05, 2003-06, 2003-09, 2003-12,
            2004-01, 2004-03, 2003-06
   Mark Martinec <Mark.Martinec@ijs.si>

New with amavisd-new-20020630:    SQL LOOKUPS
New with amavisd-new-20021116:    Regular expression lookups
New with amavisd-new-20021227-p3: nonexistent SQL field now returns 0 or 1
                                  as declared by field type, or logs a warning

(applies to the semantics of amavisd.conf variables like:
   %virus_lovers,     %bypass_checks,
   @virus_lovers_acl, @bypass_checks_acl,
   $virus_lovers_re,  $bypass_checks_re,
   %local_domains, @local_domains_acl, %mailto, ... )

NOTE:
  All lookups are performed with raw (rfc2821-unquoted
  and unbracketed) addresses as a key,
      i.e.:    Bob "Funny" Dude@example.com
      not:     "Bob \"Funny\" Dude"@example.com
      and not: <"Bob \"Funny\" Dude"@example.com>


Several configurable settings in amavisd are controlled through the use
of table lookups (hash/associative array), access control lists (array),
Perl-regexp -based access control lists, SQL or LDAP lookups.
The subroutine that does all the lookups is:

  sub lookup($$@) {
    my($get_all, $addr, @tables) = @_;

It perform a lookup for a key (usually a recipient e-mail envelope address,
unless otherwise noted) against one or more lookup tables / maps.
The set of maps used to be hard-wired into the program (but no longer is),
and the order chosen is: from specific to more general, and from faster
to slower, which is usually flexible enough. Thus the default sequence
of lookups: SQL, LDAP, hash, ACL, regexp, constant. The first that returns
a definitive answer (not undef/NULL) stops the search.

The SQL and LDAP are somewhat specific and are always consulted first.
There can only be one (or none) SQL and one (or none) LDAP lookup.
This is an implementational limitation, and might be lifted some day.

The lists of static lookup tables are configurable since 20040701
(amavisd-new-2.0), and is controlled by array variables such as:

  @virus_admin_maps = (\%virus_admin, \$virus_admin);
  @viruses_that_fake_sender_maps = (\$viruses_that_fake_sender_re, 1);
  @spam_kill_level_maps = (\$sa_kill_level_deflt);
  @local_domains_maps =
    (\%local_domains, \@local_domains_acl, \$local_domains_re);
  @bypass_virus_checks_maps =
    (\%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
  @virus_lovers_maps =
    (\%virus_lovers, \@virus_lovers_acl, \$virus_lovers_re);

See amavisd.conf-default for a complete list of these @*_maps variables.
The above example shows that the default value of these arrays exactly
corresponds to the formerly hard-wired search order. Users are free to
leave these @*_maps variables at their default, referencing the legacy
variables, or the list can be replaced entirely. There may be any number
of lookup tables of any static type specified in these lists. Some restrain
is warranted nevertheless for efficiency reasons - one lookup into larger
lookup table is ofter quicker than two lookups into smaller ones.

Some lookup maps can only return boolean result (e.g. ACL), other maps
may return any value, which can be interpreted as boolean, numeric, string
or possibly other. The result of some lookup maps (e.g. regexp) may include
pieces of lookup key.

If a match is found, the subroutine lookup() returns whatever the map
returns;  undef is returned if nothing matches (which for Perl is false
as well).


A CONSTANT

Specifying a Perl scalar as an argument to lookup() is a degenerate
case of a lookup table: it matches any key, and the value of the
scalar is returned as the match value.

Specifying a scalar argument in a call to lookup() (e.g. as the last element
in @*_maps arrays) is useful as a last-resort (catchall, default) value.

One level of indirection is alowed, so the following three cases are
equivalent:
  $sa_kill_level_deflt = 6.0;
  @spam_kill_level_maps = (\%some_hash, \$sa_kill_level_deflt);

and:
  $sa_kill_level_deflt = 6.0;
  @spam_kill_level_maps = (\%some_hash, $sa_kill_level_deflt);

and:
  @spam_kill_level_maps = (\%some_hash, 6.0);

The first case allows for the value of a scalar variable to be assigned
even _after_ the assignment to @*_maps, so this still works as expected:
  @spam_kill_level_maps = (\%some_hash, \$sa_kill_level_deflt);
  $sa_kill_level_deflt = 6.0;

but the following does not (it uses a value in the scalar variable
at the time of assignment to the list, which is most likely not 6.0):
  @spam_kill_level_maps = (\%some_hash, $sa_kill_level_deflt);
  $sa_kill_level_deflt = 6.0;


HASH LOOKUPS (associative array lookups)

For arguments to subroutine lookup() of type hash-ref, the argument
is passed to subroutine lookup_hash(), which does a lookup into
a Perl hash.

Hash lookups (e.g. for user+foo@sub.example.com) are performed in the
following order:
 - lookup for user+foo@sub.example.com
 - lookup for user@sub.example.com (only if $recipient_delimiter nonempty)
 - lookup for user+foo@
 - lookup for user@  (only if $recipient_delimiter is nonempty)
 - lookup for sub.example.com
 - lookup for .sub.example.com
 - lookup for .example.com
 - lookup for .com
 - lookup for .

The search sequence stops as soon as a match is found, and the value
of the matched entry determines the result.

The domain part is always matched case-insensitively, the localpart
is matched case-sensitively when $localpart_is_case_sensitive
is true (not case-sensitive by default).

A field value undef implies that the next lookup table (if there are more)
is to be tried. In plain words, undef means "this table does not know
the answer, try the next one". Further searching in this table
(for possibly more general defaults) is terminated.

NOTE: a null reverse path e-mail address used by MTA for delivery status
notifications (DSN) has empty local part and empty domain. As far as the
lookup is concerned (which uses raw, i.e. non-quoted and non-bracketed
address form), this address is @, i.e. a single character "@".
The lookup_hash for null address goes through the following sequence
of keys: "@", "", "." (double quotes added for clarity, they are not part
of the key).

There is a subroutine read_hash() available for use in amavisd.conf.
It can read keys from a plain text file, and load them into a Perl hash.
Format of the text file: one address per line, anything from '#' to the end
of line is treated as a comment, but '#' within correctly quoted rfc2821
addresses is not treated as a comment (e.g. a hash sign within
"strange # \"foo\" address"@example.com is valid). Leading and trailing
whitespace is discarded, empty lines (containing only whitespace and comment)
are ignored. Addresses are converted from quoted form into internal (raw)
form and inserted as keys into a given hash, with a value of 1 (true).
The $hashref argument is returned for convenience, so that one can say
for example:
  $per_recip_whitelist_sender_lookup_tables = {
    '.my1.example.com' => read_hash({},'/var/amavis/my1-example-com.wl'),
    '.my2.example.com' => read_hash({},'/var/amavis/my2-example-com.wl') }



LIST LOOKUPS (ACL)

For arguments to subroutine lookup() of type array-ref, the argument
is passed to subroutine lookup_acl(), which does an access list lookup:

  sub lookup_acl($$) {
      my($addr, $acl_ref) = @_;

Domain name of the supplied address is compared with each member of the
access list in turn, the first match wins (terminates the search),
and its value decides whether the result is true (yes, permit, pass)
or false (no, deny, drop). Falling through without a match
produces undef. Search is case-insensitive.

Acl lookup is not aware of address extensions and they are not
handled specially.

If a list member contains a '@', the full e-mail address is compared,
otherwise if a list member has a leading dot, the domain name part is
matched only, and the domain as well as its subdomains can match. If there
is no leading dot, the domain must match exactly (subdomains do not match).

The presence of character '!' prepended to the list member decides
whether the result will be true (without a '!') or false (with '!')
in case this list member matches.

Because search stops at the first match, it only makes sense
to place more specific patterns before the more general ones.

Although not a special case, it is good to remember that '.' always matches,
so a '.' would stop the search and return true, whereas '!.' would stop
the search and return false (0).

Examples:

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@me.ac.uk' matches me.ac.uk, returns true and search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@you.ac.uk' matches .ac.uk, returns false (because of '!'), search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@them.co.uk' matches .uk, returns true and search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@some.com' does not match anything, falls through and returns undef

given: @acl = qw( me.ac.uk !.ac.uk .uk !. )
  'u@some.com' similar to the previous, except it returns 0 instead of undef,
  which would only make a difference if this ACL is not the last argument
  in a call to lookup()

given: @acl = qw( me.ac.uk !.ac.uk .uk . )
  'u@some.com' matches catchall ".", and returns true

more complex example: @acl = qw(
  !The.Boss@dept1.xxx.com .dept1.xxx.com
  .dept2.xxx.com .dept3.xxx.com lab.dept4.xxx.com
  sub.xxx.com !.sub.xxx.com
  me.d.aaa.com him.d.aaa.com !.d.aaa.com .aaa.com
);


Comparing hash (associative array) and ACL:

For smaller sets of keys and if only boolean results are needed,
both hash and ACL are appropriate.

- hash is still effective for lots of keys, ACL search is linear;
- hash can be read from a file;
- hash can return any value, not just true or false;
- hash can strip away address extension, ACL can not;

- ACL is simpler and appears more obvious for smaller sets;
- ACL can accommodate arbitrarily nested if-then-elseif-then-...-else cases
  whereas hash only follows a fixed order of stripping addresses;


ACL FOR IP ADDRESSES

A special type of lookup is an IP-matching access list implemented by
lookup_ip_acl(). It performs a lookup for an IP address against
access control list of IPv4 networks. It is used by amavisd to check
if the SMTP client (normally your MTA) is allowed to connect - the variable
is @inet_acl .

IP address is compared with each member of the access list in turn,
the first match wins (terminates the search), and its value decides
whether the result is true (yes, permit) or false (no, deny).
Falling through without a match produces false (undef).

The presence of character '!' prepended to the list member decides
whether the result will be true (without a '!') or false (with '!')
in case this list member matches and terminates the search.

Because search stops at the first match, it only makes sense
to place more specific patterns before the more general ones.

Network can be specified in classless notation n.n.n.n/k, or using
a mask n.n.n.n/m.m.m.m . Missing mask implies /32, i.e. a host address.

Although not a special case, it is good to remember that '0/0' always matches.

NOTE: IPv4 syntax is assumed, IPv6 is not currently supported.

Example
  given: @inet_acl = qw( !192.168.1.12 172.16.3.3 !172.16.3/255.255.255.0
                         10/8 172.16/12 192.168/16 );
matches rfc1918 private address space except host 192.168.1.12
and net 172.16.3/24 (but host 172.16.3.3 within 172.16.3/24 still matches)

More examples at amavisd.conf.


REGULAR EXPRESSION LOOKUPS

For arguments to subroutine lookup() of type Amavis::Lookup::RE
(objects), the object is passed to method lookup_re, which does a
lookup into a list of Perl regular expressions (regexp or RE for short).

The full unmodified e-mail address is always used, so splitting to localpart
and domain or lowercasing is NOT performed. The regexp is powerful enough
that this is unnecessary. The routine is useful for other RE tests, such as
looking for banned file names.

Each element of the list can be a ref to a pair, or directly a regexp
('Regexp' object created by qr operator, or just a (less efficient)
string containing a regular expression). If it is a pair, the first
element is treated as a regexp, and the second provides a return value
in case the regexp matches. If not a pair, the implied result value
of a match is 1.

The regular expression is taken as-is, no implicit anchoring or setting
case insensitivity is done, so use qr'(?i)^user@example\.com$',
and not a sloppy qr'user@example.com', which can easily backfire.
Also, if qr is used with a delimiter other than ', make sure to quote
the @ and $ .

The pattern allows for capturing of parenthesized substrings, which can
then be referenced from the result string using the $1, $2, ... notation,
as with the Perl m// operator. The number after the $ may be a multi-digit
decimal number. To avoid possible ambiguity the ${n} or $(n) form may be used.
Substring numbering starts with 1. Nonexistent references evaluate to empty
strings. If any substitution is done, the result inherits the taintedness
of the key. Keep in mind that the $ character needs to be backslash-quoted
in qq() strings. Example:
  $virus_quarantine_to = new_RE(
    [ qr'^(.*)@example\.com$'i => 'virus-${1}@example.com' ],
    [ qr'^(.*)(@[^@]*)?$'i     => 'virus-${1}${2}' ] );

Example (equivalent to the example in lookup_acl):
   $acl_re = new_re->new(
                 qr'@me\.ac\.uk$'i,
                 [ qr'[@.]ac\.uk$'i => 0 ],
                 qr'\.uk$'i,
             );
   ($r,$k) = $acl_re->lookup_re('user@me.ac.uk');
or $r = lookup('user@me.ac.uk', $acl_re);

'user@me.ac.uk'   matches me.ac.uk, returns true and search stops
'user@you.ac.uk'  matches .ac.uk, returns false (because of =>0) and search stops
'user@them.co.uk' matches .uk, returns true and search stops
'user@some.com'   does not match anything, falls through and returns false (undef)

NOTE: new_re is a synonym (shorthand) for the
      internal subroutine Amavis::Lookup::RE::new

See Perl documentation (or Google the Internet) for the description
of Perl regular expressions. They are just enhanced version of Posix regular
expressions, i.e. what your egrep, awk and sed thrive on. Here are the
most important constructs (simplified):

    .   Match any character                 inter..t
    |   Alternation                         alfa|beta|gamma
    ()  Grouping                            (pre|post)fix
    []  Character class                     [Aa]lfa[0-9]
    ^   Match the beginning of the string   ^MakeMoney
    $   Match the end of the string         com$
    \   Quote the next metacharacter        \.com$
                                            ^\$\$\$\+spam@\[127\.0\.0\.1\]$
    most other characters just match themselves

quantifiers may be placed after the pattern to modify its meaning
from 'match itself exactly once' into:
    *      Match 0 or more times            ^alfa.*omega$
    +      Match 1 or more times            alfa +beta
    ?      Match 1 or 0 times               (first)?aid
    {n}    Match exactly n times            0{6}
    {n,}   Match at least n times           !{3,}
    {n,m}  Match at least n but not more than m times


SQL LOOKUPS

For arguments to subroutine lookup() of type Amavis::Lookup::SQLfield
(objects), the object is passed to method lookup_sql_field, which does
a lookup into a SQL table by using Perl module DBI.

The amavisd.conf variable @lookup_sql_dsn controls access to the SQL
server (dsn = data source name). If the list @lookup_sql_dsn is empty,
no attempts to use SQL will be made, and no code to use DBI will be
loaded or compiled.

@lookup_sql_dsn variable is a list of triples: (dsn,user,passw).
More than one triple may be specified to list multiple (backup)
SQL servers - the first that responds will be used.

@lookup_sql_dsn =
   ( ['DBI:mysql:mail:host1', 'some-username1', 'some-password1'],
     ['DBI:mysql:mail:host2', 'some-username2', 'some-password2'] );

With PostgreSQL the dsn (first element of the triple) may look like:
      'DBI:Pg:host=host1;dbname=mail'

See man page for the Perl module DBI, and corresponding DBD module
man pages (DBD::mysql, DBD::Pg, ...) for syntax of the first argument,


SQL 'select' requests all available fields from the specified
tables, and the result is cached (just for this mail message processing).
Individual fields can be extracted one at a time from this cache
very quickly, so there is no penalty in using several calls to lookup
for different fields (for the same key) in different parts of the program.

lookup_sql() performs a lookup for an e-mail address against a SQL map.
If a match is found it returns whatever the map returns (a reference
to a hash containing values of requested fields), otherwise returns undef.
A match aborts further fetching sequence.

lookup_sql_field() also performs a lookup for an e-mail address against
a SQL map. It first calls lookup_sql() if it hasn't been called yet for
this key, but instead of returning all available fields, it returns
just a value of one particular table field. This is the subroutine
that gets called from lookup() for arguments (objects) of type
Amavis::Lookup::SQLfield.

The field value NULL is translated to Perl undef, which according
to lookup rules implies that the next lookup table (if there are more)
is to be tried. In plain words, NULL means "this table does not know
the answer, try the next one". Further searching in this table
(e.g. for more general defaults) is terminated.

Boolean fields are usually declared as a single character (instead of
integer) to minimize storage. The characters N,n,F,f,0,NUL and SPACE
represent false (0), any other character represents true. Trailing blanks
are ignored. It is customary to use Y for true and N for false.

SQL lookups (e.g. for user+foo@example.com) are performed in order
which is normally requested by 'ORDER BY...DESC' in the SELECT statement;
otherwise the order is unspecified, which is only useful if just specific
entries exist in a database (e.g. full address always, not only domain part
or mailbox part).

The following order (implemented by sorting on the 'priority' field
in DESCending order, zero is low priority) is recommended, to follow
the same specific-to-general principle as in other lookup tables:

 - lookup for user+foo@example.com
 - lookup for user@example.com (only if $recipient_delimiter nonempty)
 - lookup for user+foo (only if domain part is local)
 - lookup for user     (only local; only if $recipient_delimiter is nonempty)
 - lookup for @example.com
 - lookup for @.       (catchall)

NOTE:
 this is different from hash and ACL lookups in three important aspects:
   - key without '@' implies mailbox name, not domain name;
   - subdomains are not looked at, only full domain names are matched;
   - the naked mailbox name lookups ('user') are only performed
     when the address matches local_domains lookups.

The domain part is always lowercased when constructing a key,
the localpart is not lowercased when $localpart_is_case_sensitive is true.

Table names and field names are hard-wired in the routine
prepare_sql_queries(). Please adjust it to will. Field names should
be unique even without the table prefix. If they are not, the last one
in the SELECT field list prevails.

Below is an example database that can be used with MySQL or PostgreSQL
to test the code.

-- PostgreSQL notes (by Phil Regnauld):
--   - remove the 'unsigned' throughout,
--   - in tables 'users', 'mailaddr' and 'policy':
--     use SERIAL instead of INT...AUTO_INCREMENT when declaring id, e.g.:
--       id serial NOT NULL,
--   - create an amavis username and the database (choose name, e.g. mail)
--       $ creatuser -U pgsql --no-adduser --createdb amavis
--       $ createdb -U amavis mail
--   - populate the database using the schema below:
--       $ psql -U amavis mail < amavisd-pg.sql


-- local users
CREATE TABLE users (
  id         int unsigned NOT NULL auto_increment,
  priority   int          NOT NULL DEFAULT '7',  -- 0 is low priority
  policy_id  int unsigned NOT NULL DEFAULT '1',
  email      varchar(255) NOT NULL,
  fullname   varchar(255) DEFAULT NULL,    -- not used by amavisd-new
  local      char(1),     -- Y/N  (optional field, see note further down)
  PRIMARY KEY (id),
  KEY email (email)
);
CREATE UNIQUE INDEX users_idx_email ON users(email);

-- any e-mail address, external or local, used as senders in wblist
CREATE TABLE mailaddr (
  id         int unsigned NOT NULL auto_increment,
  priority   int          NOT NULL DEFAULT '7',  -- 0 is low priority
  email      varchar(255) NOT NULL,
  PRIMARY KEY (id),
  KEY email (email)
);
CREATE UNIQUE INDEX mailaddr_idx_email ON mailaddr(email);

-- per-recipient whitelist and/or blacklist,
-- puts sender and recipient in relation wb  (white or blacklisted sender)
CREATE TABLE wblist (
  rid        int unsigned NOT NULL,  -- recipient: users.id
  sid        int unsigned NOT NULL,  -- sender:    mailaddr.id
  wb         varchar(10)  NOT NULL,  -- W or Y / B or N / space=neutral / score
  PRIMARY KEY (rid,sid)
);

CREATE TABLE policy (
  id               int unsigned NOT NULL auto_increment,
  policy_name      varchar(32),     -- not used by amavisd-new

  virus_lover          char(1),     -- Y/N
  spam_lover           char(1),     -- Y/N  (optional field)
  banned_files_lover   char(1),     -- Y/N  (optional field)
  bad_header_lover     char(1),     -- Y/N  (optional field)

  bypass_virus_checks  char(1),     -- Y/N
  bypass_spam_checks   char(1),     -- Y/N
  bypass_banned_checks char(1),     -- Y/N  (optional field)
  bypass_header_checks char(1),     -- Y/N  (optional field)

  spam_modifies_subj   char(1),     -- Y/N  (optional field)

  virus_quarantine_to      varchar(64) default NULL,   -- (optional field)
  spam_quarantine_to       varchar(64) default NULL,   -- (optional field)
  banned_quarantine_to     varchar(64) default NULL,   -- (optional field)
  bad_header_quarantine_to varchar(64) default NULL,   -- (optional field)

  spam_tag_level  float default NULL,  -- higher score inserts spam info headers
  spam_tag2_level float default NULL,  -- inserts 'declared spam' header fields
  spam_kill_level float default NULL,  -- higher score activates evasive actions, e.g.
                                       -- reject/drop, quarantine, ...
                                     -- (subject to final_spam_destiny setting)
  spam_dsn_cutoff_level float default NULL,            -- (optional field)

  addr_extension_virus      varchar(64) default NULL,  -- (optional field)
  addr_extension_spam       varchar(64) default NULL,  -- (optional field)
  addr_extension_banned     varchar(64) default NULL,  -- (optional field)
  addr_extension_bad_header varchar(64) default NULL,  -- (optional field)

  PRIMARY KEY (id)
);

INSERT INTO users VALUES ( 1, 9, 5, 'user1+foo@y.example.com','Name1 Surname1', 'Y');
INSERT INTO users VALUES ( 2, 7, 5, 'user1@y.example.com', 'Name1 Surname1', 'Y');
INSERT INTO users VALUES ( 3, 7, 2, 'user2@y.example.com', 'Name2 Surname2', 'Y');
INSERT INTO users VALUES ( 4, 7, 7, 'user3@z.example.com', 'Name3 Surname3', 'Y');
INSERT INTO users VALUES ( 5, 7, 7, 'user4@example.com',   'Name4 Surname4', 'Y');
INSERT INTO users VALUES ( 6, 7, 1, 'user5@example.com',   'Name5 Surname5', 'Y');
INSERT INTO users VALUES ( 7, 5, 0, '@sub1.example.com', NULL, 'Y');
INSERT INTO users VALUES ( 8, 5, 7, '@sub2.example.com', NULL, 'Y');
INSERT INTO users VALUES ( 9, 5, 5, '@example.com',      NULL, 'Y');
INSERT INTO users VALUES (10, 3, 8, 'userA', 'NameA SurnameA anywhere', 'Y');
INSERT INTO users VALUES (11, 3, 9, 'userB', 'NameB SurnameB', 'Y');
INSERT INTO users VALUES (12, 3,10, 'userC', 'NameC SurnameC', 'Y');
INSERT INTO users VALUES (13, 3,11, 'userD', 'NameD SurnameD', 'Y');
INSERT INTO users VALUES (14, 3, 0, '@sub1.example.net', NULL, 'Y');
INSERT INTO users VALUES (15, 3, 7, '@sub2.example.net', NULL, 'Y');
INSERT INTO users VALUES (16, 3, 5, '@example.net',      NULL, 'Y');
INSERT INTO users VALUES (17, 7, 5, 'u1@example.org',    'u1', 'Y');
INSERT INTO users VALUES (18, 7, 6, 'u2@example.org',    'u2', 'Y');
INSERT INTO users VALUES (19, 7, 3, 'u3@example.org',    'u3', 'Y');

-- INSERT INTO users VALUES (20, 0, 5, '@.',             NULL, 'N');  -- catchall

INSERT INTO policy VALUES (1, 'Non-paying',    'N','N','N','N',  'Y','Y','Y','N',  'Y',NULL,NULL,NULL,NULL, 3.0,   7, 10,   NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (2, 'Uncensored',    'Y','Y','Y','Y',  'N','N','N','N',  'N',NULL,NULL,NULL,NULL, 3.0, 999, 999,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (3, 'Wants all spam','N','Y','N','N',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0, 999, 999,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (4, 'Wants viruses', 'Y','N','Y','Y',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0, 6.9, 6.9,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (5, 'Normal',        'N','N','N','N',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0, 6.9, 6.9,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (6, 'Trigger happy', 'N','N','N','N',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0,   5, 5,    NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (7, 'Permissive',    'N','N','N','Y',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0,  10, 20,   NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (8, '6.5/7.8',       'N','N','N','N',  'N','N','N','N',  'N',NULL,NULL,NULL,NULL, 3.0, 6.5, 7.8,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (9, 'userB',         'N','N','N','Y',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 3.0, 6.3, 6.3,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (10,'userC',         'N','N','N','N',  'N','N','N','N',  'N',NULL,NULL,NULL,NULL, 3.0, 6.0, 6.0,  NULL, NULL,NULL,NULL,NULL);
INSERT INTO policy VALUES (11,'userD',         'Y','N','Y','Y',  'N','N','N','N',  'N',NULL,NULL,NULL,NULL, 3.0,   7, 7,    NULL, NULL,NULL,NULL,NULL);

INSERT INTO policy VALUES (12,'4 subfolders',  'N','N','N','N',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 2.3, 6.33, 6.73, 9.3, 'virus','spam','banned','badh');
INSERT INTO policy VALUES (13,'2 subfolders',  'N','N','N','N',  'N','N','N','N',  'Y',NULL,NULL,NULL,NULL, 2.3, 6.33, 6.73, 9.3, 'caution','spam','caution','caution');
INSERT INTO policy VALUES (14,'next', NULL,NULL,NULL,NULL, NULL,NULL,NULL,NULL, NULL, NULL,NULL,NULL,NULL, NULL,NULL,NULL,NULL, NULL,NULL,NULL,NULL);


-- sender envelope addresses needed for white/blacklisting
INSERT INTO mailaddr VALUES ( 1, 5, '@example.com');
INSERT INTO mailaddr VALUES ( 2, 9, 'owner-postfix-users@postfix.org');
INSERT INTO mailaddr VALUES ( 3, 9, 'amavis-user-admin@lists.sourceforge.net');
INSERT INTO mailaddr VALUES ( 4, 9, 'makemoney@example.com');
INSERT INTO mailaddr VALUES ( 5, 5, '@example.net');
INSERT INTO mailaddr VALUES ( 6, 9, 'spamassassin-talk-admin@lists.sourceforge.net');
INSERT INTO mailaddr VALUES ( 7, 9, 'spambayes-bounces@python.org');

-- whitelist for user 14, i.e. default for recipients in domain sub1.example.net
INSERT INTO wblist VALUES (14, 1, 'W');
INSERT INTO wblist VALUES (14, 3, 'W');

-- whitelist and blacklist for user 17, i.e. u1@example.org
INSERT INTO wblist VALUES (17, 2, 'W');
INSERT INTO wblist VALUES (17, 3, 'W');
INSERT INTO wblist VALUES (17, 6, 'W');
INSERT INTO wblist VALUES (17, 7, 'W');
INSERT INTO wblist VALUES (17, 5, 'B');
INSERT INTO wblist VALUES (17, 4, 'B');

-- $sql_select_policy setting in amavisd.conf tells amavisd
-- how to fetch per-recipient policy settings.
-- See comments there. Example:
--
-- SELECT *,users.id FROM users,policy
--   WHERE (users.policy_id=policy.id) AND (users.email IN (%k))
--   ORDER BY users.priority DESC;
--
-- $sql_select_white_black_list in amavisd.conf tells amavisd
-- how to check sender in per-recipient whitelist/blacklist.
-- See comments there. Example:
--
-- SELECT wb FROM wblist,mailaddr
--   WHERE (wblist.rid=?) AND (wblist.sid=mailaddr.id) AND (mailaddr.email IN (%k))
--   ORDER BY mailaddr.priority DESC;



Special handling of optional SQL field 'users.local'
----------------------------------------------------

A special shorthand is provided when SQL lookups are used: when a match
for recipient address (or domain) is found in SQL tables (regardless of
field values), the recipient is considered local, regardless of static
@local_comains_acl or %local_domains lookup tables. This simplifies
life when a large number of dynamically changing domains is hosted.
To overrule this behaviour, add an explicit boolean field 'local'
to table 'users' (missing field defaults to true, meaning record match
implies locality; a NULL field 'local' is not special, it means undef
as with other fields, causing search to continue into other lookup tables).

Since amavisd-new-20030616-p7:
changed the default value for local_domains_sql lookup for the catchall
key '@.' under conditions: when user record with key '@.' is present in
the database and a field 'local' is not present. Previously it surprisingly
defaulted to true, now it falls back to static lookup table defaults,
the same as if the record '@.' were not present in the table or as if
the field value 'local' was NULL.

Case sentitivity of string comparison
-------------------------------------

Amavisd-new expects string comparison to be case sensitive (but does not
mind if it isn't). When forming a SELECT clause it lowercases parts of
keys that are supposed to be case-insensitive, such as the domain name.
The local part of the e-mail address in SQL search keys is lowercased
if and only if the $localpart_is_case_sensitive variable is true (which
is a default).

This means that case-insensitive parts of e-mail addresses as kept in the
SQL database should be in lower case, otherwise match may fail, depending
on SQL server type and the use of BINARY prefix in string data types. 

Since MySQL version 3.23.0 it is possible to declare a data type of
a column as BINARY, forcing string comparision to be case sensitive,
as it is in PostgreSQL. This is only required for sites that want to treat
localpart as case-sentitive and have $localpart_is_case_sensitive true.


LDAP LOOKUPS

LDAP lookups are enabled in amavisd.conf with:

$enable_ldap  = 1;

Definitions and default values of LDAP parameters.

  hostname      : The hostname of the LDAP server we connect to.
                  May also be a reference to an array of hosts, each
                  will be tried in order until a connection is made.
                  (Default = 'localhost')
  port          : The port where LDAP sends queries. (Default = 389)
  timeout       : Timeout (in sec) passed when connecting the remote
                  server. (Default = 120)
  tls           : Enable TLS/SSL if true. (Default = 0)
  base          : The DN that is the base object entry relative to
                  which the search is to be performed. (Default = undef)
  scope         : Scope can be 'base', 'one' or 'sub'. (Default = 'sub')
  query_filter  : The filter used to find the amavis account. The string
                  must contain a '%m' token that will be replaced by the
                  actual e-mail address.
                  (Default = '(&(objectClass=amavisAccount)(mail=%m))')
  bind_dn       : If binding is needed, this is where you specify the
                  DN to bind as. (Default = undef)
  bind_password : Binding password. (Default = undef)

The desired parameters can be specified in amavisd.conf and defaults
will be supplied for any parameters not specified, ex:

$default_ldap = {
  hostname      => [ 'localhost', 'ldap2.example.com' ],
  timeout       => 5,
  tls           => 0,
  base          => ou=People,dc=example,dc=com,
  query_filter  => '(&(objectClass=amavisAccount)(mail=%m))',
};

The amavisd-new LDAP schema is available in file LDAP.schema of the
distribution, and at http://www.ijs.si/software/amavisd/LDAP.schema

LDAP 'search' requests all available fields from the specified directory
and the result is cached (just for this mail message processing).
Individual attributes can be extracted one at a time from this cache
very quickly, so there is no penalty in using several calls to lookup
for different attributes (for the same key) in different parts of the
program.

lookup_ldap() performs a lookup for an e-mail address in an LDAP
directory. If a match is found it returns whatever the map returns
(a reference to a hash containing values of requested attributes),
otherwise returns undef. A match aborts further fetching sequence.

lookup_sql_attr() also performs a lookup for an e-mail address against
a LDAP directory. It first calls lookup_ldap() if it hasn't been called
yet for this key, but instead of returning all available attributes,
it returns just a value of one particular attribute. This is the
subroutine that gets called from lookup() for arguments (objects) of
type Amavis::Lookup::LDAPattr.

LDAP white/black listing
------------------------

amavisWhitelistSender/amavisBlacklistSender are multivalued attributes
containing either full email addresses or domain specifications. The
envelope sender address is compared against each attribute value until
a match is made.

amavisBlacklistSender: user@example.com
amavisBlacklistSender: @example.com      
amavisBlacklistSender: @.example.com

A domain specification with a leading '@.' matches a domain as well
as its subdomains.

Special handling of optional LDAP attribute 'amavisLocal'
---------------------------------------------------------

A special shorthand is provided when LDAP lookups are used: when a match
for recipient address (or domain) is found in LDAP tables (regardless of
attribute values), the recipient is considered local, regardless of static
@local_domains_acl or %local_domains lookup tables. This simplifies
life when a large number of dynamically changing domains is hosted.
To overrule this behaviour, add an explicit boolean attribute 'amavisLocal'
(missing field defaults to true, meaning record match implies locality)
The default value for local_domains_ldap lookup for the catchall key '@.'
under conditions: when user record with key '@.' is present in the database
and the attribute 'amavisLocal' is not present. Previously it surprisingly
defaulted to true, now it falls back to static lookup table defaults, the
same as if the record '@.' were not present in the table.

In general LDAP lookups are similar to SQL lookups except for the low level
LDAP/SQL specific code. The overall funtionality, lookup rules, etc. are
identical.
