History |
Using SMTP mail rules to handle spam
Introducing SMTP mail rules
 CautionYou should have a solid understanding of filter documents and basic coding to configure SMTP mail rules.
SMTP mail rules let you:
• replace all formerly built-in Internet Services spam handling rules
• inject an X-SPAM-Warning Internet header that rates the message as one of: HIGH, MEDIUM, LOW
• inject an X-SPAM-Level header that provides a numeric indication of the likelihood that the file is spam
• inject an X-SPAM-Tests header that shows which spam tests this message failed
• control rules by variables (set, by convention, at the top of the rules file)
• control handling by rules (located, by convention, at the bottom of the rules file)
• implement various SMTP crossposting and Bcc limits.
You can run mail rules:
• before any headers are received
• as each header is received
• when the end of the header indicator is reached
• as each HTML address and image link is processed
• as each attachment header is received
• at the end of each set of attachment headers
• as each HTML address and image link is processed
• when the end of the message body is reached
• when the end of the message is reached.
All SMTP mail rules files are located in the Filters folder located in the Internet Services folder on the administrator's Desktop. They are distinguished from the standard filter files by their names, which must start with the word "rules". You can use these files to customize how your site handles potential spam and junk mail. By default, Internet Services installs these four rules files:
• rules.AttachmentBlock
• rules.SubjectBlock
• rules.SMTPDebug
• rules.MailRules.
The anatomy of an SMTP mail rule
Each SMTP mail rule takes the same basic form:
• the header to perform a test on
• the test to perform
• the action to take if the test is successful.
A colon (:) always separates the header from the test
Here is an example of a simple mail rule:
X-Mailer: "Millennium Mailer" SET $spamlevel +=75
where
• X-Mailer is the header on which to execute the rule
• the test to perform is, whether the X-Mailer header contains the data Millennium Mailer
• the action to perform, in this case, is to add 75 to the $spamlevel variable
The header
The header section of a mail rule determines which header field in the envelope a rule should be run on. For example, To, From, Subject, Date, Return-Path, X-headers, and so on. This is what the different characters in the Header mean:
The test
The test section is the test to perform on the specific header. The tests in the rules scripting language generally fall into three categories:
• simple expressions
• regular expressions
• conditional expressions
Simple expressions
A simple expression is a string in quotes ("") that means does this string occur in the data of the RFC-2822 header. Strings are not case sensitive. A string can contain a simple wildcard character, such as
• a question mark "?", which means match any single character
• an asterisk "*", which means match any group of characters
The quoted string can also have a NOT in front of it, which means reverse the sense of the test. Let's take a look at what would happen if the following RFC-2822 header was processed by the following rules:
Date: Tue, 11 Feb 2003 16:27:41 -0500
• Date: "Feb 2003" spam
Condition is true and message would be marked as spam.
• Date: "Tue, 11 Feb 2003 16:27:41 -0500" spam
Condition is true and message would be marked as spam.
• Date:NOT "200?" spam
Condition is false (not true) and message would not be marked as spam.
• Date:"*Feb*" spam
Condition is true and message would be marked as spam.
Regular expressions and extended regular expressions (regexp and eregexp)
A regular expression is a common way of describing string pattern matching. Regular expressions are usually found in Unix environments and allow a more granular level of control over matching than simple expressions. They also allow the data matched to be stored in a variable for later use. Internet Services also supports extended regular expressions (eregexp), however, regular expressions are at least twice as fast for equivalent searches.
Example 1
From: regexp:".*@[0-9]+\\..*" spam
Check if the first part of the domain name is all numeric, and if so mark as spam.
Example 2
Subject: regexp:"[A-Z\\?\\!\\.\\,0-9]+" spam
Check if the subject is all uppercase with numbers and punctuation, and if so, mark as spam.
Example 3
Content-Type: regexp:".*name=\"\\(.+\\)\"" SET $attname="\\1"
Find the attachment name in the Content-Type header, and save it in a variable named $attname for use in another rule (such as for anti-virus purposes).
In addition to the regexp: condition, the following may be used:
• eregexp: extended regular expression syntax available
• eregexpi: extended regular expression syntax with case-insensitive character matching
Extended regular expressions allow for more powerful matching than the current regular expression syntax allows, at the cost of slower runtime pattern matching. The eregexpi version should be used if you are writing a regular expression that includes a lot of these constructs: [Aa]. For example:
eregexp:"[Vv][Ii][Aa][Gg][Rr][Aa]"
can be rewritten as:
eregexpi:"viagra"
Conditional expressions
A conditional expression is used to test the state of a variable or invoke a built-in function. Conditional expressions always take the form IF (expression), where the expression is either a variable test or a call to a built in function. Let's take a look at a couple of examples of conditional expressions taken from the shipping rules.MailRules file:
^: IF (@istrustedip($senderip)) DONE
means
• if the IP address in in the trusted IP list, skip rules processing
• before any headers are processed (^), if the sender's IP address ($senderip), which is the built-in variable, is in the trusted list take the action DONE (skip further processing).
From: if (@isspamaddress($from)) SET $spamlevel += 101 AND $spamtests += "FROM_IN_SPAM_FILTERS;"
means
• check if From name is in the spam address list
• when processing the From header, if the From address is in the spam address list, set spam level to extreme (> 100) and add the reason to the $spamtests variable.
The action
The action executes and alters Internet Services processing, once an RFC-2822 header has matched a rule and the tests have been done. There are three actions a rule can do:
• alter the message that arrives
• reject the message altogether
• change the behavior of the rules system.
Actions that alter the delivered message
There are four actions that can alter the delivered message:
Actions that reject the message
There are two actions that can reject the message:
Actions that affect rules processing
There are two actions that affect rules processing:
Let's take a look at some examples of actions in mail rules:
# set the icon of a message in the low spam range
# admin settable variables are defined here
: IF ($LowSpamMin <= $spamlevel && $spamlevel <= $LowSpamMax) INJECT "X-FC-Icon-ID: 23050"
This rule states that after all headers have been processed, if the $spamlevel variable is in the LOW range, add an X-FC-Icon-ID header. This sets the icon in FirstClass.
 NoteThere are many X-FC headers that affect message attributes, such as formID, IconID, and forms data. These can be seen when Internet Services renders a message sent out through SMTP mail.
^: IF (1) SET $CrosspostLimit=$Form.Config.2606.Number AND $CrosspostIncr=5 AND $XpostSpamLevel=20 AND $XpostSpamIncrVal=5
This rule states, before any headers are processed, ^ sets a variety of variables that affect later processing. You can change the values to make this rule behave differently.
Subject:IF (@inblocklist($subject)) DISCARDMESSAGE
If you treat the subject block list as bad words, this rule would discard inappropriate content without sending NDNs. This rule states, when the Subject header arrives check if any of the text is in the subject block list and, if so, quietly discard the message.
Using rules.MailRules
The rules.MailRules file provides a high degree of control in managing your spam or unwanted mail. It contains a series of lines that represent RFC-2822 envelope headers to examine, a test on the content of that header, and an action to perform if the test succeeds.
The syntax of the rules.MailRules file is based on a combination of Internet Services script and the HeaderMatch document script, and is designed for administrators familiar with these Internet Services coding rules. The expression part of Internet Services script, in turn, is based loosely on C and JavaScript. The mail rules component gives access to the RFC-2822 headers as they are being received. There are two core RFCs that define Internet (SMTP) mail:
• RFC-2821
Defines how two SMTP servers talk to each other, the message sender, and the message recipient. RFC-2821 also defines if mail can be accepted by the recipient or if it should be rejected.
The "To:" and "From:" fields do not have to correspond to who sent and who received the message (and usually do not with spam).
• RFC-2822 (and RFCs 2045-2049)
Defines the format of the email messages that pass between sender and recipient. An RFC-2822 message typically contains a series of lines referred to as the envelope, followed by a single empty line, followed by the message body. For example,
Subject: Gone fishing
where
• Subject is the RFC-2822 envelope header
• : is the colon that always follows any envelope header
• Gone fishing is the content for the envelope header.
The standard set of envelope headers (for example, To, From, Subject, Date) can be extended with user-defined extension headers. These headers must begin with (X-). For example,
X-OriginalArrivalTime: 11 Feb 2003 00:38:15.0656 (UTC) FILETIME= [DD6C4E80:01C2D165]
What rules.MailRules can and can't do
The rules.MailRules file can contain one of three types of lines:
• a blank line
Blank lines have no meaning and are just used to make the file more readable
• a comment line
Comment lines start with # and are also used for readability
• a rules line.
The rules.MailRules file cannot:
• script Internet Services's handling of the SMTP conversation (RFC-2821) other than to NDN a message or recipients
• decode MIME (RFC-2045+) (rules.AttachmentBlock can block MIME attachments).
How the default rules.MailRules file works
By default, the rules.MailRules document protects your system, although you can modify the document to accommodate your needs. The default rules.MailRules document contains:
• rules to allow trusted sites, IP addresses, and email addresses to bypass rules processing
• crosspost limits (messages with over 15 total recipients are likely to be spam)
 NoteIf you use crossposting to indicate spam on your system, you should add any valid mailing lists that routinely come into your system to your trusted addresses list, since they may contain large numbers of recipients.
• NDN mail if the IP address is in the spam IP list.
Messages get either NDNed or injected with a warning header, junk header, spam header, or a warning icon, based on the UCE/Spam settings and spam scores:
• extreme scoring spam (>100), either NDN or inject a warning header, red warning icon:
 and set Priority as junk
• if spam score is High (50-100), inject both warning header and red warning icon, and set Priority as junk
• if spam score is Medium (25-50), inject warning header, spam level header, and orange warning icon :
 • if spam score is Low (10-25), inject warning header, spam level header, and green warning icon:
 Understanding the SMTP conversation
An important part of understanding the rules.MailRules file is understanding the SMTP conversation. This conversation delivers a message and initiates at what point and in what order various rules events occur. What follows is a timeline of events in the transmission of a simple RFC-2822 message into Internet Services via SMTP. Here is what happens during an SMTP conversation:
  Let's run a sample message through a sample rules file. Say you have the following rules.MailRules file:
# if the message is from a trusted IP address, we're done
^: IF (@istrustedip($senderip)) DONE
# admin settable variables are defined here
^: IF (1) SET $spamMax=50
# checked for spammers in Received headers
Received: regexp:"\\[0-9][0-9]*\\.[0-9][0-9]*\\.[0-9][0-9]*\\.[0-9][0-9]*\\)" SET $IP = "\\1"
Received: IF (isspamip($IP)) NDN
# check subject
Subject: IF (@inblocklist($subject)) SET $spamlevel += 50
Subject: " " SET $spamlevel += 25
Subject: IF (@allcaps ($subject)) SET $spamlevel += 25
# errors-to makes something less likely to be spam
Errors-To: "*@*" SET $spamlevel -= 20 AND $spamtests += "-ERRORS_TO;"
# If any header says Viagra, this is junk
*: "Viagra" SET $spamlevel += 25
# rules to deal with spam level, processed at the end of the headers
: IF ($spamlevel >= $spamMax) NDN 550 "Sorry, your message has triggered a spam block, please contact the postmaster."
Let's see how the above rules.MailRules file handles a mail message. When another mail server connects to Internet Services, SMTP pleasantries are exchanged up to DATA command. This is how the rest of the message is handled:
• ^ rules are run
SpamMax is now 50 and if the mail server is a trusted IP, all rules' processing stops.
• To: user@is.com
Arrives on SMTP channel, * rule runs, "user@is.com" is compared to "Viagra" and, if no match, nothing happens.
• From:user@is.com
Arrives on SMTP channel, * rule runs, "user@is.com" is compared to "Viagra" and, if no match, nothing happens.
• Subject: HELLO OUT THERE!
Arrives on SMTP channel, rule "Subject:IF (@inblocklist($subject)) SET $spamlevel + = 50" runs, nothing happens.
The "Subject: " " SET $spamlevel + = 25" rule runs and $spamlevel is now 25.
The "Subject: IF (@allcaps($subject)) SET $spamlevel += 25" rule runs and $spamlevel is now 50.
The * rule runs, which compares "HELLO OUT THERE!" to "Viagra" and, if no match, nothing happens.
• "<CRLF>"
A blank line arrives triggering "" rule, ": IF ($spamlevel >= &$SpamMax) NDN 550 "Sorry your message has triggered a spam block, please contact the postmaster"" runs.
Since $spamlevel is 50, the message is NDNed. The other mail server sees the NDN message and stops trying to connect.
When another message comes in the process begins again.
Syntax
This is the basic syntax of the rules.MailRules file:
<header> ":" 1*<sp> <condition> 1*<sp> <action>
The rule.MailRules script syntax for <header>
• ::= <RFC822 header name> (for example, From)
• ::= <RFC2045 and RFC2183 attachment header name> (for example, Content-disposition)
• ::= '*' means any/all headers
• ::= '^' means before any headers
• ::= '' (empty) means end of headers
• ::= '<' means links in HTML message body (<A> and <IMG>)
• ::= '>' means the text in the message body
• ::= '@' means at the end of each set of attachment headers
• ::= '.' means at the end of the message
The rule.MailRules script syntax for 1*
indicates one or more
The rule.MailRules script syntax for <sp>
indicates a space
• ::= ASCII SPACE or TAB
The rule.MailRules script syntax for <condition>
• ::= ["NOT" 1*<sp>] <">simple expression<"> # supports * and ?, case insensitive
• ::= ["NOT" 1*<sp>] "regexp:" <">regular expression<"> # full regular expression, including tagged groups for replacement
• ::= ["NOT" 1*<sp>] "eregexp:" <">extended regular expression<"> # extended regular expression, including tagged groups for replacement
• ::= ["NOT" 1*<sp>] "eregexpi:" <">extended regular expression<"> # extended, case-insensitive regular expression, including tagged groups for replacement
• ::= "IF" *<sp> "(" <expression> ")"
The rule.MailRules script syntax for <expression>
• <expression> ::= ["("] <term> [<relational> <expression>] [")"]
• <term> ::= [<not>] <lhs> [<conditional> <lhs>]
• <not> ::= "NOT" | "!"
• <lhs> ::= <constant> [<mathop> <lhs>] | [<incr> | <decr>] <variable> [<mathop> <lhs>] | <function> | <expression>
• <constant> ::= <number> | <string>
• <number> ::= ["+" | "-"] <octal> | <decimal> | <hexadecimal>
• <octal> ::= "0" 1*<octal digit>
• <octal digit> ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7"
• <decimal> ::= 1*<decimal digit>
• <decimal digit> ::= <octal digit> | "8" | "9"
• <hexadecimal> ::= "0" ["X" | "x"] 1*<hex digit>
• <hex digit> ::= <decimal digit> | ["A" | "a"] | ["B" | "b"] | ["C" | "c"] | ["D" | "d"] | ["E" | "e"] | ["F" | "f"]
• <function> ::= "@' 1*<alpha> "(" [<arglist>] ")"
• <arglist> ::= <argument> ["," <arglist>]
• <argument> ::= <constant> | <variable>
• <mathop> ::= "+" | "-" | "*" | "/" | "&" | "^" | "%"
• <incr> ::= "++"
• <decr> ::= "--"
• <conditional> ::= "==" | "!=" | "|" | "&" | "~=" | "![=]~" | "=[=]~" | <lt> | <gt> | <le> | <ge>
• <lt> ::= "LT" | "<"
• <gt> ::= "GT" | ">"
• <le> ::= "LE" | "<="
• <ge> ::= "GE" | ">="
• <relational> ::= <and> | <or>
• <and> ::= "AND" | "&&"
• <or> ::= "OR" | "||"
The rules.MailRules script syntax for <action>
• ::= <set> *[1*<sp> AND 1*<sp> <set>]
• ::= "BCC" <local address>
• ::= "BLACKLIST" [<time in seconds>]
• ::= "DISCARDHEADER"
• ::= "DISCARDMESSAGE"
• ::= "DONE" # stop processing rules for this message
• ::= "INJECT" <"><header>":" <value><"> # (can include replacement groups, such as \1 and \2)
Example
: IF ($LowSpamMin <= $spamlevel && $spamlevel <= $LowSpamMax) INJECT "X-SPAM-Warning: Low"
This means that if the spam score at the end of a header test is between 'Low' and 'Medium', inject the header "X-SPAM-Warning: LOW" into the Internet header for this message.
• ::= "NDN" <errcode> [1*<sp> <string>] # implies DONE
Example
: IF ($spamlevel > $HighSpamMax && $XtremeCausesNDN == 1) NDN 550 "Sorry, your message has triggered a spam block, please contact the postmaster."
This means that if the spam score at the end of a header test is greater than 'High' and if you have chosen NDN, then NDN the message with an SMTP code of 550 and your choice of text, for example, "Sorry, your message has triggered a spam block, please contact the postmaster."
• ::= "REPLACE" <"><header>":" <value><"> # (can include replacement groups such as \1 and \2)
• ::= "SPAM" # shorthand for SET $priority=junk AND $machinegenerated=1
• ::= "STRIKE"
Example
: IF ($MedSpamMax < $spamlevel) SPAM
This means that if the spam score at the end of a header test is greater than 'medium' setting, mark the message as spam.
Variables
The rules.MailRules file uses variables to enable features. Variables are placeholders, which can store some data for the duration of the processing of a single message. Variables can be tested using conditional expressions and assigned using the SET action. There are two kinds of variables in mail rules: built-in and user-defined.
Built-in variables
Built-in variables are defined by the rules system and contain data about the message being processed. Built-in variables give mail rules access to information about the message being processed. Make sure you are familiar with built-in variables and you use them correctly or misleading or incorrect results will occur. For example, using $#To before the To: header has been received will return the value zero.
Some built-in variables can be SET to alter the attribute of a message. (R) is readonly, (RW) is readwrite.
The possible $AcceptableHTMLParseErrors identifiers and their corresponding values are:
Example:
To set the value to Bad images and Table errors, use either:
$AcceptableHTMLParseErrors = 0x00020004
or
$AcceptableHTMLParseErrors = WarnBadImages + ErrTableError
User-defined variables
User-defined variables are defined by putting $anytext into your rules. You should set a value before you use them. For example, using this rule:
:IF ($myvar > 50) NDN 550 "Go Away"
without having created this rule:
^: IF (1) SET $myvar = 51
would cause your rule to never execute, since $myvar has no value.
Functions
Functions give you access to built-in functionality of Internet Services, such as testing to see if an address is in your filter list. Functions are used in conditional expressions, and they return various values depending on the function. You cannot define your own functions. Here is a list of available functions and their syntax:
These functions behave the same as Internet Services statements:
• @split(...)
• @substr(...)
• @length(...)
• @indexof(...)
• @upper(...)
• @lower(...)
• @rand()
The SET action
The SET action facilitates a number of boolean operators to accommodate the test portion of a rule:
<set>
::= "SET" 1*<sp> <variable> *<sp> "=" *<sp> <value> *[ 1*<sp> "AND" 1*<sp> <variable> *<sp> "=" *<sp> <value> ]
Here are some examples:
• <not>
• ::= "NOT" | "!"
• <and>
• ::= "AND" | "&&"
• <or>
• ::= "OR" | "||"
• <gt>
• ::= "GT" | ">"
• <lt>
• ::= "LT" | "<"
• <ge>
• ::= "GE" | ">="
• <le>
• ::= "LE" | "<="
Syntax for the variable component of SET
The <variable> component of the <set> action allows for both predefined and custom described variables:
::= "SET" 1*<sp> <variable> *<sp> "=" *<sp> <value> *[ 1*<sp> "AND" 1*<sp> <variable> *<sp> "=" *<sp> <value> ]
Predefined variables are called <builtin> and custom described variables are called <alpha> for text only variables or <alphanum> for combined text numerical variables:
::= "$" ["{"] <builtin> or 1<alpha>*<alphanum_> ["}"]
Syntax for the value component of SET
The <value> component of the <set> action (subsection of the <action> script) defines system functions that return a value.
::= "SET" 1*<sp> <variable> *<sp> <assignop> *<sp> <value> *[ 1*<sp> "AND" 1*<sp> <variable> *<sp> <assignop> *<sp> <value> ]
<assignop> ::= "=" | "/=" | "*=" | "-=" | "+=" | "%=" # if value is a <string>, on;y = and += are valid
Using rules.AttachmentBlock
You can use the rules.AttachmentBlock file to block objectionable attachments from reaching your users' Mailboxes. When an email arrives with an attachment that matches the criteria in this file, it is replaced by text pulled from this file.
The format of the rules.AttachmentBlock file is simple: enter each attachment type you want to block (for example, *.com, *.exe, *.scr, 'APPL:????').
Use the multi-character wildcard (*) to match every header of a certain type. Use the single-character wildcard (?) to match on a specific character position of a blocked attachment.
You can also block attachments with specific extensions using "Download limitations" on the Security tab on the Group form. Because using this method to block attachments is controlled by organizational unit (OU) or user group, it allows the message with attachment through Internet Services.
While you can't download server-blocked attachments with the FirstClass client, you can redirect or autoforward a message with blocked attachments through Internet Services to another (non-FirstClass) mail server. Then you can retrieve the message using POP3 or IMAP4 protocols.
Using rules.SubjectBlock
The rules.SubjectBlock file contains undesirable words and phrases that may be found in the "Subject" field of incoming mail. Internet Services uses rules.MailRules to set a spam score for this content, which you can then choose to block before the mail message reaches your users, or tag to allow your users to handle it using their own end-user mail rules.
The rules.SubjectBlock file does not automatically block undesired content from your site unless you've selected "SubjectBlock causes NDN" on the UCE/Spam - Mail Rules tab on the Basic Internet Setup form. Instead, if a word or phrase listed in your rules.SubjectBlock file exists anywhere in the "Subject" field of a message, Internet Services adds a value to the spam level, scores the message, and gives it the priority of Junk, if applicable. This process prevents accidental non-delivery of legitimate email and lets you control how to handle potential spam for your individual site.
The values and scores Internet Services assigns are based on the rules configured in the rules.MailRules file. For example, this rule:
Subject: IF ($Form.Config.2605.Checkbox == 0 && @inblocklist($subject)) SET $spamlevel += 100 AND $spamtests += "SUBJECTBLOCK;"
checks to see if both the
• "Subject" field of a message contains any words that are in the rules.SubjectBlock list
and
• "SubjectBlock causes NDN" field checkbox is cleared.
If both are tests are TRUE, the spam score is increased by 100.
How to add rules.SubjectBlock file entries
The format of the rules.SubjectBlock file is one word or phrase per line. To add comments, preface the line with #. For example,
# these are silly words we want to block
Viagra
Free mortgages
Get rich
ADV:
The @inblocklist function performs case insensitive comparisons by default. To perform case sensitive comparisons, use the optional "true" parameter @inblocklist($subject, true)
You can create more extensive blocking lists that scan other fields in a message and also work with SMTP rules to score content accordingly. This is described later in this document.
How rules.MailRules uses the rules.SubjectBlock file
The rules.MailRules file contains rules to test the content of the "Subject" field against undesirable words and phrases you've added to your rules.SubjectBlock file. The rules.MailRules file also contains rules to find tricks spammers use to hit sites.
You can add your own rules as required or disable any default rules by using # at the beginning of the line. For example:
# Subject: IF ($Form.Config.2605.Checkbox == 0 && @inblocklist($subject)) SET $spamlevel += 100 AND $spamtests += "SUBJECTBLOCK;"
Here are some entries in the default rules.MailRules file that Internet Services uses to test the "Subject" field of mail messages:
-----------------------
# Subject tests go here
• Subject: IF ($Form.Config.2605.Checkbox && @InBlockList($Subject)) NDN 550 "$sorry"
Checks if the subject contains any words in the rules.SubjectBlock list and checks if the "SubjectBlock causes NDN" checkbox is selected. If both tests return TRUE, an NDN is sent.
• Subject: IF ($Form.Config.2605.Checkbox == 0 && @inblocklist($subject)) SET $spamlevel += 100 AND $spamtests += "SUBJECTBLOCK;"
Checks if the subject contains any words that are in the rules.SubjectBlock list and checks if the "SubjectBlock causes NDN" checkbox is cleared. If both tests return TRUE, the spam score is increased by 100.
• Subject: regexp:"^$" SET $spamlevel += 10 AND $spamtests += "SUBJ_HAS_NO_SUBJECT;"
Checks if the subject line is empty. Messages with no subjects have a greater chance of being spam. If TRUE, the spam score is increased by 10.
• Subject: " " SET $spamlevel += 20 AND $spamtests += "SUBJ_HAS_SPACES;"
Checks if the subject has extra spaces in it. Spammers sometimes use subjects such as "Hi... Remember me? <random number>", which would get caught by this check because of the extra spaces. If TRUE, the spam score is increased by 20.
• Subject: IF (@allcaps($subject)) SET $spamlevel += 25 AND $spamtests += "SUBJ_ALL_CAPS;"
Checks if the subject is written in all uppercase letters. Using all uppercase letters is usually an indication of spam, and the spam score will be increased by 25.
• Subject: eregexpi:"(^|[^[:alnum:]])(v|\\\\/)[^[:alnum:]]{0,2}[i1l\|!¡ÌÍÎÏìííîï][^[:alnum:]]{0,2}[a@ÀÁÂÃÄÅÆàáâãäå][^[:alnum:]]{0,2}g[^[:alnum:]]{0,2}r[^[:alnum:]]{0,2}[a@ÀÁÂÃÄÅÆàáâãäå]($|[^[:alnum:]])" SET $spamlevel += 101 AND $spamtests += "SUBJ_VIAGRA;"
• Subject: eregexpi:"x[^[:alnum:]]{0,2}[a@][^[:alnum:]]{0,2}n[^[:alnum:]]{0,2}[a@][^[:alnum:]]{0,2}x" SET $spamlevel += 101 AND $spamtests += "SUBJ_XANAX;"
• Subject: eregexpi:"d[^[:alnum:]]{0,2}r[^[:alnum:]]{0,2}u[^[:alnum:]]{0,2}g[^[:alnum:]]{0,2}s" SET $spamlevel += 100 AND $spamtests += "SUBJ_DRUGS;"
The preceding three tests, check for different variations of the words "Viagra", "Xanax" and "drugs" which are designed by spammers to bypass antispam filters but are recognizable to humans. If the test returns TRUE for either line 30 or 31, the spam score is increased by 101. If the test returns TRUE for line 32, the spam score is increased by 100.
• Subject: IF (@PunctCount($Subject) >= 5) SET $spamlevel += 10 AND $spamtests += "EXCESS_PUNCT;"
Checks if the subject contains five or more characters of punctuation. Excessive punctuation is sometimes an indication of spam. If this test returns TRUE, the spam score is increased by 10.
• Subject: IF (@WordCount("lists.Rude", $Subject) > 1) SET $spamlevel += 100 AND $spamtests += "SUBJ_RUDE_WORDS;"
Checks if the subject contains more than one word that is in the lists.Rude file found in the Filters folder. If this test returns TRUE, the spam score is increased by 100.
Using rules.SMTPDebug for your custom rules
You can add custom mail rules that won't be replaced during an upgrade to rules.SMTPDebug. This document is mainly intended for temporary rules to aid SMTP debugging efforts using the per-connection logging variables, but it is not limited to that.
All mail rules may be used in this document, and rules in this document are executed before rules in the rules.MailRules document.
The debug logging levels can be set for:
Example 1
# Log the complete SMTP message when the sender's IP is 123.123.123.123
^: IF ($SenderIP == 123.123.123.123) SET $SMTPServerLogging = 5
Example 2
# Log the complete SMTP message, and turn on MIME and message decode logging when the Subject is "This is a test message from Susan".
# Note that mail headers received before the Subject: header will not be logged.
Subject: IF ($Subject == "This is a test message from Susan" SET $SMTPServerLogging = 5 AND $MailDecodingLogging = 5 and $MIMEDecodingLogging = 5
Understanding spam scoring
Spam scores determine how Internet Services flags incoming messages based on SMTP rules file and the spam levels set on the UCE/Spam - Mail Rules tab of the Basic Internet Setup form. When an SMTP server delivers a message to your site, Internet Services uses the rules in the rules.MailRules file to score it according to a variety of different criteria. For example, crossposting limits, message headers, content, and so on.
Internet Services has preset spam scores that are adequate for most FirstClass sites. If you want to fine tune scoring for your site, you can set your own levels on the Mail Rules tab.
For example, if you find some of your messages are being marked as spam because of a low score range, you could increase the numbers of "Min" to 100 and "Max" to 125 (and change the other field scores accordingly). This means that Internet Services won't mark anything up to 99. If you've selected "Extreme causes NDN", but find Internet Services is rejecting too many of your messages, you can raise the "High spam score".
Spam scoring rules
Spam scoring follows these rules:
• adds 100 to spam scoring, if mail header contains words in the rules.SubjectBlock file
• adds 20 to spam scoring, if subject contains a string of five (5) or more consecutive spaces
• adds 25 to spam score, if subject contains only CAPITAL letters
• subtracts 20 from spam score, if If ERRORS_TO field is valid
• Mailers are split into two groups: the first group adds 75 to the score and the second group adds 25 to the score. This occurs if the mailer is a known spam mailer such as Floodgate, Group Mail, Millenium Mailer, AutoMail, JMail, eGroups Message Poster, and so on.
• adds 50 to spam score if the Message-ID header is missing and the message was not sent by a "busted mailer": AT&T Message Center Version 1 and JMail
• adds 50 if the Subject header is missing and the message was not sent by a "busted mailer"
• adds 25 if the Date header is missing and the message was not sent by a "busted mailer"
• adds 10 if the Subject is empty
• adds 101 if the Subject contains any variation of Viagra or Xanax
• adds 100 if the Subject contains any variation of the word drugs
• adds 10 if the Subject contains 5 or more punctuation characters
• adds 100 if the message contains the header X-Speedi-Job
• adds 100 if the message contains the header X-Ssi-Job
• adds 101 if the message contains the header X-Originating-IP with the IP address of this Internet Services somewhere in the value
• adds 5 if the message contains either X-CS-IP or X-IP headers
• adds 100 if the message body contains words in the rules.SubjectBlock file, and 'SubjectBlock causes NDN' is not checked
• adds 50 if the message body contains words in the lists.BodyList1 file
• adds 100 if phrase CAN-SPAM Act (or variations) is in the message body
• adds 101 if the message body contains any variation of Viagra
• adds 101 if the message body contains the word 'free', disguised by adding punctuation between the letters
• adds 50 if there is a recognizable 'unsubscibe' link in the HTML message body
• adds 101 if there are less than 10 characters in HTML message body, but one or more links (<A or <IMG)
• adds 101 if there are one or more links in the HTML message body that are indicative of spam messages.
Understanding Junk scoring
Internet Services marks messages with a High spam score (51-100) or above as Junk. Internet Services also follows the Precedence header of a message to determine if it must mark a message as Junk. If the sender sets the precedence of the message as Junk, he receives Junk priority on inbound mail.
Certain mailing list programs include a Precedence, which is a header in the messages they send. This is how Internet Services maps the different header values into FirstClass priority values:
The following are the conditions that would immediately cue Internet Services to increase a message's spam score enough to mark it as Junk:
• the subject or body contains a word in the rules.Subjectblock list, and such messages have not been set to be blocked with an NDN
• the subject contains any variation of the word "Viagra", "Xanax" or "drugs"
• the subject or body contains more than one word in the lists.Rude list in the Filters folder
• any X-Mailer text in the lists.x-mailer-1 list in the Filters folder is found
• specific X-headers used by spammers are found
• the body contains false disclaimers saying the message isn't spam because it conforms to the American CAN-SPAM Act of 2003
• the body contains any variation of the word "Viagra" or "free"
• an attachment name is detected and both it and the subject are in the lists.VirusNetskyAttachment list in the Filters folder
• the body contains URLs to images used by spammers to track who has read the message
• the body contains URLs to domain names ending in .biz
• the body contains URLs to images that look to users like they are at one server when they are actually at another
• the body contains URLs that lead users to a different site than they think they are heading to.
Creating your own blocking lists
You can create your own blocking lists to prevent spam from entering your site. These documents work similarly to the rules.SubjectBlock document. Rules.MailRules uses them to cue Internet Services to perform specific actions (add spam scores, NDN sender, and so on) based on tests according to mail content. However, blocking lists go one step further; they check numerous fields in a mail message instead of just the "Subject" field.
You can create your own lists for:
• objectionable words and phrases
• X-Mailers (programs used by spammers)
• busted mailers (non-spam mailers that don't add a Message-ID header to a message)
• IP addresses
• virus subject names
• virus attachment names.
As with all rules documents, lists documents must reside in the Filters folder and have a unique name prefixed with the word 'lists'. For example,
• lists.myreallybadlist
• lists.mymoderatelybadlist
• lists.questionablebadlist
How rules.MailRules uses lists files
The default rules.MailRules file contains rules to test the content of message fields against the content of your blocking files. These include the rules.SubjectBlock, rules.AttachmentBlock, and all 'lists' files.
By default, the Filters folder contains basic lists documents that work with hard-coded rules in the rules.MailRules document to block different types of spam. For example, the lists.VirusNetskySubject and the lists.VirusNetskyAttachment documents are called by this rule in the rules.MailRules document:
@: IF (@length($attname) && @InWordList("lists.VirusNetskySubject", $Subject) && @InWordList("lists.VirusNetskyAttachment", $attname)) NDN 550 "No worms allowed"
This rule states that if Internet Services detects a subject that is entered in the lists.VirusNetskySubject file or an attachment name that is entered in the lists.VirusNetskyAttachment file, it will NDN (with text) the offending sender. The Filters folder contains other default 'lists' documents similarly cued by corresponding mail rules.
If you create your own lists, you must make sure there is a corresponding mail rule in the rules.MailRules file. You can either modify an existing rule or create your own rules. For example, if you create these lists in your Filters folder:
• lists.myreallybadlist, which adds 100 points to the spam level
• lists.mymoderatelybadlist, which adds 50 points to the spam level
• lists.myquestionablelist, which adds 25 points to the spam level
you would have these rules in the rules.MailRules file:
Subject: IF (@inwordlist("lists.myreallybadlist", $subject)) SET $spamlevel += 100 AND $spamtests += "SUBJECT_IN_LIST;"
Subject: IF (@inwordlist("lists.mymoderatelybadlist", $subject)) SET $spamlevel += 50 AND $spamtests += "SUBJECT_IN_LIST2;"
Subject: IF (@inwordlist("lists.myquestionablelist", $subject)) SET $spamlevel += 25 AND $spamtests += "SUBJECT_IN_LIST3;"
If you want to fine tune the spam levels for your site, you can do so on the Mail Rules tab or directly in the rules.MailRules file.
User-created mail rules
Any FirstClass user with the appropriate privilege can create personal mail rules for their own Mailbox and any conferences for which they have the appropriate permissions.
Internet Services tags messages that fail any of the spam tests with 'X-SPAM-xxxx' headers. Users can test these headers using their own mail rules to file or discard messages they think are spam.
As an example, if a user sees a message with a warning icon:
he can view the message's Internet header to see the spam level and why the message was marked as spam. Here is an example of what the Internet header looks like for a message with a 'High' X-SPAM-warning:
 Using mail rules in conjunction with SMTP rules provides both you and your users with the ability to efficiently deal with Internet spam on your system.
About the junk mail handling preference
If a user has their junk mail handling preference set to Delete Silently, any message marked as Junk will be deleted automatically. By default, this applies to all messages with a spam level of HIGH (greater than 50 spam level).
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||