Rules-Based Filtering

Initial efforts to track down spoofed e-mail was the genesis of the rules-based filter, which began very simple and became much more complex over time. The technical strategy was to filter incoming e-mail messages based on refined rules-sets and forward the clean mail. Rule-sets triggered on information in the header control files of the mail messages. The message content was not used in the filtering process. All filtered mail was processed via one of two paths.

In the early hours of the filter prototype, we discovered that a majority of e-mail bombs contained invalid originator SMTP addresses without the @ sign. Our first prototype filter jailed all messages with invalid SMTP address formats. However, our cyber-bomb opponents adapted, created bogus addresses with random symbols. Our filters were actively probed by bombers across the Internet (and across the globe) in order to understand how our filters were implemented [10]. Below is a small code fragment, written in PERL, from the first alpha version filter prototype, illustrating simple rule-set processing on the header files.

# author: Tim Bass Jan. 23, 1997

# Sendmail will run with the 'queue only and 
# not deliver'  option, -odq, as a daemon 
# process (sendmail -bd -odq).
# This script is executed by crond 
# Currently executed every five minutes

# open the qf files

open(QF,"<$Header")|| die "Can't open files\n";

# search the qf files for interesting things
# currently, only doing minimal filtering on
# the sender.  It is easy, however, to filter
# on other header fields, as required.

while(<QF>){

#  the /^S/ expression looks in the qf file 
#  for the line beginning with S, the sender

if(/^S/){ 

#  then we look in that line for senders
#  without @ signs

if(!/\@/){ 

#  and we move the suspected header and data
#  files to the suspected queue, qjail

if(!/<>/){ 
      rename($Header,$JailFile);
      rename($Data,$JailFile2);
      } 
}

#  and we copy these suspected senders with data
#  files to the copy queue, qcopy (just in case)

elsif(/whitehouse/i){
       system("/bin/cp $Header $CopyFile");
       system("/bin/cp $Data $CopyFile2");
                   }
       elsif(/whistleblower/i) {
       system("/bin/cp $Header $CopyFile");
       system("/bin/cp $Data $CopyFile2");
       }

close(QF);  } } }
close(QUEUE);

# then we move the 'good stuff' remaining out 
# of the processing queue into the clean queue;
# /usr/lib/sendmail -q -oQ/usr/spool/qclean"

The bombers were adapting to new filter rule-sets within 24 to 48 hours. The most common bomb signature was repeated e-mail with the same sender-receiver pairs. With few exceptions, this fact provided a successful indicator of hostile e-mail. We refined the filtering algorithm based on this observation and provided the programming requirements to the development team. This algorithm became a key element in mitigating numerous types of e-mail bombs during attacks.

It is important to briefly point out that there are occasional exceptions to most filter-rules. Trapping all of the exception conditions at the application layer can prove to be a difficult (perhaps impossible) programming challenge. It is impractical, from a resource perspective, to obtain a 100 percent solution. However, the tradeoff of jailing a very small percentage of good e-mail vis-a-vis minimizing the risk and exposure to the enterprise was a management decision, not a technical one.

The prototype filter design also provided valuable information which assisted investigators discover the content, type and origin of the criminal e-mail. Trapping and jailing politically charged mail became a high priority as our team began refining filter rules based on illicit e-mail origin, SMTP mailer, originator, sender-receiver pairs, etc. Our software development team provided numerous enhancements to the original filter prototype which resulted in reduced countermeasures manpower and improved filter granularity. The speed of the sendmail process receiving and forwarding mail in real-time made queuing the messages prior to filtering necessary. The additional latency in the mail-filter processing was actually offset by performance improvements in the SMTP infrastructure. However, we did not quantify these observations.

In addition, by queuing the messages, the entire SMTP header file and control messages could be used in the filter process. This proved to be extremely valuable in the process of examining mail bombs, understanding the nature of the attacks, and simultaneously insuring all e-mail was correctly delivered with minimal delay.