mult.ifario.us

This entry outlines how to integrate procmail and bogofilter with Mail.app.

For motivation, the recent rash of word-spew spam rendered the junk filter in Mail.app ineffective (for me). With no way to upgrade the filter functionality built into Mail.app, I decided to get an external, upgradeable, extensible filter working. Once all was said and done, I had procmail and bogofilter working and word list management AppleScripts integrated into Mail.app:

Mail Processing Pipeline Construction and Configuration

Mail.app has a lot going for it, like great search capabilities, a multi-threaded front-end, and integration with AddressBook, among other things. However, there are a couple of gotchas for using Mail.app with standard MDAs like procmail that expect to operate on streams and spool files:

• Mail.app, as far as I can tell, can only pull mail from a POP3, IMAP, or Exchange server. (It may be possible to simply use one of the mbox files inside Mail.app's folder hierarchy as a spool, but without documentation about file locking and caching behavior, I'm not going to touch it.)
• Mail.app's rule processing is side-effect-free, so any in-client message processing would have to be done in one or more independent AppleScript scripts that did not alter the message. (This would require, for example, that each AppleScript move the message, which is undesirable.)

To get around the issues, I settled on a mail processing pipeline that still presents a POP3 interface to Mail.app:

Mac OS X 10.3 comes with procmail and fetchmail, so that much is easy. For the local POP3 server, qpopper), and the setup instructions in Adriaan Tijsseling's blog are straightforward.

With BerkeleyDB built and installed from scratch (the version from darwinports puts files in strange locations that are incompatible with the bogofilter build), a fresh download of bogofilter 0.16.1 compiles cleanly with:

./configure --with-libdb-prefix=/usr/local/BerkeleyDB.4.2

(I do have the BerkeleyDB lib and include directories declared in the CPATH and C_INCLUDE_PATH environment variables.)

To get bogofilter to insert an X-Bogosity header into each mail, the .procmailrc that I'm using is:

MAILDIR=/Users/prb

:0fw | bogofilter -u -e -p

:0: Mailbox

The .fetchmailrc (with the host, login, and password changed...) is:

set daemon 120

defaults no rewrite

poll mailhost.com with protocol POP3:
  user login password secret
  mda "procmail -d %T"
  ssl, no keep, no forcecr

Integration with Mail.app

With the mail processing pipeline working, I shut off Mail.app's junk mail processing and added a rule to trigger on messages that bogofilter marks as spam:

The more difficult piece of integration is a way to manage the bogofilter word list from within Mail.app. Because I'm using the -u option that automatically adds to the counts in the word list, I need to rerun bogofilter with either -Ns (decrement the ham counts, incremenet the spam counts) or -Sn (decrement the spam counts, increment the ham counts), as appropriate, to reclassify a message.

Here is script that writes the content of a selected message or messages out to a file and then runs the appropriate shell commands:

using terms from application "Mail"
  on perform mail action with messages msgs
    if (count of msgs) is not equal to 0 then
      repeat with msg in msgs
        set t to path to temporary items
        set posixT to POSIX path of t
        set nam to (t as string) & "bogotemp"
        set Pnam to (posixT as string) & "bogotemp"
        do shell script "rm -f " & Pnam
        set f to open for access file nam with write permission
        write ((source of msg) as string) to f
        close access f
        do shell script "/usr/local/bin/bogofilter -Ns < " & Pnam
        do shell script "rm -f " & Pnam
      end repeat
    end if
  end perform mail action with messages
end using terms from

This is the equivalent of the much more compact

bogofilter -Ns < msg

on the commandline. It is also possible to get message statistics from bogofilter in Mail.app. For example, this script populates a new message with the words and statistics from a selected message using the same temporary file trick:

using terms from application "Mail"
  on perform mail action with messages msgs
    set bogodirectory to "/usr/local/bin/"
    set bogodb to "/Users/prb/.bogofilter/wordlist.db"
    if (count of msgs) is not equal to 0 then
      repeat with msg in msgs
        set t to path to temporary items
        set posixT to POSIX path of t
        set nam to (t as string) & "bogotemp"
        set Pnam to (posixT as string) & "bogotemp"
        do shell script "rm -f " & Pnam
        set f to open for access file nam with write permission
        write ((source of msg) as string) to f
        close access f
        set spamicity to bogodirectory & "bogofilter -e -t  < " & Pnam
        set score to "spamicity: " & (do shell script spamicity)
        set summary to bogodirectory & "bogolexer -p -I " & Pnam
        set summary to summary & " | " & bogodirectory
        set summary to summary & "bogoutil -v -p " & bogodb
        set summary to summary & " | sort | uniq | sort +3 -r"
        set lst to paragraphs of (do shell script summary)
        set text item delimiters to return
        set wordList to score & return & return
        set wordList to wordList & (end of lst & return & rest of lst)
        tell application "Mail" ¬
               to make new outgoing message ¬
               with properties {subject:score, ¬
               content:wordList, visible:true}
        do shell script "rm -f " & Pnam
      end repeat
    end if
  end perform mail action with messages
end using terms from

If you're lazy or want something to use as a starting point, here are the scripts I'm using. To install, unpack in ~/Library/Scripts/Mail\ Scripts.

I'm a relative newbie to AppleScript and the internals of inter-application communication on Mac OS X, so exploring the different ways to integrate procmail and bogofilter with Mail.app was educational. At the very high end, it is apparently possible to build native plug-in bundles for Mail.app. (I couldn't even get started because XCode kept crashing on me...) Example source code (here and here) is available, but I wasn't able to locate any official documentation. I also explored the possibilities with the specific capabilities of MacPython and Mac::Glue.

Notes

Thanks to Lucas Bergman, FiveSight's guru of all things mail, for helping with the rc files and various mucking around.

Actually writing AppleScript is awful, and the unpalatable syntax is compounded by the lack of any useful debugging other than popping-up dialogs through the course of a script. I tried to avoid it, but it appears to have been the shortest path. (If you must, it looks like AppleScript: The Definitive Guide is a good source of information and examples.)

An incomplete list of alternatives to the do-it-yourself approach with procmail:

• Ben Han has a nice (free) package JunkMatcher that can perform regular expression-based filtering along with a bunch of other tests. His approach uses a callback to the scripting interface of Mail.app from a Python application to do the work.
• SpamSieve looks like a great deal at $25 (It took more than $25 worth of my time to get this all working...). Nonetheless, it doesn't provide for arbitrary extensibility, and in its current incarnation, it can only mark but not move marked messages from a POP3 account. (This is, as the SpamSieve FAQ points out, Mail.app's fault, not SpamSieve's.)