This is a FAQ for Procmail, the mail processing utility for Unix.
This Procmail FAQ is an attempt at answering the most often asked questions and straightening out the most frequent misconceptions about Procmail. This is no substitute for the manuals, and indeed presupposes some familiarity with the program's regular documentation.
If you feel you have trouble understanding the tips in this FAQ and/or the manual pages, please be invited to check out the newbie links on the companion link page, which also has a lot of other Procmail-related links for you to investigate, including links to several tutorials which provide a sort of "quick start" documentation.
Please note that this FAQ does not attempt to help you getting started with Procmail; refer to the tutorials mentioned above if that's what you're after.
On the other hand, if you are more comfortable reading crib sheets than prose, you might want to look at the quick reference (still in beta; feedback and suggestions appreciated).
The official URL of this page is http://www.iki.fi/era/procmail/mini-faq.html -- this is a "virtual" URL which resolves to a different host. Please use this "virtual" URL rather than whatever your browser thinks it is you are currently looking at. This site can and will move in the relatively near future.
The following mirror sites are available:
Please use one of the mirrors if you can.
The author wanted to call this a "Mini-FAQ" but it keeps getting bigger. There are plans to rename it the "Bronto-FAQ."
As this document has changed and grown (it is currently more than twelve times the size of the arguably more elegant original version 1.0) it has become a bit hard to know where exactly to expect information about some things. I apologize for this. In the Contents, I try to include a mention of all very frequently asked questions, even if they're in a subsection of a subsection (further adding to the bloat, I'm afraid). The below table of contents is an abridged "best of" instead of a full TOC. (This makes little sense on the web page but is currently made to fit into various versions. I'll fix it, someday. I think.)
The author's contact information is at the bottom of this page.
procmail: Skipped "(something)" errors ...
.procmailrc as usual?
:0: mean?
From_ header?
\/
* wildcard?
.rc file
egrep compatible
^FROM macro
SHELL= definition
.forward file
*?
biff work right for my own folders?
BCC header?
user@host?
EXITCODE to use?
.forward from showing in bounces?
Version information:
$Id: mini-faq.prep,v 1.348 2000/02/21 11:30:18 era Exp $
The
version history
details recent developments.
Procmail is a mail processing utility, which can help you filter your mail; sort incoming mail according to sender, Subject line, length of message, keywords in the message, etc; implement an ftp-by-mail server, and much more.
Procmail is also a complete drop-in replacement for your MDA. (If this doesn't mean anything to you, you don't want to know.)
Procmail runs under Unix. See Infinite Ink's Mail Filtering and Robots page for information about related utilities for various other platforms, and competing Unix programs, too (there aren't that many of either).
You can download Procmail from the main Procmail site or a number of mirrors. The Links page has a listing of well-established mirror sites.
In ancient times, then-current versions were posted to comp.sources.misc (vol 43, July 1994, is the newest I could find).
The recommended version of Procmail to install is 3.13.1 . [The previous reasonably stable versions were 3.11pre4 and 3.11pre7. Don't use any other "pre" versions, or 3.12. You should also avoid the ancient version 3.10. 3.14 was released recently, but contains some minor bugs. You would wait for 3.15, which should be announced shortly.]
http://www.procmail.org/procmail-snapshot.tar.gz contains a snapshot of the current development version. Philip Guenther's "to do" list at http://www.procmail.org/todo.html lists both implemented and planned changes relative to the latest release.
Please note that some of the tips in this FAQ (or elsewhere) might not work for versions older than 3.11pre4. In particular, a great number of sites seem to be stuck on 3.10, which is a bad choice for a number of reasons. There should be plenty of incentive to upgrade to 3.13.1.
The installation procedure is fairly straightforward but probably not the first thing you should attempt after you get a Unix account.
If you feel adventurous, and have a friend with a working copy of Procmail for your type of operating system and hardware, you can just snatch her/his binary. However, you need to be aware that this defeats some checks which the installation program performs, such as determine where your mail spool is, what kinds of file locking should be employed, etc. Be particularly wary if you use NFS-mounted mail spool directories.
The distribution comes with a
simple FAQ
(locally produced HTML version -- the
text-only original
is also available)
which covers some issues faced when first getting acquainted with
Procmail, such as how to view the manual pages, but it primarily
addresses various installation problems.
There's also answers to some very frequently asked questions,
some of which are not dealt with in the document you're reading now.
Please look at least at the TOC of the "original" FAQ as well.
(An abridged table of contents of the "original" Procmail FAQ is
in the "Quick Questions" section below.
Various files in the Procmail distribution package also contain
nuggets of very valuable information,
especially for the administrator setting up Procmail for the first time.)
If you can't find the answer to your question in either of these FAQs, please take a look at the Links section towards the end of this document -- but first, make sure you have found all of Procmail's manual pages; there are several.
If you are new to Unix, you should probably read up on regular expressions (grep/sed/awk/perl etc) and a little on mail handling before attempting to tackle the Procmail manual pages.
Q: I just downloaded Procmail, and want to learn to write my own recipes. Where do I start?
A: The distribution package comes with some pointers, and includes manuals, of course. In addition to that, the links page, which is a companion page to this FAQ has links to several good tutorials. There is also a small collection of links near the end of this FAQ.
There are many good tutorials and the purpose of this FAQ is not to compete with them, although some basic questions about Procmail's syntax recur in various on-line forums often enough to warrant their inclusion here as well.
Q: Is there a Procmail for Windows NT?
A: No, and it's somewhat unlikely that anybody would undertake a port. Read Bart Schaefer's excellent summary of the problems. Excerpt: "I've seriously looked at porting it to NT, yes. The problems are pretty severe."
Q: How can I run an arbitrary Perl or shell script on all or selected incoming mail?
A: Install Procmail. Read the manual pages (there are several). Thank you.
:0 * conditions, if any | your-script-here
The conditions, in their simplest form, are regular expressions to match against the header of each incoming mail message. Correction: Even simpler, you can leave out the condition lines completely if you want to do your action (in this case, run a shell script) unconditionally.
More-complicated conditions can also be exit codes of other shell scripts or programs, or tests against the full body of the message, or against Procmail variables (Procmail's variables are also exported to the environment of subprocesses, so they are essentially environment variables. There are details about this later in this FAQ.)
Actions can also be to save the message to a folder (appended to a
Unix mailbox file, or written to a new file in a directory) or to forward
the message to one or more other addresses. Finally, the action can be
a nested block of more "recipes,"
as these condition-action mappings are called in Procmail jargon,
to try if the outer condition is met.
The procmailrc(5) manual page has the full scoop.
Obviously, you are not restricted to Perl or shell scripts. Anything you can run from a Unix command prompt can be run from Procmail, in principle, although running interactive programs doesn't usually make much sense.
Q: I can't read. What's in the "original" Procmail FAQ?
A: Here's an abridged listing.
Not every question is a Frequently Asked Question, of course.
That's why there is a mailing list, after all.
(Additionally, not all answers in the FAQs are Frequently Wanted
and/or Understood. Sigh. ;-)
Q: What if Procmail is already installed by another user on my host?
A: Could be. Ask around. Yes, one installation per site should suffice.
Q: How do I know I have found all the manual pages?
A: You read them; most of them contain useful information. Start with the procmail(1) manual page; it contains pointers to the others under the "SEE ALSO" heading.
Make sure you find all of the pages procmail, procmailex, prorcmailrc, and regexp (or perhaps egrep or grep if you don't have a general introduction to regular expressions on your system).
Q: How do I know which version of Procmail I have?
A: You examine the output of the following command:
$ procmail -v procmail v3.11pre4 1995/10/29 written and created by Stephen R. van den Berg <srb@cuci.nl> Submit questions/answers to the procmail-related mailinglist by sending to: <procmail@informatik.rwth-aachen.de> And of course, subscription and information requests for this list to: <procmail-request@informatik.rwth-aachen.de> Locking strategies: dotlocking, fcntl() Default rcfile: $HOME/.procmailrc System mailbox: /var/spool/mail/$LOGNAME
For Formail and Lockfile, the companion utilities, it's harder to say as older versions (before 3.12) won't tell you themselves.
Q: Please please tell me the address of the Procmail mailing list!
A: See answer to previous question, or the periodic Mini-FAQ pointer posting.
There used to be an alternative list, which was started by Rhett 'Jonzy' Jones, but it appears to be dead.
There is now also a procmail-dev list for those interested
in developing Procmail, and similarly a smartlist-dev for
SmartList development. To join, send a subscribe request to
procmail-dev-request@cuci.nl and/or
smartlist-dev-request@cuci.nl, respectively.
Details about the various mailing lists related to Procmail (purposes, subscription and unsubscription instructions, etc) are available from the URL http://www.iki.fi/era/procmail/lists.html and mirrors.
Q: I have a question which this FAQ doesn't answer.
A: Ask on one of the Procmail mailing lists (see previous questions), or try a newsgroup such as comp.mail.misc. The links page has links to various resources, including tutorials and examples.
Please don't assume that the FAQ author is interested in inquiries about free consulting services. I read the Procmail mailing list and answer questions when I can. Several others do that too, so you stand a better chance of getting a decent answer by mailing the list rather than me. Thanks.
Before you send stuff to a mailing list or a newsgroup, you need to know the basics of "netiquette," i.e. how to behave online. In particular:
:-)
Lists usually have a separate "administrative" address
which is different from the address where you'd submit
discussion messages. Keep the subscription instructions around,
so you know how you got on the list in the first place --
usually getting off the list involves a very similar procedure.
See also the
Unsubscribe FAQ
There are various netiquette pages all over the net. Try e.g. http://www.cs.ubc.ca/spider/edmonds/usenet/ml-etiquette.html or do your own search.
An additional tip: You should perhaps familiarize yourself with the FAQs for related Usenet newsgroups. Apart from various sections of the comp.mail hierarchy, various Unix newsgroups -- comp.unix.shell in particular -- should be worth a visit.
Q: Why am I getting spam from the Procmail list server?
A: The list is presently open for posting by anyone, including the spammers. This may have to be changed because of the rising tide of spam, which would be unfortunate, because many people might not want to be bothered to subscribe to the list in order to just ask one question. Anyhow, stay tuned (and tune your filters). (Actually the list's setup was recently changed to filter out the most obvious spam.)
A: Try subscribing yourself anew and see if things start to change. SmartList is rather paranoid and will easily unsubscribe you if it gets bounces from your address.
By the way, the on-line mailing list archive at http://www.rosat.mpe-garching.mpg.de/mailing-lists/procmail/ should have all recent messages on store. You can check there whether you have received the newest messages.
Briefly, the regular expression to match anything that begins with
"pr" and ends with "mail" is ^pr.*mail$ but for practical
purposes, e.g. in a Subject: line, you might want to try the following
recipe:
:0: * ^Subject: pr.*mail procmail-mail
This says: Anything that begins with "Subject: pr" and contains the
string "mail" (followed by anything, or nothing) somewhere after that
is to be saved in the folder procmail-mail.
The regular expression . (dot) matches any character
except newline. The operator * says "the previous
regular expression (in this case, any character) zero or more
times", and so the whole compound expression .* matches
any string of length 0 or longer. (Use + instead of
* to force the matched string to contain at least one
character.)
Note that there is no need for a trailing .* wildcard
(and certainly not a sole *, which is a syntax error)
-- the recipe will match regardless.
The neat stuff starts when you want to ignore "pro-mail" and "ProMail" while still looking for anything else that begins with "pro" and also contains "mail", but this example ends here.
Any beginners' book about Unix will contain a more detailed tutorial
on regular expressions, most likely in conjunction with examples using
the grep and sed programs. For reference,
the manual page for egrep (an extended grep)
at your site should contain
a concise listing of regular expression operators.
(Note that the manual's assertion that Procmail is 100% egrep compatible
is not strictly speaking true. For one thing, there are many egrep
implementations.)
One frequently seen Procmailism is this:
[ ]*
The brackets contain one space and one tab, in any order.
This regular expression will catch any sequence of horizontal
whitespace (including none at all -- change the *
to a + if you want to make sure there's at least one
whitespace character).
Some mailer programs use tabs instead of spaces in some places, others will attempt to "pad" all header fields with spaces to make the headers somewhat more readable. Finally, Subject headers are written by mere humans who sometimes press the space bar twice, perhaps only because they want to see if that will break your autoresponder script. Therefore, your Procmail recipes frequently need to match arbitrary runs of whitespace characters if you want them to work in all situations.
Under Unix and other multitasking operating systems, several processes
can be running at once. This means several mail messages can be in
delivery at the same time. Without Procmail, the default system mailer
hopefully handles this just fine when mail is delivered to a system
mailbox, but if two Procmails are delivering two messages to the same
file more or less at the same time, you end up with problems.
(A typical symptom is that the first message gets a FFrom
in its first line, and the second gets a rom without the
F. This messes up the mailbox format thoroughly.)
Rule of thumb: Use file locking when delivering to a file. Don't use
file locking when delivering to /dev/null (because then it
doesn't matter if the message gets mangled, and you might not have the
permission to acquire a lock on a device), forwarding to another
address, or piping into a program. A pipeline which ends up appending to
a file should still use a lock, of course, since there is the same race
condition as when delivering straight to a file.
Rule of index finger: Using an unnecessary extra lock seldom hurts. (When it does hurt, you'll notice. :^)
Examples:
:0: # Deliver to a file, let Procmail figure out how to lock it * ^From scooby scooby :0 # Forwarding; no locking required * ^TO_dogbert ! bofh@dilbert.com :0:snoopy.lock # Explicitly name a file to use as a lock * ^Subject:.*snoopy | $HOME/bin/woodstock-enhancer.pl >>snoopy.mbox
The last one might need a little elaboration. When mail with a
Subject: line containing the word "snoopy" anywhere in it arrives,
Procmail will create a lock file called snoopy.lock and
hold on to that file while executing the recipe (in this case, piping
the mail to a program called woodstock-enhancer.pl and
appending the program's output to the file snoopy.mbox).
If a second message matches the same recipe, the second instance of
Procmail which attempts to deliver that message will politely wait
until the first Procmail lets go of the lock file.
(You don't really need to name the lock file in this isolated case.
Procmail would automatically deduce a lock file based on the name of
the file you're appending to, snoopy.mbox. But you might
have several recipes which do separate things which need to be
serialized somehow, so that one doesn't occur before the previous one
has finished its critical section. Say, maybe you have a second recipe
which modifies the program woodstock-enhancer.pl itself, or,
more realistically, modifies a separate data file used by that
program. You can't have that running while this recipe runs, so they
have to share a lock, and you have to tell Procmail
the name of the lock file to use.)
A (somewhat less common, but still) newbie mistake is to put the name of the file you want to write to as the name of the lock file to use, too. This effectively prevents Procmail from doing anything at all. The meaning of a lockfile is, "if this file exists, you have to wait."
The manual really does explain most of this, although it's not very explicit about why you would or wouldn't want to use locking in the first place.
:0: * ^TO_johnny@lunatix\.com ! jjasmith@ppp.home.in.isp.net
This will forward to the address jjasmith any mail
addressed to johnny. The trailing colon says to use a
lock file, but Procmail can't figure out what file you want to lock
because there is no file involved anywhere. Indeed, locking doesn't
make any sense here, and you'll get warning messages in the log
stating that Procmail "couldn't determine implicit lockfile."
Remove the second colon on the first line and all will be fine.
(If you think you want to use a variant of this recipe, don't. Read the BCC question below.)
Q: "Regular expressions?" What's that? Wouldn't it be much simpler to just use glob-like patterns? Does this mean I have to learn something new again? Augh.
A: Yes. Don't bother with Procmail if you want something simple and stupid. But getting started really doesn't take that much, all things counted. Start with a couple of the "quick start" or "tutorial" documents you'll find on the web, then try the manuals, especially procmailex(5). See if you can figure out the examples. Try a dry run from the command line. Before you write any major recipes of your own, create test cases for the things you find puzzling (because you and the documentation were not on the same wavelength, or perhaps because you are unfamiliar with regular expressions) and work them out with a simple test message or two. Learn to use the log from the start.
Oh, and good luck. Resistance is futile.
You will be assimilated. :-)
A:
Rephrase: Forward a copy, then proceed with normal delivery
(which ends up saving to your normal inbox on the server,
unless other subsequent Procmail recipes in your .procmailrc
are triggered, and end up delivering the message elsewhere).
:0c # That's colon, zero, lowercase cee ! self@other-place.net # That's exclamation mark, address to forward to
To do this conditionally, add conditions. To not save a copy on the server,
omit the c from :0c
For the distressingly common question, "how do I do this but
without leaving a copy on the server", the answer is to
simply leave out the c flag. Then Procmail will
simply forward, and consider the message delivered with that.
(For completely unconditional forwarding, you don't really need
Procmail at all, though.)
Q: How can I save a message to multiple folders?
A: Same as above, except save instead of forward.
:0c: # That's colon, zero, lowercase cee, colon folder-1 # Name of first folder :0c: # Second folder folder-2 # Repeat for each folder, perhaps even conditionally for some (or all) :0: # Last folder, no cee folder-n
To do this conditionally, add conditions. See the procmailex manual page for some examples.
If the folders you want to save to are directories, you can just list them:
:0 # No locking when saving to directories folder1/ folder2/ folder3/
Understanding deliveredness is probably a prerequisite for understanding
what the c flag does. See
below.
Q: How can I forward to many addresses?
A: In the simple case, just add more addresses to the action line:
:0 * ^Subject: result from cgi-bin/www-feedback$ ! first@one.com second@two.net third@three.org
If you have a largish list of recipients, you might prefer to store the addresses in an external file you can edit without mucking with your Procmail filters:
# The file $MAILDIR/addresses.txt contains the recipients, one per line :0 * ^Subject: result from cgi-bin/www-feedback$ ! `cat addresses.txt` # ^ Make sure those ^ are backticks, BTW (ASCII 96)
This will break as soon as the output of the backticks is too large for your shell to handle. You can use more trickery to get around that, but why not resort to a ready-made solution instead? A mailing list manager will scale to thousands of recipients and have some provisions for handling bounces, preventing mail loops, and automatically adding and deleting subscribers. (You might want to look at SmartList, which runs on top of Procmail.)
By the way, if mail to one address should always be redirected to one or more other addresses, you should read the "virtual domain" section below. (Don't use Procmail for this. The universe will implode if you do.)
A:
You "clone" it with the :c flag.
# First, we forward a copy to Don :0c ! gillette@hedgehogs.com # Then, we stash it in a folder :0: copied-to-don
See also the answer to the brace question.
If you have both the :f flag and the :c
flag on the same recipe, you are probably very confused.
Q:
I get these procmail: Skipped "(something)"
messages in the log, what do they mean?
A: They're Procmail's way of saying you have a syntax error in your recipes.
One frequent misunderstanding occurs when you have several actions which you would like to do in response to a matched condition. There must be one "colon" line for each action, and each action must have a colon line. See examples above and below, especially the question which has the answer "use braces".
A: It's not a "backup file", it's supposed to be backing up to a directory, and if that directory doesn't exist, it will not work. Read the paragraph before the example on the procmailex(5) manual page.
A: You can write to a file instead of a directory, of course, but then you will face a different problem: Removing old messages from the beginning of a file is much more resource-intensive (and somewhat harder) than removing the oldest files from a directory. Perhaps you should save to the same file all the time, and periodically rename the old one from underneath. You could set up a cron job to do the renaming at midnight; the cron job could also remove backup folders older than, say, three days at the same time.
A:
Typically, you want to add or change a header. This sounds like
formail. The only thing that remains then is to actually
send it off. You can of course filter first and then send (see next
question), but you might as well do both in one fell swoop (unless you
also want the modified message in your normal mail stream; again, see
the next question for more).
:0c * ^TO_sales@pizzazz\.tm\> * ! ^X-Loop: sales@mundane\.domain\.net | formail -k -X "From:" -X "Subject:" \ -I "To: sales@mundane.domain.net" -X "To:" \ -I "X-Loop: sales@mundane.domain.net" -X "X-Loop:" \ | $SENDMAIL $SENDMAILFLAGS -t
This is almost like a real-world example. It will (a) trim down
the headers considerably, sparing only the From: and Subject: of the
original. Then (b) we add some headers of our own (remember to
extract them with -X too!), and (c) the results are handed to
Sendmail. The -t option means the To: (and Cc: etc) lines
in the actual message contain the recipient's address.
If formail won't do the modifications you want, you are of course to replace it with whatever you fancy. The basic pattern is the same, anyway: pipe to the program which "fixes" the message, then pipe the results to Sendmail.
If the results don't contain suitable headers, or might contain e.g.
your own address, you should take care to tell Sendmail explicitly who
to send it to, rather than rely on sendmail -t.
Q:
How can I change the contents of a message
but otherwise proceed through my .procmailrc as usual?
A:
This is what the :f flag is for.
# Add an "X-Likely-Spam: ugh" header to all messages # which have an X-UIDL header :0fhw * ^X-UIDL: | formail -I "X-Likely-Spam: ugh" # Continues here with the slightly modified message ...
If you have both the :f flag and the :c
flag on the same recipe, you are probably very confused.
(An X-UIDL is not necessarily a sign that a message is spam. This header is added by some POP programs. If you don't use POP, or your POP doesn't add this header, it's fairly likely to be spam, but you can never be too sure. Somebody who uses POP could have resent a message to you, for instance, X-UIDL header and all.)
A: Procmail's variables are exported to the environment of any process you run from within Procmail (with a few exceptions which, typically, you don't need to worry about). (This also means that if the script is yours to modify, you might not even need to pass in anything explicitly -- just make the script read the appropriate environment variable. However, this is usually a bad solution because you don't want your scripts to depend on environment variables too much.)
In the following example, we are grabbing the contents of the Subject: header into the variable SUBJECT, and then pass that in as the -s option of our-script.
SUBJECT=`formail -zxSubject:` :0w: | our-script -s "$SUBJECT" >>output
This construct is a good general solution, but in this particular case, when the information the script needs is simply the value of a field, we should avoid the overhead of the external process call (here, formail).
There are two ways to accomplish this savings: (a) since the script gets the message on standard input, it would probably be fairly straightforward to modify the script so it gleans the required information directly from the message's headers; or (b) you can use the MATCH construct to do exactly the same kind of grab you get with formail:
:0w: * ^Subject:[ ]*\/[^ ].* | our-script -s "$MATCH" >>output
Do note that the condition, which we use for its MATCH-grabbing side effect, still has to match, i.e. the script will only be run on messages with a non-empty Subject header.
The whitespace between the [] brackets consists of a space and a tab, as usual.
A: Procmail can save messages to various formats, some of which involve individual files in named directories, others are flat files to the end of which Procmail appends new messages, etcetera. Appendix A contains some helpful descriptions of various folder formats.
Q: So what's a "spool file" then?
A:
Your primary mail spool is whatever the $MAIL environment
variable is set up to point to. Most mail readers will know to fetch
new mail from this file (some even let you use it as any other folder,
and keep old messages there as well. This might or might not be something
your administrator dislikes).
For some mail programs, it's important (or at least convenient) to separate the location where mail is kept from the place where new mail is delivered. (Emacs and Netscape might be good examples.) These will typically take care of momentarily locking the spool file whenever you fetch new mail, and want exclusive access to the real folder where mail is permanently stored.
Incidentally, the procmailex(5) manual page briefly
explains the locking conventions used by Emacs. In practice, just
ignore the locks that Emacs keeps and use spool files. (This is what
the comment about movemail is supposed to mean.)
With spool files, you probably want one spool file (or even several) for each folder. It might make sense to put these spool files in their own directory, or make sure they follow a naming convention so it's easy to tell which files are spool files and which are folders.
Q:
What does the second colon in :0: mean? Should I worry?
A: You just missed it. It tells Procmail to use locking on this recipe. Go back and read the above snippet about file locking. See the next question.
Q: What does "Couldn't determine implicit lockfile" mean?
A:
Briefly, that you have :0: where you should have
either a named lock file or just :0
(ignoring any possible flags here).
See previous question. Hope this helps.
Q: What does "Extraneous locallockfile ignored" mean?
A: See previous question. Hope this helps.
Q: What does "Extraneous filter-flag ignored" mean?
A:
Briefly, that you have :0f where you should have
:0 (ignoring any other flags and possible lock file
you might have). The f flag is perhaps a bit poorly
explained in the manual, seeing as lots of people are using it
where they shouldn't (probably "filtering" sounds too enticing
in the manual for a filtering program -- of course you want to
filter your mail, dear).
Q:
Okay, then what is the significance of the number zero on the
:0 line?
A: Nowadays, none really. In old versions of Procmail, you had to tell it how many condition lines your recipe contained (yes, "bletch"). The syntax was later extended and the number zero special-cased to mean, all the following lines which begin with asterisks are condition lines. (The asterisk at the beginning of each condition is obviously also part of this extended syntax.)
A:
Setting MAILDIR sets Procmail's working directory to
the named directory. Relative path names will be relative to this
directory from then on (remember, Procmail basically scans your
.procmailrc from top to bottom and it's perfectly
permissible to change some setting somewhere along the way).
When started from a .forward file or similar, the
initial MAILDIR will be the user's home directory.
A:
This is just a conventional way to talk about the line that starts with
the five characters "From " at the very beginning of the
headers of a mail message. This is a tricky line for several reasons:
From: header
(note the trailing colon on this one). The underscore is
used partially to underscore [sic] the fact that we are
talking about this colonless pseudo-header, not
the From: header proper.
From_ lines often lead to messages
being split where the unescaped From was,
because the MUA genuinely thinks this marked the start of
a new message, and so the recipient sees two messages, the
first ending abruptly in the middle of nowhere and the second
continuing where the first left off, only some of the actual
message text might show up as rather mysterious-looking
headers instead.
The format that uses the From_ line as message
separator is commonly referred to as "Berkeley mbox format." Not
all systems use this format, but it's pretty much a de facto standard
on Unix. A related format uses the Content-Length: header
instead, but keeps the From_ line, leading to
utter and hopeless confusion as to which is which.
A: The usual reason people are asking this is because the following, which is basically the way to do it, feels clumsy or something.
:0
* condition1|condition2|condition3|condition4
{ ... do something about it ... }
There are situations where you can't do this, such as when one of the conditions is a negated condition. You can try to fool around with de Morgan's laws to get you somewhere where this doesn't play in, or use scoring:
:0
* 1^0 condition1|condition2
* 1^0 ! not condition 3
* 1^0 ? /path/to/external-program whose exit code we want to look at
{ ... now do something about that instead ... }
Some side effects are different when you resort to scoring; for instance,
if you are also using the \/ operator to grab stuff into
$MATCH, scoring will generally grab the last matching line,
whereas a straightforward regex OR will stop already at the first one.
(You can change the 1 in 1^0 to some really big number
to prevent this. See the procmailsc(5) manual page if
you wonder about the significance of these strange numbers.)
Q: Can I list several actions under a common condition? How?
A: Here you go:
:0
* common-condition
{
:0fbw
* perhaps an additional condition on this one, even
| sed -e 's/definately/definitely/g' -e 's/seperate/separate/g'
:0c:
* perhaps an additional condition on this one, too
action1
:0
! action2
}
:c flag either (in the general case. Of course you
should have a :c if you want Procmail to branch, and do
the braces in one branch, and continue basically as if this recipe
hadn't matched in another).
.procmailrc.
See also the
description of the :c flag above.
A: Thusly:
ARG="$1" :0 * ARG ?? regex action
You can test against any Procmail regular expression. $1
is unusual in that you can't test it directly, you have to assign it
to another variable before you can actually compare.
You might want to
anchor the beginning and end of the match with the ^^
special anchor (so to match "str" but not "string", "Strauss", "Dnestr",
or "tapestry", you'd say ^^str^^.
Also note the absence of a dollar sign on the variable name.
Of course, this is all in the manual, by the way ;^)
A: Good girl/boy. Here's a partial list.
^TO_.*".
Just leave out the condition line completely.
.procmailrc file.
See the "Gotchas" section
for an example.
\/ in a regular expression
when you really mean just a literal slash.
\.
-- that's a backslash and a dot.
A lone dot, of course, will match any one character.
(It would be prudent to not count on \{
matching a literal opening brace in all future versions
of Procmail. An unescaped opening brace can rather safely
be assumed to work as it always has, matching literally.)
\/ special token
and the \< and \>
word-boundary operators.
(The characters < / > just match themselves.
Repeat: To match a literal foreslash, just type a
literal foreslash. Ditto-ditto for less-than and more-than.
The only use for double backslash is to match
a literal backslash character,
except at the beginning of a regex,
but you don't want to learn about that yet.
Read the
backslash part of the gotchas section
when you've grown a little bit bigger.)
.* or (something)?
is always completely redundant.
^.* doesn't serve any useful purpose, either.
.* just after
^TO or
^TO_
is also superfluous
and will actually dilute the effect of those macros,
because they already contain a wildcard expression at the end
which is better bounded
and thus both more efficient and more user-friendly
(unless you specifically want matches to start anywhere,
including, for example, in the middle of a word)
than just "any character any number of times."
dogs?" will indeed match
anything which contains "dog" or "dogs", but also "dogma" and
"endogenous" and any other word containing the three letters d-o-g.
And so, the optional s at the end is not really doing anything
useful there. "dog" will match "dogs" just fine, and the s is not
preventing matching on words which don't contain an s. A better
expression might be \<dogs?\> -- this will
require the match to be a "word" (token) of its own.
.procmailrc files you find on the Net
will contain a LOCKFILE=some/file clause which in most
circumstances can be seen as redundant. (It basically ensures that
you can only receive no more than one message at a time, which is
nice if you absolutely need to keep your log file clean, but potentially
a large waste of resources as it can create numerous apparently "hanging"
Procmail processes waiting for their turn if you receive many mail
messages in rapid succession.)
DEFAULT or SENDMAIL
unless you are fairly confident you know what you are doing.
That goes doubly so for ORGMAIL --
if you're changing it then you're probably going about
the problem the wrong way.
/dev/null,
or using your actual inbox as the lock file for itself :-)
can leave Procmail hanging and consuming lots
of resources.
/dev/null is special-cased
in newer versions of Procmail so that shouldn't be dangerous.
As for the "elegance" of attempting that ...)
:0
* condition
{
:0c
! staff@example.org
}
is just a redundant paraphrase of
:0c * condition ! staff@example.org
Don't let that c flag confuse you, it won't do anything
until all conditions have been met.
(This could perhaps be more explicitly documented.)
/dev/null.
If the abstraction level of the manual pages doesn't match your taste, perhaps you want to try some of the tutorials on the Links page (you could even try the Rocket Science ones if you feel you understand the basics) or the Quick Reference.
(For completeness' sake, it should be pointed out that some of the above
rules of thumb are a bit too general. For example,
after a \/ grab operator, .* or .+
obviously serves a useful purpose if you want to match the entire remainder
of the line for grabbing purposes. A leading ^.* can be useful
if you want to gobble up leading newlines if you have a ^ or
$ later in the expression which should not be preceded by
nothing.)
A: This is a bit complicated and basically belongs in the "advanced" section. But here's a pretty good explanation. (Thanks to Philip Guenther.)
Q:
Why shouldn't I use the * wildcard?
A:
First off, make sure you understand that the regular expresision
"a*" matches
the empty string, and the string "a", and the string "aa", and the
string "aaa...". Second, blanket wildcards like .* are
often a bad idea. When you're dealing with regular expressions,
it's usually best to be as specific as possible.
Example: Say you have a desire to deny all mail from the domain
wash.edu, perhaps only temporarily but somewhat urgently
because a single user there is harassing you.
Since most headers are rather easy to forge, you want to reject any
mail with this domain name in the Received: headers,
which are hard to falsify. (Specifically, it's hard to prevent
"wash.edu" from occurring somewhere in the headers if you post from
wash.edu.) So you say
:0 * ^Received:.*wash.edu /dev/null
/dev/null
(if it's harassment or spam, you want to keep the evidence
and complain to the abuser's ISP or upstream, and possibly the police),
this recipe will match on this (fairly short) string anywhere in any
Received: header. So if you have friends mailing you from
hogwash.edu, you are blackholing their mail, too.
Or if somebody is sending you legitimate mail using
software which includes "wash.edu" in its version information
in the headers (not too unlikely, seeing as a lot of mail tools
come from Washington), that will be gone as well.
Thus, it is better to be as specific as you can. After a glimpse at the headers of the messages you have received so far, you determine that each Received: line starts with the word "from" and the name of a host. If that host name is in wash.edu, that is mail we want to reject:
:0 * ^Received: from ([^ ]+\.)*wash\.edu([ ]|$) /var/tmp/harassment
/dev/null, but merely to
a low-priority folder where they won't fill up your quota even
if there is a lot of them coming in.
(Actually, this Question is made up, but various aspects of the Answer to it are Frequently Overlooked so I smuggled it all in here.)
(And actually, for Sendmail's Received: headers, you want to look at the second token after the "from" word. The first can likewise be forged and is commonly set to a bogus value by the sort of software people use for mailbombing people they don't like.)
(... Oh, and actually you probably want to leave a hole of some sort in the shield so that postmaster mail from wash.edu can make it through to your regular inbox. The most obvious tricks can most obviously be predicted and abused by a clever harasser, though. You should perhaps get on the phone to that postmaster person instead.)
The light bulb contains the seeds of its own revolution.
Before you panic, try adding VERBOSE=yes to your
.procmailrc, send yourself a test message, and examine
the information about Procmail's actions that are written to the log.
This tells you, in excruciating detail, about every decision Procmail
makes while processing your rc file.
If you have a message which is being processed differently from what you had expected, save it to a file so you can rerun it through Procmail from the command line, or show it to others when you decide you need to ask for help.
Some common error messages are explained briefly above. Go back and look for "couldn't determine implicit lockfile". (Search for the error you're getting instead if this is not it.)
All of Procmail's own error messages and some plausible explanations
are listed on the Procmail manual page. If you haven't before, now is
the time to read the manual pages. Note that there are several;
procmailrc(5) explains the syntax of the rc
files while procmailex(5) contains some good real-world
examples. There is also procmailsc(5) which discusses the
scoring system.
Free tip: You'll avoid a lot of sarcastic replies if you make sure you try this before you post to the Procmail mailing list or some newsgroup about your problems.
:-)
Do note that an interactive Procmail gets a different environment and possibly runs on a different machine than the Procmail in your .forward, though.
TMPDIR=/tmp/$USER
MAILDIR=$TMPDIR/procmail.out
DEFAULT=$MAILDIR/$USER
VERBOSE=yeah
SHELL=/bin/sh
:0
* ? test -d $TMPDIR || mkdir $TMPDIR
* ? test -d $MAILDIR || mkdir $MAILDIR
{ }
:0E
{
# Bail out if either directory didn't exist and couldn't be created
EXITCODE=127
HOST
}
# ... your experimental recipes here
With this recipe, you can use Procmail directly from the command line, typing in experimental messages as you go along, or feeding it a message on standard input, like this:
procmail -m experiments.rc <test.mbox
(assuming you had saved the above example rc file in experiments.rc and have a test message for it in the file test.mbox).
The -m option is rather vaguely documented;
it will arrange with additional safety nets by disabling
the default delivery and treating file names as relative
to the current directory (as opposed to your home directory,
which makes sense when invoking Procmail from a .forward
or similar).
[Click here to download a copy of experiments.rc]
(Incidentally, testing Procmail from the command line like this
is much more convenient than sending yourself a test message
and waiting for Sendmail to deliver it to you.
It also frees you from having to put diagnostic logging in your
regular .procmailrc, or having to put verbose
logging in your regular log file.
FWIW,
here's an old message from the mailing list
with an example of how to debug a "mystery recipe.")
[Nyfb abgr gur zbebavp sbyybj-hc :-]
This listing is by no means exhaustive. Philip Guenther's todo list and the companion warts list contains some items which are not here, although I do try to keep in sync with Philip generally.
This is mostly for version 3.11pre4 and newer. If you have an older version, you should probably upgrade. The HISTORY file in the distribution package contains a list of fixed bugs in earlier versions (although it's not very detailed).
See the malloc(3) man page if you're on FreeBSD and wonder what this means.)"Changing malloc's options seemed to do the trick for now (
ln -s '<' /etc/malloc.conf)."
\/ doesn't work correctly.
Actually, it has nothing particular to do with the '+' operator -- you get the same effect with the '*' operator. The real problem is that if you have something like " +.*", the '+' is superfluous from the point of view of what the regex should match, but it adds to the combinations that procmail tries as it goes. Along the way, procmail loses track of where the match really started. Yes, it's a mess. The workaround is to remove the superfluous operator ('+' in this case). That'll also make it match faster.
LINEBUF can give you core dumps.
:0 variable=| echo "$variable" | tr A-Z a-z
variable will be empty by the time the
echo executes.
formail -r breaks RFC822
formail -rt if you don't know what this means.
Perhaps you should always use it anyway.
Never mind that the documentation says something funny about
how the -t option implies "trusting" the sender.
Without -t, you stand a good chance of replying
to the envelope sender, which is a violation of the standard.
(See next item.)
formail -r or formail -rt
is not documented
.deb package was correctly compiled
to reflect this.
GROUP_PER_USER
switch in config.h which needs to be enabled
before compiling Procmail. (This was not done for Red Hat's
procmail-3.13-1 RPM but should hopefully be fixed in
the next. An independently fixed RPM seems to be available at
http://cs.wilpaterson.edu/~scottw/shebang/
but this appears not to be "official".)
"Service unavailable"
which is not very helpful. (See the
smrsh question
below, too.)
egrep compatible
egrep but still its very
own implementation with its own features and drawbacks. Read
the rest of this section, and the manual pages.)
The following things are frequently regarded as unintuitive or even "broken" by many people.
^FROM macro
^TO macro which catches all variants of
To:, Cc:, Resent-To:,
etcetera, there also ought to be a ^FROM which
would catch, say, From:, Sender:,
Reply-To:, and so forth. But this is in fact not
the case, and Stephen van den Berg has repeatedly stated that
he will not add such a special macro.
FROM="^(From[ ]|(Old-|X-)?(Resent-)?(From|Reply-To|Sender):)(.*\<)?"
# Use thusly:
:0
* $ ${FROM}billg@microsoft\.com
| $HOME/bin/throw-virtual-cake
^TO does;
you use it at your own risk, of course.
(This is provided as an example. Compare to the documentation for
^TO_ and pick the things you like from both.)
FROM variable.
See
below.)
:0 | formail -AMoo: -IFoo: -r\ -aBoo:
:0 | formail -AMoo: -IFoo: -r-aBoo:
* Condition which wants to continue \ () with leading whitespace
* VAR ?? ()\$string which starts with a literal dollar sign * VAR ?? [$]string which also starts with a literal dollar sign * VAR ?? (\$)third alternative solution
bar='quux' :0 * foo$bar baz :0 * $ foo$bar mumble
($) should always be unambiguous.)
.signature and leave a notice
about what happened:
sed '/^-- $/,$c\ -- \ sed: .signature removed'
NL=" " :0bfwi * ^From:(.*\<)?dquayle?@aol\.com\> | sed '/^-- $/,$c\'"$NL"'-- \'"$NL"'sed: .signature removed'
sed. The newlines are in double quotes while
all the other stuff is in single quotes. Do pay attention
to the first lines which define the NL variable,
though -- that's NL equals
double quote, newline, double quote.)
SEDKLUDGE='/^-- $/,$c\ -- \ sed: .signature removed' :0bfwi * ^From:(.*\<)?dquayle?@aol\.com\> | sed -e "$SEDKLUDGE"
.procmailrc file
.procmailrc
file on a local PC and then upload the file to their
shell account with ftp. If you do this in
"binary" mode, the line endings will likely consist of DOS
carriage return/line feed pairs, rather than just line
feeds (the convention on Unix). Procmail will not ignore
these "extra" characters, and the errors that will be logged
will look quite odd (like this maybe:
"rocmail: skipped "
#!/bin/sh if ! mv .procmailrc .procmailrc.old ; then echo Could not move .procmailrc to .procmailrc.old >&2 exit 1 fi tr -d '\015' .procmailrc.old >.procmailrc
.procmailrc.old after making
sure the new file works as intended.
-A,
or od(1) can be helpful.
$DEFAULT mailbox (or spam-filtered by
some particularly ominous spam filter that would otherwise
have been bypassed, if you have that sort of stuff
near the end of your .procmailrc).
:e flag immediately after it,
but this is hardly something you want to do for each
and every recipe.
csh or tcsh),
prepare for havoc.
SHELL=/bin/sh
at the beginning of your .procmailrc
t?csh
and would like to look further into this, the FAQ
author has some ideas for things to check.)
$@ is only available to external programs
BUGS
section of the manual, so perhaps it's not a feature :-)
-- if you need to use $@ internally, you'll have to go
via an external program such as echo:
:0ir ARGS=| echo "$@"
"$ARGS" subsequently is still not
exactly the same as "$@" is to the shell.
$SENDMAILFLAGS gets passed in double-quoted
by the ! action
! actions
with
| $SENDMAIL $SENDMAILFLAGS ...
.rc file the error is happening on
VERBOSE=yes and it should become apparent
where the problem is.
formail -A or -I
don't imply -X
-A or -I flags
in a command which also uses -X, you have to
explicitly add an -X for the fields you just
added.
formail -rt -XTo: -A"X-Loop: me@my.isp"
X-Loop header
immediately after inserting it.
You have to extract it explicitly, even though you just added it:
formail -rt -XTo: -A"X-Loop: me@my.isp" -XX-Loop:
formail has a fixed order for parsing its arguments
:-)
formail and lockfile won't tell
you what their version number is.
\/ operator is a bit unwieldy. The way that it changes
the longest-match behavior of the regex engine will bite you one day
if it hasn't already.
See
the related question in the previous section.
MATCH
-- either you get everything after the \/ operator, or nothing
at all. It would be nice to be able to specify a trailing context for what
you want to match, which would still not get copied into MATCH.
$\VAR interpolation operator will add a set of empty
parentheses at the beginning of the interpolated value.
i flag to suppress warnings
about data not being read, but it's a kind of a kludge.)
.rc file.
There is a patch for this.
It will probably be adopted in slightly modified form in a coming release.
Q: What are some typical installation problems?
A: Here are some of the most typical problems with getting Procmail to run in the first place:
.forward file, it could be that your site is
using
smrsh,
the Sendmail Restricted Shell, for mail delivery.
In this case, you should get a bounce message saying
something like
"Address <foo@bar> is unsafe for mailing to programs"
or
"Cannot use <metacharacter> in command".
Your admin has to explicitly tell smrsh
that Procmail is a program you are allowed to
call up from your .forward
(if you are -- you might not be, since
there are security risks involved;
users can basically run any program they wish
from within Procmail,
thereby invalidating the whole idea with smrsh.
Look for a compile-time fix in an upcoming version of Procmail, though.
The administrator will then be able to restrict which programs the user's
Procmail can execute ... and then you might even be able to persuade your
admin to install Procmail as your local MDA for this feature alone).
Received: lines from
old messages to find out what routes mail to you usually
takes -- the lines closest to the top are the most important.
If the mail server is a different architecture,
or can't mount your home directory, or whatever,
you will have problems. Ask your site admin for help.)
SHELL=/bin/sh
right at the top of your .procmailrc.
See also various points in Stephen's original Procmail FAQ.
Your humble FAQ author has a web page with some debugging tips at http://www.iki.fi/era/mail/procmail-debug.html
Q:
What are all those mysterious things I'm supposed to stick in my
.forward file?
A:
First off, for the record, you don't need a .forward
if Procmail is your local MDA. Ask your admin.
If Procmail is not your local MDA, you need something more or less
like what the manual suggests. Look at the NOTES
section of the procmail(1) manual page -- this is
tailored by Procmail's installation script depending on what MTA
you have installed, so if the manual page's suggestion looks very
different from the following (usually much simpler; see
below)
then the manual page's suggestion is definitely worth a try
before you try this.
The following is not exactly the same as what Procmail's manual page suggests, because it has some decorations which people have come up with on their own, which have become rather widespread and thus are probably worth a bit of explanation. So, here's one popular variation:
"|IFS=' '&&p=/usr/local/bin/procmail&&test -f $p&&exec $p -f-||exit 75#whatever"
(That should all be on a single line. Unwrap it if it appears wrapped.
Unwarp it if it appears warped. The surrounding double quotes are
significant, and should be included in the file.)
This is just a simple (but slightly convoluted) shell script.
Whitespace is often left out as much as possible, partly out of paranoia,
partly because people want it all to fit onto a single line in their editor.
Please note that this is not guaranteed to work out of the box.
Your first attempt at a .forward should be what your local
manual page suggests (and please, people, change
#whatever to your own login name with a hash mark in front).
Here's what it does:
&& "and" connective means, do the next
action only if the previous action(s) succeeded.
|| "or" connective means, do the next
action if the preceding action(s) failed.
IFS=' ' sets the shell's "internal field separator"
to a single space. This is mostly just in order to prevent some
old Sendmail hacks which were based on setting the IFS to something
else.
p is set to where-ever you expect
to find your Procmail binary. This allows us to use the convenient
shorthand $p instead of this longish pathname,
and also means that if you ever move it, there is only one place
where you need to remember to change it.
test will fail if the file named in $p
doesn't exist.
exec replaces this script we're currently executing
with Procmail if the binary is successfully loaded.
The parameter -f- says to Procmail to regenerate missing
From_ lines and update the timestamp on existing ones.
exit only happens if either the test or
the exec failed. If this is the case, the exit code
number 75 is passed back to Sendmail, which interprets it as a
temporary mailer failure and requeues the message (we hope).
(See also the
EXITCODE topic
elsewhere in this FAQ.)
exec succeeded, sendmail will
never reach here. Whatever the exec executes replaces
whatever was reading the .forward file and whatever exit code
that returns will be returned instead.
#comment doesn't do anything, but serves to make
sure your .forward file is unique, in case your site
is running a braindead over-optimizing Sendmail.
(See the
explanation for this in Stephen's original FAQ.)
If your MTA is not Sendmail, something reasonably simple like
"|/usr/local/bin/procmail -f-"
.forward style.
Anyway, the documentation for your MTA should hopefully tell you
all you need to know.
(If you have a hard time even figuring out what your local MTA is, the appendix on figuring out the mail flow might be worth reading for background information and a trick or two.)
The Filtering Mail FAQ has a section with
more .forward file variants you can try
if the one in the manual page and the ones given above don't seem to
cut it.
(You will find the same information in
Infinite Ink's Procmail Quick Start,
too.)
Q:
I have this recipe which I basically copied from someone's working
.procmailrc but it gives me funny errors such as
"Badly placed ()'s" or "formail: not found" or "formail: formail: cannot open".
What's happening here?
A:
The first error message is a Csh error message (also "Too many ('s")
-- add the following to your .procmailrc and try again:
SHELL=/bin/sh
The second and third errors comes from a Bourne shell which can't find the
command named formail. It probably means that your
PATH is not set up properly, or that a program
which the recipe needs doesn't exist on your system. (You might also
see "/bin/sh: formail: cannot open" which makes a lot more sense than
the example above. Don't ask me why some Bourne shells will print the
command name twice.)
It could also be because the recipe author used the (somewhat dubious) convention of defining commands as Procmail variables and using these variables in the actual recipes:
:0
* ? $FORMAIL -XFrom: -XSender: | $FGREP cyberpromo.com
/dev/null
The above will only work if the variables FORMAIL and FGREP are set
to suitable values such as FORMAIL=/usr/local/bin/formail
and FGREP=/usr/bin/fgrep earlier in your
.procmailrc. (And you can't just snatch these values
from here either -- ask your local gurus [or the recipe author]
to help you out if you don't understand what this does.)
Q: Yikes! Where did my mail go??
A: Usually, to a file in your MAILDIR. Look in the log for clues (you were using the logging facility, yes? Yes?) and in your MAILDIR for files called something funny such as "*" or ":0:" or "# I wonder if Procmail allows comments here" or "forwardee@address.net" (the quotes are not part of the file names and these file names are not necessarily representative).
procmail: Skipped "^Subject:.*test" procmail: Locking "*.lock" procmail: Opening "*" procmail: Unlocking "*.lock"
A:
Make sure you are using a reasonably new version of Procmail.
According to archeologists, versions prior to 3.something don't
understand the
:0 syntax
Q: Why is Procmail writing to the console? Can I stop that?
A:
It's not, unless you specifically messed with this setting when you
compiled it. Unmess and recompile. Another popular mistake is to
somehow enable the MMDF message separator string (^A^A^A^A,
four ASCII 01:s) before compiling. (This setting is the first thing
in config.h and is kind of easy to uncomment so it's not
all that hard to change it by mistake if you're using, uh, ahem,
something else than the One True Text Editor.)
Q: What are these fields that get written to the log?
A: In order of appearance
From_ line of the original
(which conveniently includes a fresh date stamp,
which you can use for various things such as
determining the approximate time of day
without calling date(1) for each
incoming message);
Subject:;
MAILDIR if it doesn't have an explicit path
(see previous question, by the way), or
a shell script pipeline,
which will be your local Sendmail or imitation
in the case of a forwarding recipe;
This is documented under LOGABSTRACT in the procmailrc manual page.
A: It's supposed to work that way. What the recipe does is, if formail detects a duplicate, it will succeed, which makes Procmail think the message is now delivered.
Q: Uh, I'm not sure I understand that.
A: It's a bit of a play with side effects. For reference, here's that recipe again:
:0Wh:msgid.lock | formail -D 8192 msgid.cache
Let's paraphrase this recipe as a dialog.
There are no conditions, so the action line is done for every message
that gets as far as this recipe. The h flag says to only
show the headers of the message to the action line, and the W
flag says to wait for the program in the action line to finish
(so we can look at Formail's exit code).
Procmail says, "hey formail, save this message for me."
Formail eats the message (actually just the headers, because of the
h flag), extracts the Message-Id, and checks if it's
already in the cache file.
If not, formail says "sorry, no can do." Procmail catches this error,
says "aw shucks, you couldn't save it? I have to continue processing
this message then" and wanders off to the next recipe.
However, if the Message-Id was already in the cache, Formail doesn't
generate an error, and Procmail (somewhat stupidly, it might seem)
assumes Formail took care of that message
(i.e. saved it somewhere, for example)
and that is is thus now delivered and no longer a problem of Procmail's.
A: It's a feature. Normally, you want Procmail to keep track of what your recipes are doing, and salvage your data if something goes wrong with one of them. If you have a recipe like this:
# Old Stan always writes boring messages, # we only really need to see the first five lines of 'em :0fbw * ^From: stanislaus@poland\.com | head -5
it will work some of the time, when the message from Stan is short
enough, but that's a coincidence. With a longer message, though, Unix
starts paying attention to what is happening, because it will have to
buffer some of the data, and then when the buffered data is never
read, an error occurs
(specifically, a SIGPIPE signal is generated).
The error is passed back to Procmail, and
Procmail tries to be nice and give you back your original message as
it was before this malicious program truncated it. Never mind that in
this case you wanted to truncate the data. Anyway, the fix is
easy: Just add an :i flag to the recipe
(:0fbwi instead of :0fbw) to make Procmail
ignore the error.
A:
Procmail by default sends a notification message for all delivered
messages. The COMSAT pseudovariable controls this;
see the documentation for details. So for starters, make sure you have
COMSAT set to the right thing -- in a verbose log, you
should see something like
procmail: Notified comsat: "you@offset:file"
for each delivered message.
Unfortunately, many systems come with an ancient version of comsat(8) which doesn't pay attention to the last part of this notification. Thus, you will see apparently random biff notices, or none at all for mail delivered outside your system spool.
The solution is to [get your admin to] install a comsat daemon which understands the newer, extended form of the comsat protocol.
Q: How do I get Procmail and Netscape to cooperate?
A: Netscape is not good about file locking and you will find that it forgets which messages have already been read every time Procmail appends to a folder. (Peter B. West has diagnosed this in some detail.) You can use Procmail to tag messages and use Netscape's filtering capabilities for the last mile or e.g.read your folders over IMAP and let Procmail deliver to the IMAP server instead. (suggested by Mike de Laine).
Netscape's JavaScript filtering capabilities are allegedly not all that bad, if you can find the documentation.
This section attempts to explain a little something about the context in which Procmail runs. Specifically, you need to understand this if you wonder why you can't match on the BCC header, how to cope with many users sharing the same POP mailbox, or generally want to do something like a virtual domain.
If that's not something you worry about, you should perhaps still read the next section. But if you want to skip directly to the other questions, you can click here.
Learn the following by heart:
SMTP doesn't care what's in the headers of a message. It goes where the envelope says. The headers don't matter. The headers don't matter. The message goes where the envelope says.
What, the astute reader asks, does this have with Procmail to do?
Well, Procmail is frequently used to handle mail. And Procmail can only look at what's in a message's headers. And since the headers might just contain more or less anything, trying to let Procmail figure out whom a message for is pretty futile.
Nevertheless, this is something a lot of people are trying to do.
If you are content with the answer "Procmail is the wrong tool for this," you can stop reading now and skip to the next question.
Let's take an example. The following message
looks like it was sent from foo@bar.net
to nospam-list@nospam.org.
And yet, it was actually sent by some mailing list package
at nospam.org and landed in your inbox,
because you're a subscriber to nospam-list@nospam.org.
How can this be?
From nospam-list-request@nospam.org Sun Nov 15 18:39:39 1998 Received: by yourhost from nospam.org and so on and so forth From: Frank O. Olsen <foo@bar.net> To: People Who Don't Want Unwanted Mail <nospam-list@nospam.org> Subject: How? How do I get off this list? -- Frank < ... 18 lines of .signature trimmed ... >
When a message is transferred over an SMTP connection, it's encapsulated. It's kind of like the message is nicely folded into an envelope, and the programs which handle it just look at whatever is scribbled onto the envelope. Then when the message reaches the MTA at the final destination, this program (typically Sendmail or Qmail) opens the envelope and delivers the actual message.
What happens when the Sendmail at nospam.org
sends the above message is that it passes in the -request address as the
envelope sender (which turns up in the From_ pseudo-header)
and your address as the envelope recipient. Fine and dandy, says
your local MTA, and asks nospam.org to follow up with the actual message.
Only at this stage are the headers (and the body, of course) transmitted
-- obviously, what's in the message itself doesn't matter anymore,
because the envelope data is what says where the message came from
and where it should go.
The Bcc: header, as implemented in e.g. Sendmail, is really just a special case of this: Before sending the message, Sendmail copies any Bcc: recipients from the headers to the envelope, zaps the Bcc: headers, and sends it off just like any other message.
Normally this doesn't affect Procmail a lot, because in the typical
scenario, Procmail already knows implicitly who a message is for --
the envelope sender data determines whose .forward (if
applicable) and/or .procmailrc gets read, and all of this
is nicely sorted out by the time your Procmail recipes get a chance
to do anything like deliver the message somewhere.
But what if you want to use your account as a switchboard for many
other addresses? Procmail is a fairly nice program for this sort of
application, but if you are feeding mail for several distinct
recipients into your switchboard .procmailrc, you have
to make sure you also somehow tell Procmail where the message should
ultimately go.
Here are some ways to accomplish that:
.forward file.
X-Envelope-To: or Apparently-To: to the
headers before passing it on to Procmail for delivery.
Some of the "related questions" below discuss some of these issues to some extent.
The default settings with Sendmail 8.9.x for Procmail as LDA don't
actually pass in the actual envelope recipient, but at least with
plussed addresses you get the part after the plus (as in, if I get
mail to era+pr@iki.fi I'd have access to the information
that it was sent to something +pr, but I don't know if it was sent
specifically to era or to an alias which points to this
address).
With Sendmail, using the Procmail mailer (not just Procmail as the default delivery agent) is a solution which works nicely (but you can't invoke that from plain aliases, you have to use something like the mailertable). (Thanks to Philip Guenther for information about how to do this with Sendmail.)
With qmail, you can have the MTA insert envelope information into the headers which subsequently get stripped by the local delivery agent. This is nice for POP sites where both the upstream and the local host run qmail. (Thanks to Bart Schaefer for this information.)
Q:
Why can't I match on the BCC: header?
A:
You can't match on the BCC: header because it is not
present in the message you receive. See
above.
This is what the "Blind" in "Blind Carbon Copy" stands for;
none of the recipients are supposed to see
who's been BCC:ed.
(Okay, so the spec has a few more twists.
This is how Sendmail does it.
Some other mailers will show the BCC list
to the people who are being BCC:ed.)
Q: The Received: header seems to often contain something like Received: from elsewhere (...) by somewhere (...) for <recipient@host.domain> even if the recipient was Bcc:ed -- can't I rely on that?
A: No. Next question?
The problem is that (a) not all MTA:s add this information, and (b) even if yours sometimes does, it might not always. E.g. Sendmail won't stamp the recipient in the Received: header if the message was sent to several recipients at your site.
Q: How do I implement a virtual domain in Procmail?
Or, is Procmail even the right tool for this?
A: The short answer is no; don't. See the section Why Headers Don't Matter, above.
If you stubbornly refuse to take no for an answer, you might want to look at Chris Lindsey's "solution"
It is an unfortunate fact that many sites offer "virtual hosting" without providing a robust way to forward mail to the client with all the information intact. A working vitual host setup needs to forward the envelope information of every message to the downstream, so that the recipient information can reliably be recovered, or maintain separate mailboxes for each user at the upstream (not normally a viable or desirable solution). Unfortunately, providers are only slowly realizing why this is necessary, and good information on how to do this right is apparently not easy to come by. If you are having problems with your upstream, persuade them to give you your money back if they can't solve this problem to your satisfaction.
Q: How can Procmail figure out who to Cc: when there are several recipients?
A: If you're asking this, you should probably be reading the answer to the previous question first. The rest of this is just a demonstration of how futile this is.
It has been asked a number of times if it's possible to implement a good forwarding scheme which can take care of things like Cc:s to different users in the same virtual domain.
Here's what you might have started out with:
:0 * ^TO_jill@mydomain\.virtual ! jill@real.address :0 * ^TO_jane@mydomain\.virtual ! jane@somewhere.else
The problem now is that if something comes in
To: both jane and jill,
(with infinite variations of Cc:s, Resent-To:, etc)
it will only be forwarded to jill. (Or, if you have
a :c flag on each recipe, the recipients will each
get one copy per local recipient of the same message.
Or, if your upstream already exploded the message into one copy
per recipient at your site, jill will get both her own copy and
jane's, or both will get two copies. Or, you mess up while trying
to fix this and your mail server goes into a headspin and you wake
up the next morning in Elbonia.)
The short answer is that Procmail is not the right tool for this.
If you really want this to work, you will also want
BCC:s to work
(see the BCC question above),
which is usually not possible without tweaking your Sendmail
configuration. In which case you might as well implement some or all
of the actual forwarding mechanism in Sendmail, which is the more
appropriate tool anyway.
If you are set on getting this to work anyway, see the "Kevorkian" answer below.
The basic problem is that when mail gets delivered into mailboxes at the upstream, the envelope information (like who the message is for) is usually discarded. So you shouldn't deliver into a mailbox that is not the correct mailbox, you should continue to transport the mail until you are ready to deliver to the final destination.
You can tweak Sendmail to copy the envelope recipient information into the message header (see below), but you don't have to do that if you don't deliver the message prematurely.
In a typical my-home-computer-is-my-domain setting, you could instead transport mail to your home computer using UUCP. fetchmail can also handle this for you, allowing you to have all mail for your domain delivered to a single POP account with the envelope information intact. (Look for "multidrop" in the fetchmail documentation. It appears that it actually relies more or less on the headers, not on the actual envelope recipient data. But if envelope information is available, fetchmail can use it.)
Q: Procmail is broken in this respect, isn't it?
A: No. If every problem begins to look like a nail, you need to have more tools than just the hammer in your toolbox.
Search backward for "stubborn" if you really really want to do this anyway.
Q: But I don't want to bug my upstream every time I create a new local address.
A:
You bug your upstream to set this up correctly for you, once.
(If they insist on having a list of your local users available
before they can deliver your mail correctly, it means they
want to be bugged. And you are a paying customer, correct?
Anyway, having a list of users at the upstream makes sense for
some things; if they want one, you should agree on a procedure
for updating this list which is as painless as possible for both parties.
man rdist and all that.)
A: Miquel van Smoorenburg has written a patch for this which can be found at ftp://ftp.cistron.nl/pub/people/miquels/sendmail/8.9.2-envto+etrnprog.dif -- allegedly this, or something like it, is being considered for distribution with Sendmail 8.10.
See also http://www.rosat.mpe-garching.mpg.de/mailing-lists/procmail/1996-11/msg00251.html which is a tangentially relevant message about this from Philip Guenther. (Or look at this local copy.)
Also check out the Sendmail FAQ, which discusses this at some length. And please visit http://www.sendmail.org/ if you haven't already.
In particular, there is an item in the Sendmail FAQ which explains
how to add the envelope information at the upstream using
(Procmail, naturally, and) the virtusertable:
http://www.sendmail.org/faq/section3.html#3.29
The Fetchmail FAQ has a Sendmail rule for adding Delivered-To: to all mail before delivering it. This has to be added to the upstream's Sendmail configuration, obviously.
Q: But I want to shoot myself in the foot. Won't you help me?
A: Download a copy of kevorkian.pl (never mind the Simpsons references). See the Cc: to many question above for details. (If you just want to look at the script, but your browser wants to download it and/or execute it, try the copy named kevorkian.txt instead.)
This is the kitchen sink.
A: You don't, you use formail to split it it back into separate messages and feed each into its own little procmail process.
formail -s procmail experiments.rc < test.mbox
(This will use the recipes in the rc file experiments.rc.)
You may want to add an -n switch if you have a large amount of messages to process, but you should be really, really sure you understand the implications before you try it. (In other words, RTFM.)
Q: How do I do different actions depending on the time of day, day of week, etc?
A:
The "brute force" answer is to run date(1) each time
you receive a message, and compare the results against some regular
expression, but a simple and efficient shortcut is to use the date
stamp which is already in the From_ line -- it will
be in your (machine's) local time zone and reasonably close to
(usually within seconds or even split-seconds of)
the message's arrival.
Here's an example which forwards messages elsewhere, but only on weekdays between 9 and 5. (Actually, until 16:59:59.)
:0 * ^From [^ ]+[ ]+(Mon|Tue|Wed|Thu|Fri) ... ... .. \ ([ 0]9|1[1-6]):[0-5][0-9]:[0-5][0-9] ! frankz@golden.net
:0 # Wed Aug 14 15:59:59 +0300 1994
* ^From [^ ]+[ ]+... \/... .. ..:..:.. [+-]?.... ....
{
FROM_=$MATCH
:0
* FROM_ ?? ^^\/[a-z][a-z][a-z]
{ MONTH=$MATCH }
# Exercise: Make sure month name is always properly capitalized,
# and/or convert it to a month number
:0
* FROM_ ?? ^^... \/([ 0][0-9]|1[0-2])
{ DAY=$MATCH }
:0 # Y10K problem! :-)
* FROM_ ?? ()\/[1-9][0-9][0-9][0-9]$$
{ YEAR=$MATCH }
# Should probably bail out here if any of MONTH DAY YEAR unset
:0: # save to daily-yyyy-mmm-dd
daily-$YEAR-$MONTH-$DAY
}
A: The canonical way to get the sender's trimmed-down address is
formail -rtzxTo:
This looks a bit counter-intuitive, but it works fine,
provided that you don't mind the fact that formail -rt will
prefer e.g. the value of Reply-To: over From:.
This will in fact do two things, which is not apparent if you're not too
familiar with formail. One: It will generate a reply header
(formail -rt). Two: Out of this new header, it will extract
the To: field (formail -zxTo:). By happy
coincidence, the generated reply address will only contain the actual
email terminus, without the full name or other comments. Wallah.
None of this is important if you merely want to find an address to
send off a reply to (all you need for that is formail -rt
and pass it to Sendmail). An address with a full name will do just
fine for replying; any decent mail program should accept that
(although you probably need to put it in double quotes to pass it as a
single argument). For keeping a database of addresses, though, the
stripped-down version is better (but still not perfect, and keep in
mind that the host.domain part is not case sensitive).
Typical newbie mistake: formail -rtzxFrom:;
this will actually return nothing at all, but you could in fact
expect it to return your own address.
If this approach doesn't suit you for some reason, you could always
post-process the address through some simple script which
successively strips off layers of comments and whitespace.
A fully RFC822-compliant script isn't trivial to write, but for
many purposes, a simple sed script will probably do.
(If you're truly masochistic, try to figure out how to get
sendmail to strip the address and hand it back
to you :-)
Perhaps you also want to look at formail -rD which keeps
record of a limited number of addresses and expires the oldest ones as
it gets fed new addresses.