Spam filtering with Procmail

Barnaby W. Füi

Elite Member
Aug 14, 2001
12,343
0
0
Welp, it finally happened. Up until now, my home mail address was mostly untouched by spam. But just from today my inbox looks like this (garbled from fusetalk, hitting quote will show it correctly I think):

1332 N Sep 19 MS Internet Mail Delivery Serv Failure Announcement
1333 N Sep 19 chrisss63@wanadoo.fr Latest Net Security Upgrade
1334 N Sep 19 Microsoft Network Mail Service Report
1335 N Sep 19 Microsoft Corporation Network Newest Microsoft Upgrade
1336 N Sep 19 Net Email System Abort Advice
1337 N Sep 19 Microsoft Technical Assistance Newest Internet Critical Update
1338 N Sep 19 Microsoft Corporation Technica current microsoft critical pack
1339 N Sep 19 mailautomat@puremail.net
1340 N Sep 19 MS Corporation Program Securit Latest Internet Security Upgrade
1341 N Sep 19 kxrjnlekogcoi@technet.com Latest Upgrade
1342 N Sep 19 Net Service
1343 N Sep 19 MS Technical Support Latest Network Upgrade
1344 N Sep 19 MS Corporation Network Securit Latest Net Critical Pack
1345 N Sep 19 Inet Message Storage System Error Announcement
1346 N Sep 19 microsoft net delivery service Failure Letter
1347 N Sep 19 MS Technical Bulletin Network Critical Update
1348 N Sep 19 Network Delivery System bug report
1349 N Sep 19 MS Corporation Security Depart
1350 N Sep 19 message service Bug Report
1351 N Sep 19 MS Corporation Internet Securi Latest Net Security Patch
1352 N Sep 19 Microsoft Last Critical Update
1353 N Sep 19 mivie@wanadoo.fr Error Letter
1354 N Sep 19 Microsoft Network Message Serv Failure Announcement
1355 N Sep 19 MS Security Section Newest Internet Pack
1356 N Sep 19 marni1@libero.it New Internet Upgrade
1357 N Sep 19 Inet Mail Storage Service error letter
1358 N Sep 19 Network Mail System returned mail
1359 N Sep 19 MS Program Security Department Newest Security Patch
1360 N Sep 19 Inet Delivery Service Notice
1361 N Sep 19 postrobot@yahoo.com abort message
1362 N Sep 19 Microsoft Corporation Program Last Net Security Patch
1363 N Sep 19 Email Storage System Mail Returned To Mailer
1364 N Sep 19 MS Corporation Security Depart Latest Internet Patch
1365 N Sep 19 Internet Message Service
1366 N Sep 19 Microsoft
1367 N Sep 19 ognupiqnqh@urreuffh.com Newest Internet Pack
1368 N Sep 19 Microsoft Inet Mail System
1369 N Sep 19 Microsoft Security Support Latest Network Pack
1370 N Sep 19 MS Corporation Public Assistan Newest Net Critical Upgrade
1371 N Sep 19 Net Message Storage Service Message User unknown
1372 N Sep 19 Microsoft Email Service Failure Announcement
1373 N Sep 19 Internet Email Service Bug Letter
1374 N Sep 19 Microsoft Security Assistance Current Critical Patch
1375 N Sep 19 MS Inet Mail Storage Service Error Letter
1376 N Sep 19 Microsoft Corporation Security New Net Critical Pack
1377 N Sep 19 MS Mail Delivery Service Bug Report
1378 N Sep 19 MS Customer Services Newest Patch
1379 N Sep 19 Microsoft Corporation Technica
1380 N Sep 19 Microsoft Corporation Customer New Network Security Upgrade
1381 N Sep 19 Admin Mail: Returned To Sender

:disgust:

Anyways, I'm already using Procmail for mailing list organization, is there any "best" spam filtering stuff? Spam Assassin is the only one that comes to mind, I guess that's pretty popular. Anyone use it?
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
I use spam assassin and I love it. its rare when i get any spam that gets through, but it did take a while to get it to that point. I'd say out of the average of 200 spams a week maybe 3 or 4 get through.

i also get a couple false negatives from legit places like my credit card comp etc that insist on using bad html code...I add them to a white list.

edit: spelling...brain faster than hands
 

Barnaby W. Füi

Elite Member
Aug 14, 2001
12,343
0
0
Originally posted by: Need4Speed
I use spam assassin and I love it. its rare when i get any spam that gets through, but it did take a while to get it to that point. I'd say out of the average of 200 spams a week maybe 3 or 4 get through.

i also get a couple false negatives from legit places like my credit card comp etc that insist on using bad html code...I add them to a white list.

edit: spelling...brain faster than hands

How do you deal with false negatives? Having to look through the ones detected as spam doesn't seem much better than just leaving it as is. Then again, it would be in a different mailbox, so at least it wouldn't crowd my real mail..
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
i also get a couple false negatives from legit places like my credit card comp etc that insist on using bad html code...I add them to a white list.

that should have said false positives.


its not really that bad. for the first few weeks you feed it all kinds of mail as ham (good mail) and spam to help the baysien (spelling, too lazy to look it up) filter learn whats what. After that if you still get some false positives, I use a white list. and to be honest, there are hardly any addys in there. only my credit card co and a couple of legit mailing lists I am on. the rest gets filtered pretty well.

As I said, I only get 3 or 4 false negs a week, so they are pretty easy to spot and only takes a sec to weed out. Certainly much more efficient that having to do it all by hand.
 

Barnaby W. Füi

Elite Member
Aug 14, 2001
12,343
0
0
Originally posted by: Need4Speed
i also get a couple false negatives from legit places like my credit card comp etc that insist on using bad html code...I add them to a white list.

that should have said false positives.


its not really that bad. for the first few weeks you feed it all kinds of mail as ham (good mail) and spam to help the baysien (spelling, too lazy to look it up) filter learn whats what. After that if you still get some false positives, I use a white list. and to be honest, there are hardly any addys in there. only my credit card co and a couple of legit mailing lists I am on. the rest gets filtered pretty well.

As I said, I only get 3 or 4 false negs a week, so they are pretty easy to spot and only takes a sec to weed out. Certainly much more efficient that having to do it all by hand.

Ok, cool. How many mails should you feed it? I imagine I probably have ~2000 or something like that. edit: nm, I'm just gonna start reading the docs, I'm sure they answer this stuff all the time :p
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
I basically fed it emails on a weekly basis for a couple months or so, only takes a couple min a week to do that. if yo have a huge stash of emails already, then your time to train the filter will be much less than mine...obviously thats a good thing.

here are my default settings that are in my /etc/mail/spamassassin/local.cf
# This is the right place to customize your installation of SpamAssassin.
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
###########################################################################
#

# SpamAssassin config file for version 2.5x
# generated by http://www.yrex.com/spam/spamconfig.php (version 1.01)

# How many hits before a message is considered spam.
required_hits 4.0

# Whether to change the subject of suspected spam
rewrite_subject 1

# Text to prepend to subject if rewrite_subject is used
subject_tag *****SPAM*****

# Encapsulate spam in an attachment
report_safe 1

# Use terse version of the spam report
use_terse_report 0

# Enable the Bayes system
use_bayes 1

# Enable Bayes auto-learning
auto_learn 1

# Enable or disable network checks
skip_rbl_checks 0
use_razor2 1
use_dcc 1
use_pyzor 1

# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_languages all

# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_locales

I have mine setup to filter all mail coming in, not just certain users. my $HOME/.spamassassin/user_prefs I left at the default settings.

Hope it works as well for you as it did for me....


 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
Originally posted by: BingBongWongFooey
Cool, thanks. pyzor looks pretty cool, written in Python no less :D


I'll have to take your word on that :) not too familiar with python...
 

Barnaby W. Füi

Elite Member
Aug 14, 2001
12,343
0
0
Hm, someone mentioned this on python-list:

A new mass mailer worm hit the net at about 9:00 PM EST last night
(at least, that's when I started getting the spam from it.) It seems to be
using a really old hole in Windows, meaning that if you've kept up to date
with your patches, you should be safe from infection. At least, that's
the information I have to date. I put my "junk" folder on auto delete
to protect my mailbox, and that's it.

A lot of other people mentioned they were getting mountains of fake MS mail too.
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
ive heard that as well around the office the last couple days....i generally shy away from anything that doesnt come from someone I know...or that I didnt ask for.
 

Barnaby W. Füi

Elite Member
Aug 14, 2001
12,343
0
0
Gah... finally got it working. I got held up for a long time on an unexplainable problem.

I have postfix -> procmail -> spamassassin. I had set up my .procmailrc to pipe through spamassassin, but it wasn't doing it. BUT, when I switched the spamassassin command to sed, the sed worked and indeed modified the messages. When I ran procmail by hand and pasted a mail into it, it would show up in my mailbox with spamassassin stuff in it. So basically, mail coming from postfix -> procmail did not run spamassassin correctly, for whatever reason. So I looked in main.cf and I was calling procmail like this:

mailbox_command = /usr/pkg/bin/procmail -t -m /usr/pkg/etc/procmailrc -m $HOME/.procmailrc

However I realized that when I was testing on the command line, I called it with no args at all, and it still went to my mailbox, and read my .procmailrc and the whole deal. So I removed that stuff and just kept "/usr/pkg/bin/procmail -t" and it WORKED! Thank god, because that was really baffling me for a while. I guess spamassassin wasn't liking some environment variables or something. Honestly I don't even care, as long as it works now. :)

Anyways I think this is pretty cool. I have postfix checking some pretty leniant blacklists, and some other checks:

maps_rbl_domains =
cbl.abuseat.org,
blackholes.easynet.nl,
sbl.spamhaus.org,
opm.blitzed.org,
relays.ordb.org

smtpd_recipient_restrictions =
permit_mynetworks,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
reject_unauth_pipelining,
reject_non_fqdn_hostname,
reject_invalid_hostname,
reject_maps_rbl,
check_relay_domains

Then it goes to procmail. Here is the spamassassin-relevant parts of my .procmailrc:

:0fw: spamassassin.lock
| spamassassin

:0:
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
spam.mbox

:0:
* ^X-Spam-Status: Yes
maybespam.mbox

And then I use mutt for reading mail, and vim to edit. I have a few things in my .vimrc to go along with this:

com Spam !sa-learn --spam --single --no-rebuild <%
com Ham !sa-learn --ham --single --no-rebuild <%
com Despam %!spamassassin -d

The first one lets me tell sa-learn that a message is spam by doing :Spam. The second is the opposite. The third one is for false negatives; If I decide they aren't spam, I can just run :Despam and it'll remove all of the spamassassin markup from the message.

Also, with the recent flood of MS-related crap, this helps in spamassassin user_prefs file:

score MICROSOFT_EXECUTABLE 4.0

Tweak 4.0 as you see fit, of course.
 

Need4Speed

Diamond Member
Dec 27, 1999
5,383
0
0
glad you got it working! we basically have the same setup so I will try some of your shortcuts as well.
thanks