VMware Virtual Appliances Community
VMTN_Admin
Enthusiast
Enthusiast

Spam Vigilante - Mail Filter Virtual Appliance

http://www.vmware.com/vmtn/appliances/directory/255

A mail proxy based on FreeBSD with spam (SpamAssassin) and virus (ClamAV) scanning. Can be used with any existing mail system.

0 Kudos
553 Replies
telackey
Contributor
Contributor

nshastings:

In case there was any trouble receiving my e-mail, basically my message was that what I would try next is:

1. Reboot the appliance and login

2. Run /etc/periodic/daily/474.m-postfix-accounts

3. Run /etc/periodic/daily/476.m-spamviewer

4. Login to the Spam Viewer as "vadmin" and check for the accounts.

0 Kudos
telackey
Contributor
Contributor

Update Notice (2006-07-16):

A new update has been released for Spam Vigilante. This is a feature update release.

The "View All" feature has been updated so that no e-mail addresses need entered before messages can be displayed. The search filter feature has been updated so that the "To" and "(B)CC" fields and the quarantine types are considered when filtering. This allows filters such as "banned" or "my_name@" to be used. The latter is particularly useful when a user has more than one e-mail address associated with their account. A new selective delete feature has been added, supplementing the previous "Delete All" link. This lets a user to delete one or more messages by selecting the checkbox for each one and then clicking "Delete."

Instructions are available at:

http://www.redbudcomputer.com/updates/spam/INSTALL.txt

Change log at:

http://www.redbudcomputer.com/updates/spam/CHANGES.txt

Download at:

http://www.redbudcomputer.com/downloads.htm

0 Kudos
lsadvanced
Contributor
Contributor

I've two request:

1) It is possible to have one image defined with SCSI disk to be used on VMware ESX?.

2) It is possible for vadmin see all spam/virus quarantine without define user and mailbox? (see all quarantine folder).

Thanks

Luca

0 Kudos
telackey
Contributor
Contributor

1) It is possible to have one image defined with SCSI disk to be used on VMware ESX?.[/B]

I don't know, but I'll look into it. My idea would be to create a new image with both a SCSI and an IDE disk, and use a ghosting utility to image the contents over. There would probably be additional configuration to change the FreeBSD settings to recognize and use the new disk, but I'd need to see what all would be entailed. When it was complete, I'd remove the IDE disk from the configuration.

2) It is possible for vadmin see all spam/virus quarantine without define user and mailbox? (see all quarantine folder).[/B]

Yes, this was just released in the 2006-07-16 update: http://www.redbudcomputer.com/downloads.htm

Thanks for your questions and interest!

0 Kudos
lsadvanced
Contributor
Contributor

Ok thank you,

I wait for the SCSI version, so I can use it with ESX.

Luca

0 Kudos
siliconjunkie
Contributor
Contributor

When I first came to review it, I was going to rate it 4 stars (reasons below). But after seeing how resposive the author is, it will get 5 stars from me.

My wishlist:

1) A way to change the config without re-running the setup. Preferably from the web interface.

2) Statistics both domain wide and per users. Maybe just the output of pflogsumm or something simple like that to start.

I know there are a couple others, but they escape me at the moment.

0 Kudos
telackey
Contributor
Contributor

siliconjunkie:

Thanks! I have opened tickets for both of your requests (actually, 2. was already open, but I've add an additional note).

I'll be shooting that a form of both of these will be in the next release of the full Vigilante appliance package. They probably won't be added through a patch because of the new software that will need installed.

Keep them coming, as I'm very open for suggestions on what should be added for the next release.

0 Kudos
jobber_jobber
Contributor
Contributor

Hi again,

I'm still making use of this appliance, as I think it's got great potential, and I'm really pleased the author takes on board the feedback he gets.

I've been testing this using a few domains I have, that aren't in real use, but sadly receive lots of spam mail.

Today's statistics (at almost 22:00 local time) are:

Reject: 3

SPAM: 229

Bad Header: 4

Passed: 509

Virus: 20

Banned: 0

Totals: 509 / 765 = 66.5359 % accepted

That means out of the 765 e-mail that Spam Vigilante has processed, it's passed on 509 of them to my mailbox, classifying them as "CLEAN" (see script I've used below).

I've not checked all 509 mails, but I'm pretty sure, all 509 were SPAM! Which means more SPAM is getting through than is blocked. Which is not really good enough for what I was expecting of Spamassassin.

Has anyone got any good ideas please on how I can improve the detection of SPAM and reduce the number of mails that get through to me, whilst not stopping any hams? (Currently I've used the default settings as provided with the appliance)

Hoping someone can help me tweak my config,

Jobber

\----


Script used:

cat /var/log/maillog | awk '$8 == "SPAM," \{ spam++ }

$8 == "CLEAN," \{ ham++ }

$8 == "BANNED" \{ ban++ }

$8 == "INFECTED" \{ virus++ }

$8 == "BAD-HEADER," \{ badh++ }

$7 == "reject:" \{ reject++ }

END {

print "Reject:",reject+0;

print "SPAM:",spam+0;

print "Bad Header:",badh+0;

print "Passed:",ham+0;

print "Virus:",virus+0;

print "Banned:",ban+0;

print "Totals:",ham,"/",ham+reject+spam+virus+ban+badh,"=",ham/(ham+reject+spam+virus+ban+badh)*100.0,"% accepted";

}'

The script came from the web somewhere, but can't find it again, so can't credit the author - if I find it - I will add another post. It has been amended slightly to include bad header mails and banned mails.

\----


0 Kudos
jobber_jobber
Contributor
Contributor

telackey,

Just one concern, about the next release:

You say:

I'll be shooting that a form of both of these will be

in the next release of the full Vigilante appliance

package. They probably won't be added through a

patch because of the new software that will need

installed.

Having spent time configuring my installation of the appliance, storing some scripts in directories, and building up a log/statistical history, it will be a lot of trouble if the next release (or any other future release) is a complete appliance replacement.

Would it be possible to find a way to make all future releases as some form of patch, or other type of install, which would retain any custom settings / files?

Thanks again,

Jobber

0 Kudos
telackey
Contributor
Contributor

jobber:

Yes, those stats seem rather low. Since the mail every domain receives is different, it is hard to draw exact conclusions, but using the script you provided I show an acceptance rate of just over 40%.

The first thing I would check is the spam score level picked during setup. Is it higher than the default of 6.31? A higher value can allow more spam through. It can be changed by stepping back through setup or by editing /usr/local/etc/amavisd.conf directly and restarting amavis.

The next thing I would ask is if you enabled the RulesDuJour during setup? If not, I would definitely recommend doing so.

If you have a reasonably low spam cutoff level (5-6.3) and have enabled the RulesDuJour, you might find that you need to increase your ruleset.

The best way is to edit the RulesDuJour configuration file (listed under Important Files in the docs, /etc/rulesdujour/config) and add more names to the TRUSTED_RULESETS. The appliance by default uses "TRIPWIRE SARE_EVILNUMBERS0 SARE_RANDOM SARE_ADULT". A complete list of available rules for RulesDuJour is available from the RulesDuJour site, http://www.exit0.us/index.php?pagename=RulesDuJour. There are a gooldy number of options that might increase your catch percentage.

0 Kudos
telackey
Contributor
Contributor

jobber asked:

Having spent time configuring my installation of the appliance, storing some scripts in directories, and building up a log/statistical history, it will be a lot of trouble if the next release (or any other future release) is a complete appliance replacement.

Would it be possible to find a way to make all future releases as some form of patch, or other type of install, which would retain any custom settings / files?[/B]

Well, the short answer is no, as not everything can be done as a patch, and also the number of patches can also grow beyond what can be maintained, as I am only one person.

The longer answer is that I will certainly do my best, however, not to do things that will needlessly require people to loose their customized settings.

In that vein, I hope you'll be pleased with the contents of my next post.

Message was edited by:

telackey

0 Kudos
telackey
Contributor
Contributor

Spam Vigilante Update Notice (2006-07-22):

I am pleased to announce the release of the first ever "Feature Pack" for the Spam Vigilante mail filter appliance. It is a statistics pack that introduces graphical and text statistics output available over the web. The statisics gathering software packages used are amavis-stats and pflogstats.

The Feature Pack contains modifications of the packages and automation that should completely automate the install process from ports. The process does take some time however, as many packages are downloaded and compiled to offer the new features.

The package is "stand-alone" from other updates released so far, and is totally optional.

For more information please see the README at http://www.redbudcomputer.com/updates/spam/fpacks/sv_stats_fpack_20060723/README.txt

To download, visit: http://www.redbudcomputer.com/downloads.htm

As always, feedback is appreciated.

\-------

Future:

Currently on the list is work to bring the appliance over to SCSI for the ESX users, and continued work to finish the release/injection function to the Spam Viewer.

\* EDIT *

Just a few minutes after first posting this I realized the spam statistics from pflogstats were not adequate, so I basically rewrote that section for this application. Those changes are in the package as of now.

0 Kudos
jtowne
Contributor
Contributor

What about a webgui for adding the domains you want to spam wash and the destination for that domain. I have 50 domains that are hosted on 5 different servers. I need to update/replace 3 mail hubs and this looks like a nice way of doing it.

0 Kudos
th-tron
Contributor
Contributor

Hello telackey,

thank you for this gooood stuff! It helps me to work around my mailserver troubles - he couldn´t fetch any pop mails cause of last update Smiley Sad !

With your appliance i can pipe it into them.

The cofiguration i made is via fetchmail and without exchange integration.

At the moment it fetches 2 mulitdrop account an put it in to a single user.

Is it possible to push it to the main smtp-server that it will be spread there to the users? At the moment i have to do it by hand ;-( !

How to configure this case?

Thank you for help!

0 Kudos
telackey
Contributor
Contributor

th-tron:

I am afraid my answer won't get you the whole way to the solution, but it should get you started.

There are two possibilities that I would look into directly. The first is that since you mention the server is not Exchange, assuming it is some UNIX-based server you may wish to:

1. Deliver to a mailbox on the server

2. Configure procmail rules for that user that redistribute the mail.

Both of those actions take place on the \_receiving_ mail server, not the appliance.

Note that procmail really does not recommend that it be used in this way, on the other hand, the "multiple users to one mailbox" problem is tricky one that doesn't always allow one a large number of options.

The second is similar, but would take place on the appliance. It would be to configure multidrop support for fetchmail. I think I might try this one first. Basically, instead of the standard (quotes taken from the fetchmail manual):

user "jsmith" with pass "secret1" is "smith"[/i]

You'd use:

Here’s what a simple retrieval configuration for a multi-drop mailbox looks like:

poll pop.provider.net:

user maildrop with pass secret1 to golux ’hurkle’=’happy’ snark here

This says that the mailbox of account ‘maildrop’ on the server is a multi-drop box, and that messages in it should be parsed for the server user names ‘golux’, ‘hurkle’, and ‘snark’. It further specifies that ‘golux’ and ‘snark’ have the same name on the client as on the server, but mail for server user ‘hurkle’ should be delivered to client user ‘happy’.[/i]

This shouldn't be too difficult to configure, however, it is serious to get right, so you'll definitely want to read the manual http://fetchmail.berlios.de/fetchmail-man.html . One section in the manual you'll want to examine especially is that if you ISP's server offers Envelope-To headers, you can specify their format with the "-E" option. You'd need to add this to the fetchmail startup at /usr/local/etc/rc.d/fetchmail.sh

I hope that gets you started!

0 Kudos
telackey
Contributor
Contributor

jtowne:

Yep, I'll add it to my list for the config UI.

In the meantime, it sounds like you'll need to edit the 'relay_domains' and the 'transport' files for postfix, located in /usr/local/etc/postfix. The first would list all the domains that you handled, and the second the mapping of the domain to the appropiate server for final delivery.

Just in case you aren't familiar with postfix, both will need postmap run on them after making changes before the changes would take effect.

0 Kudos
th-tron
Contributor
Contributor

thanks for help, you gave the right hint!!

Your inention was right. It is a Linux-server the mails should delivered and spread by it.

The link was very helpful and I modified the fechtmail.cf in the following way:

poll pop.provider.de

with protokoll pop3

localdomains maincompany.de company2.de

user maincomp with pass test1 to * here

user daughter with pass test2 to * here

it seams to work quite good. An X-envelope headder didn´t exist so I chose this way.

Thanks a lot!

0 Kudos
th-tron
Contributor
Contributor

oh sorry

I meant the following:

fetchtmail.cf

poll pop.provider.de

with protokoll pop3

localdomains maincompany.de company2.de:

user maincomp with pass test1 to * here

user daughter with pass test2 to * here

;

0 Kudos
robmitch
Contributor
Contributor

Hi there,

Great appliance, very happy with it, about to start playing with the feature pack. I seem to be having a problem with it though - it doesn't appear to be catching some spam! The particular spam I'm having trouble with appears to be bypassing the filter by using mime type "Content-Type: multipart/alternative; boundary="----_=_NextPart_xxx_" (where xxx is a number). This appears to be the only consistent thing I can see in the message headers. What I'm figuring is that postfix is being tricked into thinking it's part of a multipart message and just sending it through, rather than applying the spam filters. Seems a basic trick to fix, what have I missed?

Thanks,

R.

0 Kudos
telackey
Contributor
Contributor

robmitch:

That is interesting. I think it is unlikely though that it is not getting scanned; more likely (I think) is that it is not getting scored high enough to be quashed. Using as an example, I personally received a message today with:

Content-Type: multipart/alternative;

boundary="----=_NextPart_000_0001_01C6B10C.E5D82730"

It was let through, but it was also processed. Its score was:

X-Spam-Score: 4.578

X-Spam-Level: ****

X-Spam-Status: No, score=4.578 tagged_above=2 required=6.31

As you can see from the headers, it scored a 4.578, but I had a level of 6.31 set. Occasionally, some can get through without any X-Spam-* headers added; this usually means they were scored less than 2, the default score to start tagging.

I think it is worth confirming, however. Where I would start is by adjusting the sa_tag_level_deflt level in /usr/local/etc/amavisd.conf down to 0 and restarting amavis. Almost all mail will be tagged then, and you’ll be able to confirm if the mail is getting processed. If it is not, definitely feel free to contact me so I can help you figure it out.

The next steps involve helping SpamAssassin out, so it can catch more spam. The two best things to do are to train it, using sa-learn (man sa-learn), and to increase its rules base. You’ll want to train it on both good messages (ham) and the bad (spam) that got through. This can be tricky if your destination server is Windows-based/Exchange, but if it is not, you can probably just scp over some mailboxes to the appliance and run the appropriate sa-learn commands. The excellent sa-learn docs are available from Apache at: http://spamassassin.apache.org/full/3.1.x/dist/doc/sa-learn.html .

The other part can be some extra rules. In the post http://www.vmware.com/community/message.jspa?messageID=441356#441665 I’ve got some details about adding new rules for SpamAssassin. It is a balance between speed and effectiveness as one adds rules, because through one can perform more tests, it takes greater resources of processor and memory. Depending on the volume of mail that needs handled, a good number of rules can normally be used.

I hope this helps. I'd appreciate hearing how it goes.

0 Kudos