August 19th, 2007 by rick

Akismet vs Defensio Round 1

11 comments on 830 words

In one corner you have the heavyweight: Akismet. In the other corner, Defensio, the upcoming challenger. Who’s better at catching spam?

As most people know, you can’t have nice things on the internet. Spammers usually find ways to take advantage of open services, which is why they’re all walled behind email activated logins, captchas, etc.

I don’t really know why, but the spammers seemed to discover Lighthouse very quickly. Rather than building my own library and having to deal with the constant spam arms race, I’m a big fan of outsourcing the fighting to dedicated specialists. At the time, the only service I knew of was Akismet. They’ve worked out well for me in Mephisto (my open source blogging engine that powers this site, the Lighthouse homepage, and the Warehouse homepage), so I integrated it into Lighthouse.

It worked well for a few months, but after awhile it seemed like Akismet stopped working. I double checked my setup (even fixing one bug where I was reporting false negatives properly), but it seemed that the spammers had figured out how to fool Akismet. It was about this time that Mathieu Balez from Defensio contacted me about including it. He provided a beta account to test with Lighthouse, so for the last few weeks I’ve been testing it alongside Akismet. Though, a glorious mixup with the log file rotation meant I lost about a week’s worth of data. At any rate, here’s what I got for about a week of usage:

  • 282 tickets (Only tickets on public projects are checked)
  • 256 spam
  • 47 tickets where Akismet and Defensio did not agree.
  • Akismet missed 2 spams and falsely marked two tickets as spam.
  • Defensio missed 1 spam, and falsely marked 40 tickets as spam.

From this, you can see that the performance of the two engines is similar, except Defensio is more eager to mark items as spam. I was expecting more missed spam from Akismet actually, because there hasn’t been a single spam that I’ve seen since we added Defensio support.

How does Defensio differ? It sports a very similar API with a few added methods. First, it can scan articles as they’re posted. I’m assuming this provides an extra layer of protection by finding topics that may be off topic. They also provide a method to retrieve basic statistics. Though I think the most important additions are the inclusion of unique signatures and ‘spaminess’ scores. Unique signatures allow me to report spam/ham on previously checked comments without having to reconstruct the original request again. The spaminess value provides a convenient way to sort comments. Perhaps you want to see valid comments with the highest spaminess value, or spam with the lowest spaminess value to look for false reporting.

I’ve since vastly improved the way Lighthouse tracks spam, so it should be easier to find the false positives quicker. This small sample of results is helpful, but the true test of a spam catching service is how well it adapts to the spammers’ techniques. I’ll be keeping a closer eye on how well things are working. Please do leave some comments if you notice something odd.

By the way, here’s the Viking rails plugin I’m using in Lighthouse. It supports identical interfaces for both Akismet and Defensio (Akismet just ignores the calls it doesn’t support). Look for future support in Mephisto and Beast…

Discussion

  1. joost baaij joost baaij said on August 19th

    It’s a sad arms race, but a nessesary one unfortunately.

    A variety of techniques works best and you must change things every once in a while. The basic ‘did you pass math’ form field has become almost useless. I don’t know why. They must be parsing out the forms in a more or less intelligent way.

    My plugin (see my url) catches a fair amount of spam though. Works via dns-based blacklists. Especially the spamhaus blacklist is very reliable and has zero false positives.

    It’s a race in the truest sense of the word. Guess we’re the blue team then.

  2. Liz Liz said on August 20th

    +1 for defensio support in mephisto! Would love some kind of combo ( I’ve found Akismit to work pretty well)... ;)

  3. Daniel Haran Daniel Haran said on August 20th

    I saw their first public demo at Montreal’s Democamp. The plugin they showed had some very nice UI improvements to let you see results sorted by spamminess score – and see a lot of comments at once.

    Two optimizations come to mind: tweak the spamminess score threshold, and warn the admin when a certain number of questionable comments accumulate in the spam folder.

    The real competition shouldn’t be simply on marginal differences in effectiveness scores, but on the amount of time we need to spend tending our applications’ spam folders. In that respect, Defensio is proposing some interesting ideas.

  4. Matt Matt said on August 21st

    4 vs 41 mistakes is similar performance?

    Anyway I looked up the historical stats for your key in the Akismet system. Your key has analyzed 3784 comments, with you reporting 47 missed spam and 3 false postives, for an overall accuracy rate of 99.9868%, which is similar to the level you have in your smaller sample size above. Statistically, that is lower than most of our users though.

    I dug into your reported mistakes, of the three FPs two were valid and one was not actually a false positive, but a spam comment with a compliment in it, all of the spam were valid, but 32 of them appeared to be the same comment.

  5. Dries Dries said on August 21st

    Matt: you got your math wrong. If Akismet makes 50 mistakes out of 3784 messages, its efficiency is 98.6%, not 99.986%. The percentage error is computed as (mistakes / total) * 100. The efficiency is then 100 – ((mistakes / total) * 100). You did 100 – (50/3784), not 100-100*(50/3784) ...

    Also, for correctness, “efficiency” and “accuracy” are two different things …

    An efficiency of 98.6% is decent but not particularly exciting. That said, I agree that 4 mistakes is better than 41 mistakes. :)

  6. rick rick said on August 21st

    I also had a bug in Lighthouse where it wasn’t marking reporting back to Akismet when I marked users as spammers (which then marked all their ticket comments as spam). So far I’ve marked 327 ticket comments as spam.

  7. Matt Matt said on August 22nd

    Dries, you’re right! Thanks for the correction. That revised number definitely puts you under the average.

  8. Matt Matt said on August 22nd

    Dries, one question: why efficiency instead of accuracy?

  9. Dries Dries said on August 22nd

    Efficiency is a well-defined term. It is (spam + ham) / (spam + ham + false positives + false negatives) and that’s what you were trying to use.

    Accuracy is not a well-defined metric and can be computed in various ways: see http://wiki.apache.org/spamassassin/MeasuringAccuracy for a number of possible methods.

    For example, accuracy is often computed against the test/training corpus, not using the real data in the field.

    By using the term ‘efficiency’ people know what you are referring too. When using the term ‘accuracy’, you have to explain how you computed that value for people to be able to compare and interpret it.

    Ah, semantics! :)

  10. Carl Mercier Carl Mercier said on August 23rd

    We’ve been in touch with Rick and determined why the false-positive rate he experienced during this initial test was so high. With some changes to our back-end, and optimal usage of the API on his, the error rate should improve dramatically going forward. Part of the fun of being in private beta! ;)

  11. Jay Jay said on August 28th

    I also use Mephisto and was disappointed with Akismet. I’m in the process of integrating reCaptcha into my blog to prevent spammers. At least that way spammers will have to manually submit each piece of spam.

Sorry, comments are closed for this article.