lkml.org 
[lkml]   [2004]   [Jan]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [OT] Confirmation Spam Blocking was: List 'linux-dvb' closed to public posts
    From
    Date
    Linus Torvalds <torvalds@osdl.org> writes:

    > Especially if the "random words" in the spam end up being weighted by
    > real frequency, you just _cannot_ use single-word bayes filters on
    > it. Or if you do, you'll eventually have those words either being
    > neutral, or (worst of all cases) you'll have real mail be marked as spam
    > after having aggressively trained the filter for the spams.

    > It might not be that big of a deal especially if you have a fairly
    > narrow scope of emails in your ham-list, but people who get mail from
    > varied sources _will_ get screwed by this, one way or the other.

    After having put a couple thousand messages a day through bogofilter for
    around half a year now, this is, so far at least, not born out by my
    experience. Single word Bayesian filters are still working fine for me in
    practice and legitimate e-mail is not being misclassified as spam because
    of this sort of dictionary poisoning. All the misclassifications I've
    seen have been for very obvious reasons unrelated to Markov chains (I
    generally have to explicitly train bogofilter a few times on invoices and
    shipping notices from commerce sites, for example, since most
    commerce-related words occur with a very high frequency in spam), and it
    seems unlikely that they would be measurably helped by multiple-word
    Bayesian algorithms.

    Perhaps this will become a problem eventually (where eventually involves
    more than one hundred thousand messages), but if so, I've not yet seen any
    evidence of it.

    Maybe I just have that narrow scope of e-mail that you refer to. I'm not
    sure how to measure that. My gut instinct is that most people have a
    pretty narrow scope of e-mail that they receive, relative to all the
    possible legitimate e-mail messages (and I'm much more skeptical of
    Bayesian filters when applied site-wide rather than to a single mailbox).

    Using multiple words is probably better along some axes (faster training,
    perhaps), but a sufficiently trained single-word filter doesn't appear to
    have any real difficulties. I'm inclined to believe that people who are
    experiencing these sorts of problems with Bayesian filters are using
    inferior implementations, haven't sufficiently trained their filters, or
    have a radically different range of legitimate e-mail than I do.

    --
    Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 14:00    [W:0.025 / U:0.860 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site