[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Spam through the newsgroup gateway
Re: Spam through the newsgroup gateway
Thu, 15 Nov 2018 19:38:19 -0800 (PST)
On Sunday, November 11, 2018 at 3:53:08 AM UTC+5:30, Bob Proulx wrote:
> Alexandre Garreau wrote:
> > Bob Proulx wrote:
> > > Public Service Announcement: Please do not reply to spam. If a valid
> > > message is in reply to a spam message then it refers to it and in a
> > > sense validates it. To talk about spam please use an independent
> > > thread so as not to validate the original spam.
> > Why so?
> The best anti-spam engines in practice are learning engines such as
> Bayes and other. Spam characteristics change so quickly and their
> human senders keep trying to be more sneaky than before. We use no
> fewer than three! SpamAssassin, Bogofilter, and CRM114. By far
> CRM114 is the best of those three. But there are subtle differences
> that keep me playing one off the other and therefore continuing to add
> engines rather than remove them.
> Since they are learning engines they must be trained in order to
> learn. The best training has been training on error. When the
> classification is different it must be corrected.
> All messages are fed through the anti-spam classification engines
> twice. Once on the frontend in order to classify the message to
> determine if it should be automatically discarded. And then once
> again after the messages go through the mailing list to train on any
> errors. Since the mailing lists are relatively spam free (IMNHO) then
> I assume that any message through the mailing list is a desired
> message. If any of the learning engines think otherwise then it
> triggers training to learn that message as non-spam.
> SpamAssassin knows the structure of email, what's a header and what is
> the body. Bogofilter and CRM114 have no knowledge of email structure
> and process the message as a raw file looking at tokens in the headers
> and structure and learning them as either indicators or not
> dynamically. For them this includes IP addresses and email addresses
> and everything. Everything is open to gripping upon.
> Just recently, due to our conversations about the newsgroup gateway
> here, I have modified this algorithm slightly. I now look for the
> newsgroup gateway header. If a message entered through the newsgroup
> then I ignore it. There isn't anything I can do about it. Training
> on it makes no sense. Therefore I ignore it. No training. But until
> recently I did train on newsgroup messages too.
> If someone replies to the message then the email headers and the
> structure of it and, goodness forbid if they quote any of the message
> (top posting on the entire spam is worst), then all of that may have
> been associated with spam but when it comes through the mailing list
> now it will be associated with non-spam. Training the learning
> engines on it will pull the database to thinking that that type of
> message, spam though it is, is desirable on the mailing list and will
> pass it through in the future. It will eventually correct but may
> take a while. A while being around a month for the size of the token
> database we keep. From week to week the trend in spam changes.
> > If not sending anything to whoever sent the mail, will they
> > track the mailing-list or its archive to find some other mail referring
> > to it, and take this as an encouragement and post more spam?
> Not likely. I think for spammers it is mostly send and forget (like a
> "fire and forget" military missile).
> > Otherwise, what's the problem of validation if it's for a single spam?
> > Let's say someone got their antispam block that spam: it seems to me
> > normal, whenever a discussion is being about some spam that has been
> > relayed by the list, that the user either see the aforementioned spam,
> > to aknowledge the problem other are living (and get a sample of it), or
> > not to see the thread at all, as they're not concerned.
> If it is a single spam it isn't the end of the world. It is all just
> incremental. Because it will be used to train the learning engines.
> And they will recover given enough time and good later input. But
> every little bit counts!
> > Ideally there should be a way to trigger metadata so that when you
> > answer to something you do while marking it as spam for people seeing
> > your message, like a mail header for it.
> There are systems in use where the community can vote upon messages.
> They usually require multiple votes, say five, from known quality
> voters, and then the message is hidden. But mostly we see those with
> web page forums. Since this is a mailing list in order to install
> such a thing we would need to have users trained on how to do this.
> As another data point in this area the Debian mailing lists have an
> address where people can "bounce" the spam to for further training of
> their anti-spam learning engines. And as a notification to the
> listmaster that spam is flowing in and needs help to be blocked (they
> use procmail rules, we do too) if they get a new type that slips
> through. (Mutt has a 'b'ounce mail action, other mailers may or may
> not.) We could set up something like that but one does not exist at
> the moment. With some more work it could be useful if people were to
> contribute spams that slip through into the mailing list to it.
> Sorry for the long delay in answering this message. Life and time is
> what keeps everything from happening all at once.
You seem to be managing a splendid job with ML-news gateway spam
[ Compare https://groups.google.com/forum/#!forum/comp.lang.python ]
Wonder how easy it would be for you to share your know-how in capsule/summary??
(assuming the folks managing comp.lang.python are interested]