Re: [Pan-users] Re: Global scoring with exclusions

[Dave]

On Thursday 14 June 2007, Duncan wrote:
> Dave <address@hidden> posted
> address@hidden, excerpted below, on  Thu, 14 Jun 2007
>
> 17:11:26 +0100:
> > I have a global score set up to mark trollish cross-posts, ie to three
> > or more newsgroups.  But I need to exclude some groups from this global
> > score.
> >
> > I've tried the following:
> >
> > %Score crossposts to 3+ NGs, exclude BY and VM groups
> > [~blueyonder\.announce*,~virginmedia\.announce*,*.*]
> > Score:: =-9999 %CrossPosts
> >         Newsgroups: (.*:){3} %Crossposted3PlusGroups Xref: (.*:){3}
> >         %Crossposted3PlusGroups
> >
> > %Score crossposts to 3+ NGs, exclude BY and VM groups
> > [*.*,~blueyonder\.announce*,~virginmedia\.announce*]
> > Score:: =-9999 %CrossPosts
> >         Newsgroups: (.*:){3} %Crossposted3PlusGroups
> >         Xref: (.*:){3} %Crossposted3PlusGroups
> >
> >
> > %Score crossposts to 3+ NGs, exclude BY and VM groups
> > [~blueyonder.announce*,~virginmedia.announce*,*.*]
> > Score:: =-9999 %CrossPosts
> >         Newsgroups: (.*:){3} %Crossposted3PlusGroups
> >         Xref: (.*:){3}  %Crossposted3PlusGroups
> >
> > %Score crossposts to 3+ NGs, exclude BY and VM groups
> > [~blueyonder\.announce*,~virginmedia\.announce*,*.*]
> > Score:: =-9999 %CrossPosts
> >         Newsgroups: (.*:){3} %Crossposted3PlusGroups
> >         Xref: (.*:){3} %Crossposted3PlusGroups
>
> You don't mention what pan version. 

Sorry, it's 0.127.

> Those section/newsgroup lines (all 
> but the third example) aren't going to do what you want in new-pan
> (>0.90), period.  That's because it's stricter slrn style than old-pan
> was, and the newsgroups lines are NOT regex based, only * wildcard.
> Therefore, those \. entries will match LITERAL backslash chars in the
> actual group name.

Yes, I tried with and without the \ escape, just in case :-)

> Here's the slrn scoring doc: http://www.slrn.org/docs/score.txt
>
> Now, it /does/ mention ~ in the section/newsgroup lines negating, so you
> got that part right.  However, the way it is worded implies you /cannot/
> mix negative and positive group matches.  The ~ must be the first char
> following the opening bracket and negates the entire section match.  You
> therefore choose positive or negative matching, not both.  If you choose
> negative, it's all /but/, so the *.* is already the positive match.  You
> don't specifically list it.  You only list what /not/ to match.  Further,
> ~ as the first char negates the entire entry, so it appears only there.
> Further appearances would again match literal ~ chars in the group names

Thanks, I missed that.  Further thinking on the subject suggests that I score 
all groups with 3+ crossposts at -9999 then do a fixed value score of 5000 
for staff posts in the internal groups.

> Therefore, try this:
>
> [~blueyonder.announce*,virginmedia.announce*]
>
>
> Now for your scoring match, the actual Newsgroups: header.
>
>         Newsgroups: (.*:){3} %Crossposted3PlusGroups
>
> OK, that's a regex, so the form is right, but the match isn't.  Why?

Actually, that part of it does work.  Posts to three or more groups are 
already being scored -9999.  It was the exceptions I had problems with.

> Because the newsgroups header doesn't use colon separators, it uses
> commas.  So try this instead:
>
>        Newsgroups: (.*,){3} %Crossposted3PlusGroups
>
> The next question is exactly how many groups did you want the xpost to
> match?  The comma only appears /between/ groups, not at the end, so if it
> appears three times, it'll match four groups, but not three, because that
> would be two commas, not three.  Also, your multiplier is /exactly/
> three, so as-is, it will match /exactly/ four groups, not five or more.
>
> You probably want this (two commas plus, no upper limit, so 3+ groups):
>
>        Newsgroups: (.*,){2,} %Crossposted3PlusGroups
>
> Which of course could also be written:
>
>        Newsgroups: ,.*, %Crossposted3PlusGroups
>
> Since the regex isn't anchored at either the left (^) or right ($), it
> matches anywhere in the line.  So, it should match as long as there are
> at least two commas, with any number of other characters (even zero,
> you'd use a plus instead of the asterisk if you wanted to match one or
> more) between them.
>
> However, note that the doc specifies precisely two types of comments,
> those on their own line begun with a percent char, and comments after
> scores, also begun with a percent char.  The doc does **NOT** state that
> ANY line may end in a comment (or put it this way, if it does, I missed
> it).  Therefore, depending on how strictly the documentation is followed,
> the lines above MAY be parsed as matching a LITERAL

Those comments don't currently affect the score rule, but since it's not 
explicity allowed I suppose that could change.  Point noted.

> "%Chrossposted3PlusGroups".  I don't believe that's what you want either,
> so that leaves us with one of the two entries below (put a comment on its
> own line above or below the matching line, if necessary):
>
>        Newsgroups: (.*,){2,}

That doesn't work.  Based on your logical explanation above, it should.

These lines do (seems to be a server/client thing dependant on the poster, 
need both lines to catch all cross-posts)

        Newsgroups: (.*:){3}
        Xref: (.*:){3}

I got these from a website somewhere but can't find it now.  

>        Newsgroups: ,.*,
>
> Of course, you'll need to adjust the Xref match line as well.  Without
> checking, however, I can't say for sure whether it uses colons or commas
> as separators.  You may have the colons part right, there, but would
> still need to dump the comment and adjust the frequency.
>
> See if you get any better luck with the above changes.  If not, maybe pan
> isn't quite following the documented format after all. =8^\

OK, some testing has revealed that setting an absolute score causes the 
scoring system to exit after matching the rule.  By setting the score on the 
crosspost rule to relative -9999 we can then carry on and set an absolute 
score in my ISPs internal groups on staff posting names only to 5000 causing 
it to exit at that point.  Other, following rules only get implemented if the 
cross-post rule failed *and* it's not a VM or BY internal group.

This should minimise the scoring overhead by limiting which groups the score 
rules apply to.

The final sub-set in question is:

%Score crossposts to 3+ groups to relative -9999
[*.*]
Score:: -9999 %Crosspost3plus
        Newsgroups: (.*:){3}
        Xref: (.*:){3}

%Score staff posts to internal groups to absolute 5000
[virginmedia.*,blueyonder.*]
Score:: =5000 %VMStaff
        From: fbutler@
        From: alex_brown@
        From: ian_hagon@
        From: Stewart\ Dunn
        From: address@hidden
        From: address@hidden
        From: address@hidden

This is succesfully scoring all crossposts to 3 or more groups at -9999 , 
scoring all VM and BY staff posts to 5000, including over-riding VM and BY 
staff posts which are cross-posted to to 3+ groups (mainly service 
announcements affecting multiple service, eg phone, tv and broadband)

There are also some global rules prior to this to mark known trolls and to 
mark both my posts and follow-ups to my posts as well as some groups specific 
rules afterwards.

It's all running smoothly now.  Thanks for the pointers.

All we need now is a way to automatically delete the -9999 posts instead of 
just hiding them :-)
-- 
Dave