Re: New question about interaction between spamass-milter and clamav.

[Steve]

On Tue, 2009-07-07 at 21:43 -0400, Steven W. Orr wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 07/07/09 14:53, quoth EASY address@hidden:
> > On Tue, 2009-07-07 at 14:15 -0400, Steven W. Orr wrote: On 07/07/09 13:21, 
> > quoth EASY address@hidden:
> >>>> On Tue, 2009-07-07 at 13:06 -0400, Steven W. Orr wrote: On 07/07/09 
> >>>> 13:00, quoth EASY address@hidden:
> >>>>>>> On Tue, 2009-07-07 at 12:42 -0400, Steven W. Orr wrote: I need 
> >>>>>>> some help with whether I'm doing something wrong.
> >>>>>>> 
> >>>>>>> Here's my setup:
> >>>>>>> 
> >>>>>>> I have sendmail-8.14.3 spamass-milter-0.3.1-13 (set to reject) 
> >>>>>>> spamassassin-3.2.5
> >>>>>>> 
> >>>>>>> All was good in the world, but slowly the spam that got through
> >>>>>>>  was creeping up.
> >>>>>>> 
> >>>>>>> I found that if I added clamav-milter that a lot of the stuff 
> >>>>>>> would get caught. I came to this list a while back to ask about
> >>>>>>>  the pros and cons of whether the clamav-milter should go
> >>>>>>> before or after spamass-milter. I decided on doing
> >>>>>>> clamav-milter first. Both milters were set to reject.
> >>>>>>> 
> >>>>>>> Then the false negs started climbing again and I heard about 
> >>>>>>> all the clamav addons that were available via scamp from
> >>>>>>> 
> >>>>>>> https://sourceforge.net/projects/scamp/
> >>>>>>> 
> >>>>>>> That made a huge difference and life was good again.
> >>>>>>> 
> >>>>>>> Again, things were escalating and I decided that the real 
> >>>>>>> problem is that every time clamav rejects something, 
> >>>>>>> spamassassin doesn't learn from it. The SA AWL and the bayes 
> >>>>>>> tables never get informed except for the false negs that get 
> >>>>>>> through that get fed to sa-learn --spam. I thought that it 
> >>>>>>> would make more sense for SA to run the clam test itself. I 
> >>>>>>> looked around and sure enough I found clamav.pm  which is a 
> >>>>>>> plugin for SA.
> >>>>>>> 
> >>>>>>> So where I am now is that I have eliminated the clamav-milter, 
> >>>>>>> and I'm running a clamd so that clamscam works, and I installed
> >>>>>>>  the clamav plugin to SA. The sendmail incantation I'm using is
> >>>>>>>  this standard one:
> >>>>>>> 
> >>>>>>> INPUT_MAIL_FILTER(`spamassassin', 
> >>>>>>> `S=local:/var/run/spamass-milter/spamass-milter.sock, F=, 
> >>>>>>> T=C:15m;S:4m;R:4m;E:10m')dnl
> >>>>>>> 
> >>>>>>> and the params for running spamass-milter are
> >>>>>>> 
> >>>>>>> -m -u steveo -r 5 -d misc -i 192.168.0.1/24 -i 127.0.0.1/24
> >>>>>>> 
> >>>>>>> I can't believe you've read this far. But if you got here then 
> >>>>>>> I can now ask my question:
> >>>>>>> 
> >>>>>>> Why is it that I am now seeing  my server asking for retries? I
> >>>>>>>  am getting rejects like I expect to get (and like I got 
> >>>>>>> before). For example,
> >>>>>>> 
> >>>>>>> Jul  5 05:17:17 saturn spamass-milter[3478]: 
> >>>>>>> queueid=n659HG7c007407 Jul  5 05:17:18 saturn sendmail[7407]: 
> >>>>>>> n659HG7c007407: from=<address@hidden>, size=2174, class=0, 
> >>>>>>> nrcpts=1, msgid=<address@hidden>, 
> >>>>>>> proto=ESMTP, daemon=MTA, relay=sombody_good [good.guy.we.like] 
> >>>>>>> Jul  5 05:17:18 saturn sendmail[7407]: n659HG7c007407: Milter 
> >>>>>>> add: header: X-Virus-Scanned: ClamAV 0.94.2/9538/Fri Jul  3 
> >>>>>>> 10:27:11 2009 on myserver.syslang.net Jul  5 05:17:18 saturn 
> >>>>>>> sendmail[7407]: n659HG7c007407: Milter add: header: 
> >>>>>>> X-Virus-Status: Infected with 
> >>>>>>> Sanesecurity.Spam.10285.UNOFFICIAL Jul  5 05:17:19 saturn 
> >>>>>>> sendmail[7407]: n659HG7c007407: Milter: data, reject=554 5.7.1 
> >>>>>>> virus Sanesecurity.Spam.10285.UNOFFICIAL detected by ClamAV - 
> >>>>>>> http://www.clamav.net Jul  5 05:17:19 saturn sendmail[7407]: 
> >>>>>>> n659HG7c007407: to=<address@hidden>, delay=00:00:01, 
> >>>>>>> pri=32174, stat=virus Sanesecurity.Spam.10285.UNOFFICIAL 
> >>>>>>> detected by ClamAV - http://www.clamav.net
> >>>>>>> 
> >>>>>>> 
> >>>>>>> But now I'm also seeing retries when clamav is deciding that 
> >>>>>>> the status is unknown. Somehow, spamass-milter is returning a 
> >>>>>>> 4.5.1
> >>>>>>> 
> >>>>>>> Jul  5 04:55:56 saturn sendmail[19195]: n658tqMf019195: 
> >>>>>>> from=<address@hidden>, size=3879, class=0, nrcpts=1, 
> >>>>>>> msgid=<address@hidden>,
> >>>>>>>  proto=ESMTP, daemon=MTA, relay=201-94-178-5.jau.flash.tv.br 
> >>>>>>> [201.94.178.5]
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> Jul  5 04:55:56 saturn sendmail[19195]: n658tqMf019195: Milter 
> >>>>>>> add: header: X-Virus-Scanned: ClamAV 0.94.2/9538/Fri Jul  3 
> >>>>>>> 10:27:11 2009 on myserver.syslang.net
> >>>>>>> 
> >>>>>>> Jul  5 04:55:56 saturn sendmail[19195]: n658tqMf019195: Milter 
> >>>>>>> add: header: X-Virus-Status: Unknown
> >>>>>>> 
> >>>>>>> Jul  5 04:55:56 saturn sendmail[19195]: n658tqMf019195: Milter:
> >>>>>>>  data, reject=451 4.3.2 Please try again later
> >>>>>>> 
> >>>>>>> Jul  5 04:55:56 saturn sendmail[19195]: n658tqMf019195: 
> >>>>>>> to=<address@hidden>, delay=00:00:01, pri=33879, 
> >>>>>>> stat=Please try again later
> >>>>>>> 
> >>>>>>> I'm not sure that I have a complaint. It's just that I did not 
> >>>>>>> have any notion that spamass-milter had the ability to do 
> >>>>>>> anything but either accept or reject with a 500 series code and
> >>>>>>>  I wanted to understand what was going on. I figured that SA 
> >>>>>>> only returns an assessment, but it's spamass-milter that does 
> >>>>>>> the actual rejecting.
> >>>>>>> 
> >>>>>>> Can someone explain this? (And again, thanks for reading.)
> >>>>>>> 
> >>>>>>> 
> >>>>>>> It makes sense to virus scan first and drop. In an ideal world 
> >>>>>>> you should be taking out as much 'obvious' rubbish as early as 
> >>>>>>> you can in the SMTP stage. Only give Spamassassin anything that
> >>>>>>>  other anti UCE methods cannot latch on to. Spamassassin takes 
> >>>>>>> some tweaking. I won't use the Bayes feature at all - but that 
> >>>>>>> is a personal thing. You have to add rules that meet your 
> >>>>>>> requirements and see the kind of trends you have. Some examples
> >>>>>>>  of the false positives put up at : http://pastebin.com/ may
> >>>>>>> get you some suggestions.
> >>>> Ok, but you're actually not on the same subject as what I'm asking: 
> >>>> How is it possible for spamass-milter to returns a 400 series retry 
> >>>> request? I have it set to either reject if the score is greater than 
> >>>> 5 or accept.
> >>>> 
> >>>> I am NOT asking whether I should do virus checking first. I'm also 
> >>>> not asking for how to fine tune SA. I just don't see how these 
> >>>> retries are possible given the way it's configured.
> >>>> 
> >>>> 
> >>>> My apologies. I did not spot the 400 question. My inclination is this
> >>>>  will be specific to the MTA. It is the MTA that does the 4xx (defer)
> >>>>  or 5xx (reject). The milter simple says 'yes' or 'no'. The only 
> >>>> reason I could see a 451 and that would be if the milter was 
> >>>> unavailable (misconfigured or down). I've seen this with Postfix when
> >>>>  the milter is unreachable. (Had some issues with permissions on the 
> >>>> unix socket). My guess is sendmail has a similar mechanism. If you 
> >>>> are sure the milter is up and running and you can feed a test telnet
> >>>>  message all the way through, then somewhere your MTA is set to give
> >>>> a temporary fail rather than permanent.
> > Excellent! We're back on track. The milter is up and running. The spamd is
> >  running and clamd is running. What I see in the log files are 100% 
> > consistent:
> > 
> > If SA calls the CLAMAV plugin and it's clean then no score is added because
> >  of the plugin. But then I'm left with two possibilities. The first is that
> >  the milter rejects because the clamav plugin said that it is a virus. The 
> > second is like this:
> > 
> > Jul  5 09:37:03 saturn sendmail[17349]: n65Db1c4017349: Milter add: header:
> >  X-Virus-Status: Unknown Jul  5 09:37:03 saturn sendmail[17349]: 
> > n65Db1c4017349: Milter: data, reject=451 4.3.2 Please try again later
> > 
> > So someone is saying unknown. (That's the plugin) That unknown is then 
> > being reported back by spamass-milter with a 451 and then from there it's 
> > somehow turned into a "4.3.2 Please try again later".
> > 
> > It's not making sense to me, if for no other reason that the milter is 
> > supposed to only either accept or reject based on whether the score is 
> > greater than the threshold (or not).
> > 
> > Does this help?
> > 
> 
> > Simple question for ten points. If you disable the clamav in SA, and feed a
> >  message through does it work or are you getting the deferral?
> 
> > I'm not 100% at the order of play here, but my guess would be: clamav -> 
> > dns/network based tests -> spam scanning. Can you confirm without the 
> > clamav running the rest works flawlessly?
> 
> Yes.
> 
> BTW, clamav is not dns based. It's signature based.
Yes, I know that, having used it for a number of years on various
gateway devices :-)

My little diagram was a rough approximation of the process going on. Let
me expand;

INET MAIL --> FIREWALL --> [YOUR SERVER]
WHERE [YOUR SERVER] is processing mail like this;

[MTA]--> SA MILTER --> {CLAM} {SA DNS NETWORK TESTS} {SA SCANNING}

If it passes clam, SA will then perform network based tests using DNS
(if enabled).

This is all going off track. You need to confirm;
1. If you disable clam-av can you feed test messages through
2. How is your MTA talking to SA? PIPE, SOCKET OR INET?
An aside, is syslog or daemon log showing any complaints?