Re: high cpu utilization with spamass-milter 0.2.0?

[Changeling]

--- Dan Nelson wrote:
> In the last episode (Jul 28), Changeling said:
> > Measuring over a 15 minute interval, using 0.1.3a
> spamd averages
> > about 0.96 seconds (max time was 3.7 seconds)
> while using 0.2.0 spamd
> > averages about 5.26 seconds (max time was 34.86). 
> Spamass-milter
> 
> You might see a slightly longer runtime because
> 0.2.0 generates a bunch
> of headers to mimic what sendmail would add,
> including a Received:
> header containing the originating IP number and
> date, and
> X-Envelope-From: header.  The IP number lets
> spamassassin do dnsbl
> checks, so that could affect processing speed.  Try
> timing the two
> versions after restarting spamd with the -L option
> (which disables DNS
> checks).
Here are the average run-times of spamd on my server
under different scenerios:
0.1.3a
Default config: 1.08 seconds
No Auto-Whitelist: 1.11 seconds
No DNS: 0.83 seconds
No Auto-Whitelist or DNS: 1.01 seconds

0.2.0
Default config: 3.27 seconds
No Auto-Whitelist: 1.10 seconds
No DNS: 3.67 seconds
No Auto-Whitelist or DNS: 0.85 seconds

These statistics make it APPEAR as if the problem is
0.2.0 and auto-whitelisting somehow.  However it's
entirely possible that the auto-whitelisting problem
is being caused by spamd taking slightly longer to run
which causes spamd to hold its locks on the
auto-whitelist file longer.  It's possible this is
causing a chain reaction where additional spamd
processes are being spawned which have to wait to lock
the file.  Each spamd process slows the system (and
therefore also slows the spamd process holding the
lock) and results in very little work getting done.

> 
> > doesn't seem to take any more or less time either
> way.  When stracing
> > I didn't see anything out of the ordinary except
> that it seemed that
> > when several emails came in in a row sometimes
> spamd would spin a
> > while trying to get a lock on the auto-whitelist
> file.  This caused
> > several spamd processes to queue up waiting to
> access the file.  I
> > turned off auto-whitelisting for spamd and now
> spamd runs as quickly
> > with 0.2.0 as it does with 0.1.3a.  However two
> things still concern
> > me: 1. It almost sounds like the problem is with
> SpamAssassin, but
> > why didnt this problem show up until I upgraded to
> 0.2.0 and why does
> > it immediately go away when downgrading to 0.1.3a?
>  2. Spamd with
> > auto-whitelisting running with 0.1.3a runs as
> quickly as spamd
> > without auto-whitelisting running with 0.2.0?
> >  It seems almost like spamass-milter is
> responsible somehow for the
> > consumption of the extra resources that are made
> available once
> > auto-whitelisting is turned off.
> 
> Do you know if auto-whitelist works based on header
> From, or envelope
> MAIL FROM?  If the latter, then with 0.1.3a
> spamassassinmay not have
> been whitelisting at all, because it didn't know the
> envelope sender.

I am fairly certain it uses the header FROM.  In any
event I know that auto-whitelisting has been working
as the database does grow in size and mail is matching
the auto-whitelisting rule in SpamAssassin.

Brian

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com