Re: debbugs spamd SIGKILL ?

[Bob Proulx]

Jim Meyering wrote:
> Bob Proulx wrote:
> > Hmm...  SIGKILL seems like an explicit action, not a program bug.  But
> > I don't see anyone logged in for a while.  "last" shows only me just
> > now.  With others much longer ago beyond any reasonable horizon of
> > explicit human action today.
> 
> Could it have been due to a kernel OOM-killing?

Yes!  It does seem to have been the result of the OOM Killer.  Or at
least the OOM Killer was very active today around that time.

    $ grep -i oom-k /var/log/syslog | less
    Jul 18 10:03:19 debbugs kernel: [41161040.435711] soap.cgi invoked 
oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
    Jul 18 10:03:23 debbugs kernel: [41161042.388922] soap.cgi invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
    Jul 18 10:03:23 debbugs kernel: [41161044.060394] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:03:25 debbugs kernel: [41161046.856279] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:03:28 debbugs kernel: [41161050.095952] soap.cgi invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
    Jul 18 10:03:34 debbugs kernel: [41161055.156707] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:03:39 debbugs kernel: [41161060.443958] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:03:44 debbugs kernel: [41161065.788361] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:03:52 debbugs kernel: [41161073.758073] soap.cgi invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
    Jul 18 10:04:04 debbugs kernel: [41161085.412128] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:04:12 debbugs kernel: [41161093.554645] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:04:19 debbugs kernel: [41161100.099448] /usr/sbin/apach invoked 
oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
    Jul 18 10:04:25 debbugs kernel: [41161106.177247] soap.cgi invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
    Jul 18 10:04:32 debbugs kernel: [41161113.161835] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:04:38 debbugs kernel: [41161119.246910] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:04:44 debbugs kernel: [41161124.830000] soap.cgi invoked 
oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
    Jul 18 10:04:50 debbugs kernel: [41161130.939834] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:04:55 debbugs kernel: [41161136.876527] soap.cgi invoked 
oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
    Jul 18 10:05:02 debbugs kernel: [41161143.740558] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:05:09 debbugs kernel: [41161150.364160] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    Jul 18 10:05:16 debbugs kernel: [41161157.936640] soap.cgi invoked 
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0

I don't see something logging a kill for spamd but the times logged
there are correlating.  It looks to me like that is it.

> > I started spamd back up again.  And with spamd running mail to it is
> > flowing again.
> 
> Thanks for noticing and the quick response!

I am setting up a "monit" configuration to automatically restart spamd
if that is needed.  Then the response will be quicker!  And
automatic.  :-)

I am considering disabling Linux Memory Overcommit.  I never leave it
enabled on enterprise servers because it is mostly incompatible with
reliable operation.  But...  This is an old server and I try not to
"impose my will" upon these shared systems too much.  We all have to
get along there together.

Bob