Re: Milter (spamassassin): to error state : spamassassin milter seems to randomly fail

[Joost van Baal]

On Fri, Aug 15, 2003 at 11:59:37AM -0500, Dan Nelson wrote:
> In the last episode (Aug 15), Joost van Baal said:
> > On Fri, Aug 15, 2003 at 10:56:39AM -0500, Dan Nelson wrote:
> > > Actually, if you can build with the same flags on your RedHat or
> > > Solaris box and get me a stack trace, that might help.  My problem is
> > > that all my stacks look like this:
> > > 
> > > #0  0x0804b8ca in mlfi_envrcpt(smfi_str*, char**) (ctx=0x8067380,
> > >     envrcpt=0x805b080) at /usr/include/c++/3.3/bits/basic_string.h:249
> > > #1  0x2808b116 in mi_clr_macros () from /usr/lib/libmilter.so.2
> > > #2  0x2808a642 in mi_engine () from /usr/lib/libmilter.so.2
> > > 
> > > gdb collapsed the inline string operation on top of the code in
> > > mlfi_envrcpt, so I have no idea where in the function it died.
> > 
> > Tnx for your fast reply.  I can do some more investigations on
> > monday, will try to build with -O0 -fno-default-inline on the Debian
> > box then. (I assume I'll have to build spamass-milter with these
> > flags, and not sendmail's libmilter, do I?)
> 
> Right.  Hopefully if the mlfi_envrcpt() function (which lives in
> spamass-milter.cpp) gets built with no inlining, there will be an extra
> frame between #0 and #1 in that stack trace, pointing to
> spamass-milter.cpp.

OK, have build with these flags.  I am working with $Id:
spamass-milter.cpp,v 1.52 2003/06/26 15:10:44 dnelson Exp $ (in
spamass-milter-0.2.0, with some Debian patches).

In the spamass-milter.cpp CVS log I see:

 revision 1.61
 date: 2003/08/06 04:45:46;  author: dnelson;  state: Exp;  lines: +11 -4
 Fix two cases where we caught an error but didn't clean up the assassin
 object.

The revision 1.60 date: 2003/08/06 04:29:48; log looks interesting too.
Would it be useful if I started using 1.61 or later?

Anyway, my /usr/sbin/spamass-milter is an "ELF 32-bit LSB executable,
Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs),
not stripped".

I have

 address@hidden:~# ps axfww | grep '[s]pamass'
 32217 ?        S      0:00 /usr/sbin/spamass-milter -p 
/var/run/sendmail/spamass.sock
 32218 ?        S      0:00  \_ /usr/sbin/spamass-milter -p 
/var/run/sendmail/spamass.sock
 32219 ?        S      0:00      \_ /usr/sbin/spamass-milter -p 
/var/run/sendmail/spamass.sock

I can run gdb on the running daemon:

 address@hidden:~# gdb
 GNU gdb 2002-04-01-cvs
 Copyright 2002 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-linux".
 (gdb) attach 32217
 Attaching to process 32217
 Reading symbols from /usr/sbin/spamass-milter...(no debugging symbols 
found)...done.
 Reading symbols from /usr/lib/libstdc++-libc6.2-2.so.3...(no debugging symbols 
found)...done.
 Loaded symbols for /usr/lib/libstdc++-libc6.2-2.so.3
 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
 Loaded symbols for /lib/libm.so.6
 Reading symbols from /lib/libpthread.so.0...(no debugging symbols 
found)...done.
 [New Thread 1024 (LWP 32217)]
 [New Thread 2049 (LWP 32218)]
 [New Thread 1026 (LWP 32219)]
 Loaded symbols for /lib/libpthread.so.0
 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
 Loaded symbols for /lib/libc.so.6
 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
 Loaded symbols for /lib/ld-linux.so.2
 0x401607ee in select () from /lib/libc.so.6
 (gdb) info threads
   3 Thread 1026 (LWP 32219)  0x400bf87e in sigsuspend () from /lib/libc.so.6
   2 Thread 2049 (LWP 32218)  0x4015ebb0 in poll () from /lib/libc.so.6
   1 Thread 1024 (LWP 32217)  0x401607ee in select () from /lib/libc.so.6
 (gdb) continue
 Continuing.
 [New Thread 813059 (LWP 6216)]
 [New Thread 814084 (LWP 6225)]

 ^C

 Program received signal SIGINT, Interrupt.
 [Switching to Thread 1026 (LWP 32219)]
 0x400bf87e in sigsuspend () from /lib/libc.so.6
 (gdb) detach
 Detaching from program: /usr/sbin/spamass-milter, process 32217
 (gdb) quit

In another session, a backtrace looked like:

 (gdb) bt
 #0  0x400bf87e in sigsuspend () from /lib/libc.so.6
 #1  0x4008b2e7 in sigwait () from /lib/libpthread.so.0
 #2  0x08055707 in mi_signal_thread ()
 #3  0x400880ba in pthread_start_thread () from /lib/libpthread.so.0

You said you've been able to reproduce the behaviour.  How did you
manage to do that?

I can't, so I guess the only thing I can do now to obtain a useful stack
trace is hoping one of the Linux Threads dumps core.  I've restarted
spamass-milter, after setting 'ulimit -c unlimited' in the shell.
(spamass-milter runs as root on my machine, I expect a core dump in / .)

BTW, the stops in the spamass-milter functionality do not happen about
once a day, but about once every so many days.  (I've seen it at:

 Aug  5 09:48:01
 Aug  7 15:15:16
 Aug 14 12:30:07
 Aug 15 12:07:39

).

I'll give you more information once I've gotten hold of a core dump.

Bye,

Joost

-- 
                          Joost van Baal
address@hidden                                              S&N ITS
(013-466-)3519                                http://banach.uvt.nl/
kamer C 230A                                   aanwezig: ma, di, vr

pgpCzXiUVgwA8.pgp
Description: PGP signature