monitoring individual httpd processes for high memory/cpu

[Jonathan Swartz]

I'm not sure if this is an appropriate use case for monit.

We use Apache/mod_perl 2 and occasionally get a child httpd process that spins out of control, either consuming ever-increasing amounts of memory or max cpu. Usually due to an infinite loop or other bug in a specific part of the site - this sort of thing happens.

I would like to monitor for such httpd children every second or so, and when finding one, send it a particular signal (USR2) which will tell it to dump its Perl stack to our error logs.

(Note that we are using rlimit/BSD::Resource, but when a process exceeds its memory rlimit, it dies immediately with no good way to log or take action. See http://stackoverflow.com/questions/6348158.)

monit comes highly recommended, but in perusing the docs and examples, it looks like it is more about monitoring and managing entire services rather than individual processes. Is that the case, or is there something I'm missing that allows individual httpd children to be monitored and signalled?

Below is my current solution using a Perl script that reads from "top -b" for efficiency. Obviously I'd much prefer to use a solid tool than reinvent my own bug-prone wheel.

Thanks for any advice.

Jon

---------

# Read process info from top -b every $frequency seconds, for

# processes owned by current user

#

my $cmd = sprintf( '/usr/bin/top -b -u %d -d %d |', $<, $frequency );

open( my $topfh, $cmd );

my $count = 0;

while ( my $line = <$topfh> ) {

    chomp($line);

    next if $line !~ /^\s*\d+.* httpd\s*$/m;

    # Get memory usage in $vsize

    #

    my ( $pid, $virt ) = ( split( / +/, $line ) )[ 0, 4 ];

    my ( $vsize, $unit ) = ( $virt =~ /(\d+)([mg])/ );

    next unless $unit;

    $vsize *= 1024 if lc($unit) eq 'm';

    $vsize *= 1024 * 1024 if lc($unit) eq 'g';

    if ($vsize > $max_vsize) {

        my $msg = sprintf( "pid %d vsize %d > %d, sending USR2",

                           $pid, $vsize, $max_vsize );

        $log->warn($msg);

        kill( 'USR2', $pid );

    }

}