[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

From: Kenneth Loafman
Subject: Re: [Duplicity-talk] OSError: [Errno 24] Too many open files
Date: Wed, 13 May 2009 05:51:02 -0500
User-agent: Thunderbird (X11/20090318)

I run Hardy and Intrepid and both default to 4096.  When I installed
Jaunty, it was at 1024, and my guess is that the upstream Debian version
is the same.


address@hidden wrote:
> any idea why suddenly 3 people come around with this error? I remotely
> suspect a change in ulimit of distros OR duplicity v0.5.17
> ..ede
> -- 
>> No difference, other than my testing was flawed.  I finally had to go to
>> the debugger to stop the process and then count using lsof.
>> I'm showing 3 file descriptors released with every gpg process after the
>> waitpid() is completed, so it would take 300+ gpg zombies to overrun the
>> 1024 limit.
>> This is a problem in the way duplicity and GnuPGInterface are handling
>> process termination.  To fix it, I'll need to add a SIGCHILD signal
>> handler to GnuPGInterface that will allow the process to terminate when
>> it ends, rather than making it wait till we get around to checking.
>> ...Ken
>> Edgar Soldin wrote:
>>> just curious, could you detect a difference between 32bit & 64bit
>>> systems?
>>> how many incrementals did you need to hit the 1024 ulimit?
>>> @yang: how many incrementals does your backup hold? just to doublecheck
>>> kens findings.
>>> ..ede
>>>> I have found the problem and its going to take me a bit to get it
>>>> fixed.
>>>> As it works duplicity creates iterators that traverse each of the sig
>>>> files.  The iterators do not close until the process is complete and
>>>> the
>>>> files they are using are being held open.  The gpg process is held as
>>>> well since we don't wait for it until we close the file.  The zombie
>>>> processes were just a symptom of the real problem, too many open
>>>> resources at one time.
>>>> This hits directly the issue of too long an incremental chain.  Each
>>>> increments sig file is going to have file descriptors open until the
>>>> backup/incremental/restore completes.  The only workaround at this
>>>> point
>>>> is to increase the 'ulimit -n' parameter for any long increment chains
>>>> and perform a full backup on a more frequent basis.
>>>> I have some ideas about how to clean this up, but need to examine the
>>>> code a bit closer before committing to a solution.
>>>> ...Ken
>> ------------------------------------------------------------------------
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

Attachment: signature.asc
Description: OpenPGP digital signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]