[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

From: edgar . soldin
Subject: Re: [Duplicity-talk] OSError: [Errno 24] Too many open files
Date: Tue, 12 May 2009 16:13:25 +0200
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20090302 Thunderbird/ Mnenhy/

Yang seems to use a recent Ubuntu ..

Yang are you reading? Which gpg version do you use? Which python version are you running?

.. ede
Thanks for the help!

I'm not sure at this point.  I built a 64bit Ubuntu 9.04 system and it
seems to be working fine.  Debian may be different, but not much.

As to what's broke, it could even be at the system level, but my bet is
on gpg.


address@hidden wrote:
I am gonna install a 64bit debian in a virtual machine and gonna
doublecheck that ... but what do you think is buggy? python, gpg ?

The only real change to GnuPGInterface is a shift right instead of a
shift left when printing the error code after an exception.  He's not
getting any exceptions, so he's not hitting the change.  Other than
that, its the same code from years ago.

My best guess is still that its a 32-bit versus 64-bit issue.


Edgar Soldin wrote:
true they close every time .. the only thing i see is that the amount of
gpg processes grows with the amount of incrementals ...

@yang: how many incrementals does your  backup hold?

the other thing is, maybe gpg itself is buggy  ..

@yang: could you try another gpg version?

and the third thing

@ken: is it really impossible for 0.5.17 to use an oldGnuPGInterface
version, that might be laying around on the system? How could yang look
for it?

.. 2cents again :) ..ede
I tried this with tiny volume sizes of 1M over a 2GB backup (2K
and did a count of the gpg processes (live, defunct) every 2 seconds
never had more than 5 (3 live, 2 defunct).  This was on a very fast
between local VMs.  Tried both backup and restore so it would test both
directions.  By the end, all were closed.

It's possible to have some defunct while the task scheduler is
but Yang is showing hundreds that are holding files open.  That's the
confusing part.


Edgar Soldin wrote:
i really have to be fast (a window of 10s let's say) and it helps to
have a slow backup connection with lots of incrementals... but I
may be
seeing ghosts here. Can it that it is perfectly normal for
duplicity to
gpg decrypt something per each incremental but close all processes
a bit later? .. just a guess
I cannot reproduce this on a 32bit machine at all.  I'll be able
to test
on a 64bit machine sometime this week and, hopefully, find the


address@hidden wrote:
i seem to get closer to it ...
the more incrementals (regardless if file: or ftp: backend) the more
defunct gpg processes appear .. but dissappear again while the
backup is
still running

I did
initial backup of 3GB
touched a new test file to have a change & did a 2nd backup
touched a new test2 file to have a change & did a 3nd backup

and since the 3rd backup the defunct gpg processes keep appearing

essentially I say: Take any more than 3 incrementals in size
backup and
do a new incremental and you will see those zombie gpg processes ..

while I am not sure that this is what happens on the ulimit
problem I
thought I should report these observations ... ede


here a 'ps xf -u user' shortly after starting the sixth run .. the
number of gpg zombies seems to be number of sets (full + incr's)
minus 2 ..

9122 pts/0    S+     0:00          \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity test backup
19128 pts/0    S+     0:00              \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity backup
19193 pts/0    S+     0:00                  \_ /bin/bash
/srv/www/user//release/ftplicity_1.4.2/ftplicity backup
19194 pts/0    R+     0:03                      \_ /usr/bin/python
/srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
--encrypt-key 00000000--sign-key
--volsize 1 --exclude-globbing-filelist
19196 pts/0    SL+    0:00                          \_ gpg
--logger-fd 4
--passphrase-fd 8 --batch --no-tty --default-key 00000000--recipient
00000000--no-secmem-warning --always-trust --encrypt --sign
19197 pts/0    SL+    0:00                          \_ gpg
--status-fd 6
--passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
00000000--no-secmem-warning --always-trust --decrypt
19199 pts/0    Z+     0:00                          \_ [gpg]
19200 pts/0    Z+     0:00                          \_ [gpg]
19201 pts/0    Z+     0:00                          \_ [gpg]
19202 pts/0    Z+     0:00                          \_ [gpg]
19203 pts/0    SL+    0:00                          \_ gpg
23 --passphrase-fd 28 --batch --no-tty --default-key
00000000--no-secmem-warning --always-trust --encrypt --sign

hmm .. you are right these processes are zombie (they appear in
top as
such) .. later on the processes disappear, while duplicity is still
running ..
but I don't feel this is normal.. or is it?
just to be more sure, I'll try a 1MB chunk full backup of my 3GB
dir .. let's see with how many zombies I'll end up ...

interestingly my ulimit = unlimited ... seems to be default in
the old
SUSE 10.2, btw also on my debian 5.0 box both 32bit
can't remember to ever have touched this setting

These are not open processes, they are zombie processes, so no
resources taken.  That simplifies it a bit.  What normally
causes this
is the failure of a parent process to properly retrieve its
exit status,
but this is not the case or more people would have this problem.

As far as I can tell, the only systems this is happening on are
and not even all of those.  Some of my systems are 64bit and
they don't
show this, so I'm wondering if this could be limited to a
version of GnuPG, or what.

Edgar, from your note I could not tell, is this happening to
you too?


address@hidden wrote:
just my quick observation .. a simple incr backup opens and
leaves over
20 gpg processes open. I don't think this is healthy and also can
imagine that this multiplies if the volumes get smaller (mine are

regards ede

address@hidden:~> ps -u user xf
32731 ?        S      0:00 sshd: address@hidden/0
32732 pts/0    Ss     0:00  \_ -bash
1167 pts/0    S      0:00      \_ /bin/bash
profile backup
1168 pts/0    S      0:00      |   \_ /bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity profile backup
1174 pts/0    S      0:00      |       \_ /bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity backup
1239 pts/0    S      0:00      |           \_ /bin/bash
/srv/www//release/ftplicity_1.4.2/ftplicity backup
1240 pts/0    S      0:05      |               \_ /usr/bin/python
/srv/www/user/_apps/duplicity-0.5.10/bin/duplicity --verbosity 4
--encrypt-key XXXXXXX --sign-key B59ECD99
--full-if-older-than 1M --volsize 50 --exclude-globbing-filelist
1249 pts/0    SL     0:00      |                   \_ gpg
--logger-fd 4
--passphrase-fd 8 --batch --no-tty --default-key XXXXXXXX
XXXXXXXX --no-secmem-warning --always-trust --encrypt --sign
1265 pts/0    SL     0:00      |                   \_ gpg
--status-fd 6
--passphrase-fd 11 --logger-fd 5 --batch --no-tty --default-key
--no-secmem-warning --always-trust --decrypt
1267 pts/0    Z      0:00      |                   \_ [gpg]
1272 pts/0    Z      0:00      |                   \_ [gpg]
1275 pts/0    Z      0:00      |                   \_ [gpg]
1278 pts/0    Z      0:00      |                   \_ [gpg]
1282 pts/0    Z      0:00      |                   \_ [gpg]
1286 pts/0    Z      0:00      |                   \_ [gpg]
1289 pts/0    Z      0:00      |                   \_ [gpg]
1292 pts/0    Z      0:00      |                   \_ [gpg]
1296 pts/0    Z      0:00      |                   \_ [gpg]
1299 pts/0    Z      0:00      |                   \_ [gpg]
1302 pts/0    Z      0:00      |                   \_ [gpg]
1305 pts/0    Z      0:00      |                   \_ [gpg]
1308 pts/0    Z      0:00      |                   \_ [gpg]
1311 pts/0    Z      0:00      |                   \_ [gpg]
1314 pts/0    Z      0:00      |                   \_ [gpg]
1317 pts/0    Z      0:00      |                   \_ [gpg]
1320 pts/0    Z      0:00      |                   \_ [gpg]
1324 pts/0    Z      0:00      |                   \_ [gpg]
1327 pts/0    Z      0:00      |                   \_ [gpg]
1330 pts/0    Z      0:00      |                   \_ [gpg]
1333 pts/0    Z      0:00      |                   \_ [gpg]
1334 pts/1    Ss+    0:00      |                   \_
/usr/bin/ncftp -u
******* backup.server.de
1335 pts/0    R+     0:00      \_ ps -u user xf

reply via email to

[Prev in Thread] Current Thread [Next in Thread]