Re: [Duplicity-talk] OSError: [Errno 24] Too many open files

[Kenneth Loafman]

I have found the problem and its going to take me a bit to get it fixed.

As it works duplicity creates iterators that traverse each of the sig
files.  The iterators do not close until the process is complete and the
files they are using are being held open.  The gpg process is held as
well since we don't wait for it until we close the file.  The zombie
processes were just a symptom of the real problem, too many open
resources at one time.

This hits directly the issue of too long an incremental chain.  Each
increments sig file is going to have file descriptors open until the
backup/incremental/restore completes.  The only workaround at this point
is to increase the 'ulimit -n' parameter for any long increment chains
and perform a full backup on a more frequent basis.

I have some ideas about how to clean this up, but need to examine the
code a bit closer before committing to a solution.

...Ken

Kenneth Loafman wrote:
> As far as I can tell, its just lots of incrementals.  duplicity has to
> open and process each signature file and, in some cases, the gpg process
> is not being closed completely causing a zombie process.
> 
> This "seems" to be worse on a 64bit system, but I have been able to
> partially reproduce it on a 32bit.  The most zombie processes I've been
> able to see are 5-7 and that's with about 60 incrementals.  Even then,
> all of them go away quickly.
> 
> ...Ken
> 
> Jeff Singer wrote:
>> I have a 64bit Ubuntu 9.04 system here that I'd be happy to attempt to
>> reproduce it over the weekend. Is it basically a backup that involves
>> lots of incrementals with many volumes that's causing this behavior?
>>
>> On Tue, May 12, 2009 at 1:51 PM, Ian Barton <address@hidden
>> <mailto:address@hidden>> wrote:
>>
>>     I have got a couple of 64 bit Ubuntu 9.1 (Jaunty) computers. I can
>>     try running a test on my system if you tell me what you would like
>>     me to do.
>>
>>     FWIW my main computer for running is Debian Lenny 64bit and I have
>>     never seen the problem on it.
>>
>>     Ian.
>>
>>
>>         Thanks for the help!
>>
>>         I'm not sure at this point.  I built a 64bit Ubuntu 9.04 system
>>         and it
>>         seems to be working fine.  Debian may be different, but not much.
>>
>>         As to what's broke, it could even be at the system level, but my
>>         bet is
>>         on gpg.
>>
>>         ...Ken
>>
>>         address@hidden <mailto:address@hidden> wrote:
>>
>>             I am gonna install a 64bit debian in a virtual machine and gonna
>>             doublecheck that ... but what do you think is buggy? python,
>>             gpg ?
>>
>>             ..ede
>>
>>                 The only real change to GnuPGInterface is a shift right
>>                 instead of a
>>                 shift left when printing the error code after an
>>                 exception.  He's not
>>                 getting any exceptions, so he's not hitting the change.
>>                  Other than
>>                 that, its the same code from years ago.
>>
>>                 My best guess is still that its a 32-bit versus 64-bit
>>                 issue.
>>
>>                 ...Ken
>>
>>                 Edgar Soldin wrote:
>>                  
>>
>>                     true they close every time .. the only thing i see
>>                     is that the amount of
>>                     gpg processes grows with the amount of incrementals ...
>>
>>                     @yang: how many incrementals does your  backup hold?
>>
>>                     the other thing is, maybe gpg itself is buggy  ..
>>
>>                     @yang: could you try another gpg version?
>>
>>                     and the third thing
>>
>>                     @ken: is it really impossible for 0.5.17 to use an
>>                     oldGnuPGInterface
>>                     version, that might be laying around on the system?
>>                     How could yang look
>>                     for it?
>>
>>                     .. 2cents again :) ..ede
>>                     -- 
>>                      
>>
>>                         I tried this with tiny volume sizes of 1M over a
>>                         2GB backup (2K
>>                         volumes)
>>                         and did a count of the gpg processes (live,
>>                         defunct) every 2 seconds
>>                         and
>>                         never had more than 5 (3 live, 2 defunct).  This
>>                         was on a very fast
>>                         link
>>                         between local VMs.  Tried both backup and
>>                         restore so it would test both
>>                         directions.  By the end, all were closed.
>>
>>                         It's possible to have some defunct while the
>>                         task scheduler is
>>                         updating,
>>                         but Yang is showing hundreds that are holding
>>                         files open.  That's the
>>                         confusing part.
>>
>>                         ...Ken
>>
>>                         Edgar Soldin wrote:
>>                              
>>
>>                             i really have to be fast (a window of 10s
>>                             let's say) and it helps to
>>                             have a slow backup connection with lots of
>>                             incrementals... but I
>>                             may be
>>                             seeing ghosts here. Can it that it is
>>                             perfectly normal for
>>                             duplicity to
>>                             gpg decrypt something per each incremental
>>                             but close all processes
>>                             just
>>                             a bit later? .. just a guess
>>                             ..ede
>>                                      
>>
>>                                 I cannot reproduce this on a 32bit
>>                                 machine at all.  I'll be able
>>                                 to test
>>                                 on a 64bit machine sometime this week
>>                                 and, hopefully, find the
>>                                 problem.
>>
>>                                 ...Ken
>>
>>                                 address@hidden
>>                                 <mailto:address@hidden> wrote:
>>                                                
>>
>>                                     i seem to get closer to it ...
>>                                     the more incrementals (regardless if
>>                                     file: or ftp: backend) the more
>>                                     defunct gpg processes appear .. but
>>                                     dissappear again while the
>>                                     backup is
>>                                     still running
>>
>>                                     I did
>>                                     initial backup of 3GB
>>                                     touched a new test file to have a
>>                                     change & did a 2nd backup
>>                                     touched a new test2 file to have a
>>                                     change & did a 3nd backup
>>                                     ...
>>
>>                                     and since the 3rd backup the defunct
>>                                     gpg processes keep appearing
>>
>>                                     essentially I say: Take any more
>>                                     than 3 incrementals in size
>>                                     backup and
>>                                     do a new incremental and you will
>>                                     see those zombie gpg processes ..
>>
>>                                     while I am not sure that this is
>>                                     what happens on the ulimit
>>                                     problem I
>>                                     thought I should report these
>>                                     observations ... ede
>>
>>                                     PS:
>>
>>                                     here a 'ps xf -u user' shortly after
>>                                     starting the sixth run .. the
>>                                     number of gpg zombies seems to be
>>                                     number of sets (full + incr's)
>>                                     minus 2 ..
>>
>>                                     9122 pts/0    S+     0:00        
>>                                      \_ /bin/bash
>>                                     
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>                                     test backup
>>                                     19128 pts/0    S+     0:00          
>>                                        \_ /bin/bash
>>                                     
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>                                     backup
>>                                     19193 pts/0    S+     0:00          
>>                                            \_ /bin/bash
>>                                     
>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>                                     backup
>>                                     19194 pts/0    R+     0:03          
>>                                                \_ /usr/bin/python
>>                                     
>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>                                     --verbosity 4
>>                                     --encrypt-key 00000000--sign-key
>>                                     00000000--gpg-options=--always-trust
>>                                     --volsize 1 --exclude-globbing-filelist
>>                                     /srv/www/user//.ftplicity/test/excl
>>                                     19196 pts/0    SL+    0:00          
>>                                                    \_ gpg
>>                                     --logger-fd 4
>>                                     --passphrase-fd 8 --batch --no-tty
>>                                     --default-key 00000000--recipient
>>                                     00000000--no-secmem-warning
>>                                     --always-trust --encrypt --sign
>>                                     19197 pts/0    SL+    0:00          
>>                                                    \_ gpg
>>                                     --status-fd 6
>>                                     --passphrase-fd 11 --logger-fd 5
>>                                     --batch --no-tty --default-key
>>                                     00000000--no-secmem-warning
>>                                     --always-trust --decrypt
>>                                     19199 pts/0    Z+     0:00          
>>                                                    \_ [gpg]
>>                                     <defunct>
>>                                     19200 pts/0    Z+     0:00          
>>                                                    \_ [gpg]
>>                                     <defunct>
>>                                     19201 pts/0    Z+     0:00          
>>                                                    \_ [gpg]
>>                                     <defunct>
>>                                     19202 pts/0    Z+     0:00          
>>                                                    \_ [gpg]
>>                                     <defunct>
>>                                     19203 pts/0    SL+    0:00          
>>                                                    \_ gpg
>>                                     --logger-fd
>>                                     23 --passphrase-fd 28 --batch
>>                                     --no-tty --default-key
>>                                     00000000--recipient
>>                                     00000000--no-secmem-warning
>>                                     --always-trust --encrypt --sign
>>
>>
>>                                                          
>>
>>                                         hmm .. you are right these
>>                                         processes are zombie (they appear in
>>                                         top as
>>                                         such) .. later on the processes
>>                                         disappear, while duplicity is still
>>                                         running ..
>>                                         but I don't feel this is
>>                                         normal.. or is it?
>>                                         just to be more sure, I'll try a
>>                                         1MB chunk full backup of my 3GB
>>                                         home
>>                                         dir .. let's see with how many
>>                                         zombies I'll end up ...
>>
>>                                         interestingly my ulimit =
>>                                         unlimited ... seems to be default in
>>                                         the old
>>                                         SUSE 10.2, btw also on my debian
>>                                         5.0 box both 32bit
>>                                         can't remember to ever have
>>                                         touched this setting
>>
>>                                         ..ede
>>                                         -- 
>>                                                                    
>>
>>                                             These are not open
>>                                             processes, they are zombie
>>                                             processes, so no
>>                                             real
>>                                             resources taken.  That
>>                                             simplifies it a bit.  What
>>                                             normally
>>                                             causes this
>>                                             is the failure of a parent
>>                                             process to properly retrieve its
>>                                             exit status,
>>                                             but this is not the case or
>>                                             more people would have this
>>                                             problem.
>>
>>                                             As far as I can tell, the
>>                                             only systems this is
>>                                             happening on are
>>                                             64bit,
>>                                             and not even all of those.
>>                                              Some of my systems are
>>                                             64bit and
>>                                             they don't
>>                                             show this, so I'm wondering
>>                                             if this could be limited to a
>>                                             particular
>>                                             version of GnuPG, or what.
>>
>>                                             Edgar, from your note I
>>                                             could not tell, is this
>>                                             happening to
>>                                             you too?
>>
>>                                             ...Ken
>>
>>                                             address@hidden
>>                                             <mailto:address@hidden>
>>                                             wrote:
>>                                                                        
>>                                                    
>>
>>                                                 just my quick
>>                                                 observation .. a simple
>>                                                 incr backup opens and
>>                                                 leaves over
>>                                                 20 gpg processes open. I
>>                                                 don't think this is
>>                                                 healthy and also can
>>                                                 imagine that this
>>                                                 multiplies if the
>>                                                 volumes get smaller
>>                                                 (mine are
>>                                                 50MB).
>>
>>                                                 regards ede
>>
>>
>>                                                 address@hidden:~> ps -u user 
>> xf
>>                                                  PID TTY      STAT  
>>                                                 TIME COMMAND
>>                                                 32731 ?        S    
>>                                                  0:00 sshd: address@hidden/0
>>                                                 32732 pts/0    Ss    
>>                                                 0:00  \_ -bash
>>                                                 1167 pts/0    S    
>>                                                  0:00      \_ /bin/bash
>>                                                 /usr/local/bin/ftplicity
>>                                                 profile backup
>>                                                 1168 pts/0    S    
>>                                                  0:00      |   \_ /bin/bash
>>                                                 
>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>                                                 profile backup
>>                                                 1174 pts/0    S    
>>                                                  0:00      |       \_
>>                                                 /bin/bash
>>                                                 
>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>                                                 backup
>>                                                 1239 pts/0    S    
>>                                                  0:00      |          
>>                                                 \_ /bin/bash
>>                                                 
>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>                                                 backup
>>                                                 1240 pts/0    S    
>>                                                  0:05      |            
>>                                                   \_ /usr/bin/python
>>                                                 
>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>                                                 --verbosity 4
>>                                                 --encrypt-key XXXXXXX
>>                                                 --sign-key B59ECD99
>>                                                 --gpg-options=--always-trust
>>                                                 --full-if-older-than 1M
>>                                                 --volsize 50
>>                                                 --exclude-globbing-filelist
>>                                                 /srv/www/...
>>                                                 1249 pts/0    SL    
>>                                                 0:00      |            
>>                                                       \_ gpg
>>                                                 --logger-fd 4
>>                                                 --passphrase-fd 8
>>                                                 --batch --no-tty
>>                                                 --default-key XXXXXXXX
>>                                                 --recipient
>>                                                 XXXXXXXX
>>                                                 --no-secmem-warning
>>                                                 --always-trust --encrypt
>>                                                 --sign
>>                                                 1265 pts/0    SL    
>>                                                 0:00      |            
>>                                                       \_ gpg
>>                                                 --status-fd 6
>>                                                 --passphrase-fd 11
>>                                                 --logger-fd 5 --batch
>>                                                 --no-tty --default-key
>>                                                 XXXXXXXX
>>                                                 --no-secmem-warning
>>                                                 --always-trust --decrypt
>>                                                 1267 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1272 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1275 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1278 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1282 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1286 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1289 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1292 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1296 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1299 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1302 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1305 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1308 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1311 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1314 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1317 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1320 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1324 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1327 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1330 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1333 pts/0    Z    
>>                                                  0:00      |            
>>                                                       \_ [gpg]
>>                                                 <defunct>
>>                                                 1334 pts/1    Ss+  
>>                                                  0:00      |            
>>                                                       \_
>>                                                 /usr/bin/ncftp -u
>>                                                 ******* backup.server.de
>>                                                 <http://backup.server.de>
>>                                                 1335 pts/0    R+    
>>                                                 0:00      \_ ps -u user xf
>>
>>
>>
>>
>>                                                 
>> _______________________________________________
>>                                                 Duplicity-talk mailing list
>>                                                 address@hidden
>>                                                 <mailto:address@hidden>
>>                                                 
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>                                                                        
>>                                                                      
>>
>>                                              
>> ------------------------------------------------------------------------
>>
>>
>>                                             
>> _______________________________________________
>>                                             Duplicity-talk mailing list
>>                                             address@hidden
>>                                             <mailto:address@hidden>
>>                                             
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>                                                                        
>>                                                      
>>
>>                                         
>> _______________________________________________
>>                                         Duplicity-talk mailing list
>>                                         address@hidden
>>                                         <mailto:address@hidden>
>>                                         
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>                                                                      
>>
>>                                     
>> _______________________________________________
>>                                     Duplicity-talk mailing list
>>                                     address@hidden
>>                                     <mailto:address@hidden>
>>                                     
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>                                                            
>>
>>                                  
>> ------------------------------------------------------------------------
>>
>>
>>                                 
>> _______________________________________________
>>                                 Duplicity-talk mailing list
>>                                 address@hidden
>>                                 <mailto:address@hidden>
>>                                 
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>                                                  
>>
>>                             _______________________________________________
>>                             Duplicity-talk mailing list
>>                             address@hidden
>>                             <mailto:address@hidden>
>>                             
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>                                        
>>
>>                          
>> ------------------------------------------------------------------------
>>
>>
>>                         _______________________________________________
>>                         Duplicity-talk mailing list
>>                         address@hidden
>>                         <mailto:address@hidden>
>>                         
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>                                
>>
>>                     _______________________________________________
>>                     Duplicity-talk mailing list
>>                     address@hidden
>>                     <mailto:address@hidden>
>>                     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>                        
>>
>>
>>                  
>> ------------------------------------------------------------------------
>>
>>                 _______________________________________________
>>                 Duplicity-talk mailing list
>>                 address@hidden <mailto:address@hidden>
>>                 http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>                  
>>
>>
>>
>>             _______________________________________________
>>             Duplicity-talk mailing list
>>             address@hidden <mailto:address@hidden>
>>             http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>
>>
>>
>>         
>> ------------------------------------------------------------------------
>>
>>
>>         _______________________________________________
>>         Duplicity-talk mailing list
>>         address@hidden <mailto:address@hidden>
>>         http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>
>>
>>
>>     _______________________________________________
>>     Duplicity-talk mailing list
>>     address@hidden <mailto:address@hidden>
>>     http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc
Description: OpenPGP digital signature