On Tue, Jun 28, 2011 at 4:56 PM,
<address@hidden> wrote:
On 28.06.2011 23:39, Kenneth Loafman wrote:
> We make it in a thread so it can proceed.
proceed for what. we can as well wait for gpg to finish up and go on with the next file. can't we?
No, it would not work that way. Think of it as a list of files being iterated in sync, kind of a sideways iteration, like a column moving across a spreadsheet. When that iteration is complete, all files are closed, i.e. the waitpid() happens. By starting threaded_waitpid() early, we're able to harvest the return code and close the processes as soon as they complete, not when the iteration is complete. Then when duplicity finally closes the process, the return code is returned and all is normal.
>At the end of the waitpid all of the closed file handles are harvested (except on a Mac) and the task is allowed to close.
i can't see code that harvests it, so you speak of waitpid doing that on it's own, do you?
if so:
why not close them manually before calling waitpid? can't hurt, might help?
Yes, the harvesting is done by waitpid itself, when the task is allowed to close. The gpg task closes the files itself.
>Without the threading, you get a pile of gpg processes that don't go away until the very end.
well e.g. ncftp calls don't stick until the end and are done totally without threading. simple system calls.
That's because ncftp is wait'ed for. The gpg tasks are not.
>The parent has to issue the waitpid to get this to happen, and on its own, duplicity waits until the very end of the iteration to take care of this task.
i already understood what waitpid should accomplish. sadly it does not under certain, not well known, circumstances. i try to think of solutions here, or at least approaches that might resolve this issue once and for all.
I appreciate the work. It is a hard problem.
Then again, the real fix is to not do long strings of incrementals. We're trying to provide a fix for what is really an operator error.
...Ken
ede/duply.net
>
> ...Ken
>
> On Tue, Jun 28, 2011 at 3:38 PM, <
address@hidden <mailto:
address@hidden>> wrote:
>
> i just read a little about the process closing with waitpid in python. i also read through GnuPGInterface.py and gpg.py.
>
> if i understand correctly this gpg.py opens the process and deals as a filehandler which pipes data in or out of the process. on close it is supposed to call the wait method, which actually calls waitpid, which should theoretically end the process thread.
>
> two things occure to me.
>
> thing one:
> i miss an explicit closing of pipes and file handlers here. i could imagine that open fh/pipes could leave a thread hanging. a loop closing all those in wait() might help here.
>
> thing two:
> the other is, why do we actually thread this? is duplicity multithreaded at all? don't we actually write/read one file after the other?
> notable exception maybe the asynchronous mode. but that's uploading vs file creation in parallel so no gpg threading needed here.
> crazy idea. let's tell GnuPGInterface.py to ignore threading, i see it is built with a fallback if threading returns a 0 pid and simply stays in the parent process.
>
> just my two things,
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden <mailto:address@hidden>