Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

> Org Agenda code does not wait for keyboard input; it's busy building the
> agenda.  This is the case with most code in Emacs: it's not written to
> be asynchronous, and it doesn't return to the main thread until done.
> So you can sprinkle yields here and there and maybe be able to move
> point around while some code is running, but that will decrease
> performance, as well as introducing another level of complexity and
> another class of bugs (e.g. what if the user modifies a buffer while the
> agenda code is scanning it?).

Thanks for the explanation.

> AFAIK there exists no way to do such a thing.  Buffers are not designed
> to be serialized/deserialized like that.  You could try writing some
> Elisp code to do it, but the end result would probably be much slower
> than existing agenda code, as well as more difficult to debug.

Yeah. Even re-initialisation of, for example, overlays in org buffer is 
likely to take too much time.

> As you can see in org-agenda.el, it's complicated.  Remember that an
> Emacs process is like a Lisp image, full of state.  The more symbols and
> other structures you copy to the async Emacs process (by printing and
> reading them as text, remember), the slower it's going to be--and it
> will always be slower than not using async.

> Asynchronous code is not faster; it's generally slower because of
> yielding and synchronization.

I see now that generating agenda in separate process will cause too much
overheads.
Anyway, I will try to throw yields into agenda code just to check how
bad the performance can degrade.

> org-ql doesn't use skip functions, just queries.

Skip functions are essentially used-defined queries as soon as the
queries are tested against every headline.
I can rewrite my skip functions into queries, but I don't expect much
improvement since org-ql seems to use org-entry-get, which is the main
performance bottleneck for my agenda generation.

Best,
Ihor

adam Porter <address@hidden> writes:

> Ihor Radchenko <address@hidden> writes:
>
>>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>>> cooperative, so functions called as threads must yield back to the main
>>> thread for Emacs to do anything else before the function returns.
>>
>> I tried to read the manual, but I clearly misunderstand something.
>> The manual says:
>>
>>>   Currently, thread switching will occur upon explicit request via
>>> ‘thread-yield’, when waiting for keyboard input... 
>>
>> So, except directly calling thread-yield, it should be possible to
>> trigger switching the current thread when keyboard input is expected.
>> I tried the following demo code:
>>
>> (defun test ()
>>   (let ((a 0))
>>     (dotimes (_ 5)
>>       (setq a (1+ a))
>>       (sleep-for 2)
>>       (message "%s" a))))
>>
>> (progn ;This should return to command loop quickly
>>   (make-thread #'test)
>>   (message "Executed...")); `eval-last-sexp' here
>>
>> I can move around the buffer while the progn is running.
>> However, it is not the case with `org-agenda-redo' for a reason I do not
>> fully understand.
>
> Org Agenda code does not wait for keyboard input; it's busy building the
> agenda.  This is the case with most code in Emacs: it's not written to
> be asynchronous, and it doesn't return to the main thread until done.
> So you can sprinkle yields here and there and maybe be able to move
> point around while some code is running, but that will decrease
> performance, as well as introducing another level of complexity and
> another class of bugs (e.g. what if the user modifies a buffer while the
> agenda code is scanning it?).
>
>>> 1.  The process would have to load the same Org buffers, which takes
>>>     time, especially in large buffers.  Depending on configuration, it
>>>     can take some time, indeed.
>>
>>> 3.  Ensuring that configuration and state between the main Emacs process
>>>     and the separate, agenda-generating process is not necessarily
>>>     simple.  Consider as well that if a buffer had unsaved changes,
>>>     those would not be readable by the other process, which would lead
>>>     to invalid results.  One could force the buffers to be saved first,
>>>     but that may not always be desirable, as saving buffers can have
>>>     side effects.
>>
>> Why cannot org-buffer simply be copied into the subordinate process? If
>> all be buffer-locals, text properties, and overlays are copied directly
>> from the main emacs process, there may be no need to even initialise
>> org-mode (the idea is to do something similar to clone-buffer).
>
> AFAIK there exists no way to do such a thing.  Buffers are not designed
> to be serialized/deserialized like that.  You could try writing some
> Elisp code to do it, but the end result would probably be much slower
> than existing agenda code, as well as more difficult to debug.
>
>> The question though is whether buffer-locals + overlays + propertized
>> .org files text + org-agenda-buffer copy can be sufficient to make the
>> org-agenda-redo run properly. Are there any other buffers, variables,
>> or other environment settings used by org-agenda-redo?
>
> As you can see in org-agenda.el, it's complicated.  Remember that an
> Emacs process is like a Lisp image, full of state.  The more symbols and
> other structures you copy to the async Emacs process (by printing and
> reading them as text, remember), the slower it's going to be--and it
> will always be slower than not using async.
>
>>> If your agenda buffers are taking too long to refresh, you might
>>> consider org-ql's views/saved-searches as an alternative. ...
>>
>> I know org-ql and I am pretty sure that it will improve performance.
>> Actually, if one can make built-in org-agenda asynchronous, org-ql can
>> probably use similar approach and become even faster :)
>
> Asynchronous code is not faster; it's generally slower because of
> yielding and synchronization.
>
>> I am trying on default org-agenda now mostly because my current config
>> is heavily geared towards default agenda and I am not sure if
>> refactoring everything to use org-ql will worth it at the end in terms
>> of performance. I use too many slow custom skip-functions.
>
> org-ql doesn't use skip functions, just queries.
>
>