Re: [O] [Orgmode] Slow speed of week and month views

[Tim Cross]

I think the only viable first step is the profiling.

One of the main reasons I like org-mode is that all the data is just
text in files and it does not have any dependencies on other external
systems apart for publishing/exporting.

While there may be a need for external utilities to improve performance
for large/complex org files, this should be the last resort. The more
org relies on external packages, the more complex things become for
maintaining support on multiple platforms.

The first step is profiling so that we really understand where the
bottlenecks are. It may be these can only be resolved in some cases
using sqlite/mongodb/something-else, but I'm not sure if we know that
yet.

Tim

John Kitchin writes:

> I can think of two possibilities for a future approach (besides a deep dive
> on profiling the current elisp to improve the speed there). They both
> involve some substantial coding though, and would probably add
> dependencies. I am curious what anyone things about these, or if there are
> other ideas.
>
> One is to use the new dynamic module capability to write an org parser in
> C, or a dedicated agenda function, which would presumably be faster than in
> elisp.  This seems hard, and for me would certainly be a multiyear project
> I am sure! The downside of this is the need to compile the module. I don't
> know how easy it would be to make this work across platforms with the
> relatively easy install org-mode currently has. This could have a side
> benefit though of a c-lib that could be used by others to expand where
> org-mode is used.
>
> The other way that might work is to rely more heavily on a cached version
> of the files, perhaps in a different format than elisp, that is faster to
> work with. The approach I have explored in this is to index org files into
> a sqlite database. The idea then would be to generate the agenda from a sql
> query. I use something like this already to "find stuff in orgmode
> anywhere". One of the reasons I wrote this is the org-agenda list of files
> isn't practical for me because my files are so scattered on my file system.
> I had a need to be able to find TODOs in research projects in a pretty wide
> range of locations.
>
> The code I use is at
> https://github.com/jkitchin/scimax/blob/master/org-db.el, and from one
> database I can find headlines, contacts, locations, TODO headlines across
> my file system, all the files that contain a particular link, and my own
> recent org files. This approach relies on emacsql, and a set of hook
> functions to update the database whenever a file is changed. It is not
> robust, e.g. the file could be out of sync with the db if it is modified
> outside emacs, but this works well enough for me so far. Updated files get
> reindexed whenever emacs is idle. It was a compromise on walking the file
> system all the time or daily, or trying to use inotify and you can always
> run a command to prune/sync all the files any time you want.
>
> sqlite is ok, but with emacsql you cannot put strings in it directly (at
> least when I wrote the org-db code), which has limited it for full-text
> search so far. Also with text, the db got up to about 0.5 GB in size, and
> started slowing down. So it doesn't have text in it for now. It has all the
> other limitations of sqlite too, limited support for locking, single
> process....
>
> I am moderately motivated to switch from sqlite to MongoDB, but the support
> for Mongo in emacs is pretty crummy (I tried writing a few traditional
> interfaces, but the performance was not that good, and limited since Mongo
> uses bson, and it is just not the same as json!). Why Mongo? Mostly because
> the Mongo query language is basically json and easy to generate in Emacs,
> unlike sql. Also, it is flexible and easy to adapt to new things, e.g.
> indexing src-blocks or tables or whatever org-element you want. (And I want
> to use Mongo for something else too ;). Obviously these all add
> dependencies, and might not be suitable for the core org-mode distribution.
> But I do think it is important to think about ways to scale org-mode while
> maintaining compatibility with the core.
>
> The main point of the database was to get a query language, persistence and
> good performance. I have also used caches to speed up using bibtex files,
> and my org-contacts with reasonable performance. These have been all elisp,
> with no additional dependencies. Maybe one could do something similar to
> keep an agenda cache that is persistent and updated via hook functions.
>
> Thoughts?
>
> John
>
> -----------------------------------
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
>
>
> On Sat, Aug 5, 2017 at 5:09 PM, Karl Voit <address@hidden> wrote:
>
>> Thanks for the insight!
>>
>> * Adam Porter <address@hidden> wrote:
>> >
>> > But doing that would, as Carsten said, require rewriting a lot of code.
>> > Essentially you'd be creating a new agenda system, so you'd have to
>> > reimplement a lot of existing code.  You could do it in parallel, rather
>> > than replacing existing code, so you wouldn't have to break the existing
>> > agenda system.  But I don't think there's any way to shortcut writing
>> > the new system.  I don't think there's any "low hanging fruit."
>>
>> My daily agenda takes twenty seconds, my weekly approximately a
>> minute and generating a monthly agenda is something I can only do
>> when I plan to leave the computer for a longer break.
>>
>> Org-mode does not scale well, I'm afraid.
>>
>> Repeatedly going through all agenda files does not seem to be a very
>> reasonable design choice for the general use-case. I would like to
>> see an alternative approach for the future of Org-mode.
>>
>> --
>> get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
>>        > get Memacs from https://github.com/novoid/Memacs <
>> Personal Information Management > http://Karl-Voit.at/tags/pim/
>> Emacs-related > http://Karl-Voit.at/tags/emacs/
>>
>>
>>


-- 
Tim Cross