Re: persistent data feature

[Tomas Hlavaty]

On Sat 11 Dec 2021 at 09:30, Ihor Radchenko <yantar92@gmail.com> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> There will, of course, be large amounts of data, and that will be slow.
>>> (People already do this with a number things, and the results are
>>> predictably sluggish.)
>>
>> To help me understand this discussion, I think it would help me to have
>> examples of such large databases currently implemented as text files,
>> along with an idea of what "large" means in this context (how many MBs)
>> and where the "slow"ness manifests itself.
>
> I can think of two examples of large text databases: elfeed and recent
> org-mode's caching (org-persist).
>
> The slowness manifests itself on reading/printing(saving) the database
> and from the requirement to load the whole database into memory for
> queries.
>
> We are talking about 300Mb for elfeed and up to 20-30Mb per object
> (usually below 1Mb though) for org-persist with 100-200Mb total object
> size.

On the other hand, my notmuch (emacs email reader) database is 30GB and
lives externally out of emacs without issues.
Why link sqlite and not xapian?
Where does it stop?