Sqlite-backed buffers and generalized buffer persistance backends

[Vladimir Kazanov]

Hi all,

Recently I started playing with an idea of buffers backed by Sqlite,
or maybe even a generalized framework for buffer backends not rooted
in files.

Here's my context:

1. I do all of my note-taking in Emacs. In the last 10 years I used
pure org-mode, org-journal, org-grain, denote and settled down on
org-roam for now.

2. All of these tools are focused on the text, and having files
backing the notes is secondary and distracting to the natural
note-taking process.

3. Moreover, indexing, linking and backlinking notes often requires a
referenceable unit more granular than a file. Org-roam provides a
sqlite-based backend handling indexing, but this has to be resynced a lot.
Denote mostly just builds on text-based tools and file names.

And all of these struggle to reconcile the filesystem abstraction with
having a graph of interlinked (textual or graphical) sub-file nodes.
It almost feels as if we need a better backend for buffers showing the
data...

As usual is the case, Emacs buffers can be separated from files rather
easily, and I built a quick proof-of-concept. See the attached
buffer-sqlite.el. It provides revert-buffer-function and
write-contents-function that use a sqlite db to save/revert contents
of a buffer when sqlite-backed-minor-mode is enabled.

To play with the code:

1. Open any buffer, like *scratch*. Do (sqlite-backed-minor-mode 1).

2. Edit the buffer.

3. Save the buffer using C-x C-s.

4. inspect the database using the sqlite3 cli utility. Note how buffer
contents become a row in the buffer_contents table.

5. Kill the *scratch* buffer.

6. Call (buffer-sqlite-load-buffer "*scratch*). Note how the buffer
was restored from the database and the minor mode was enabled.

The implementation is trivial and dirty but I feel there's some potential in the
idea. Developing this into a generalized API opens interesting
possibilities:

1. simple sqlite-backed buffers.

2. same sqlite buffers but compressed on the fly.

3. buffers backed by complicated data structures provided by sqlite.
For example, the save-buffer backend could split the buffer text into
paragraphs, hash them and save as a hash to text mapping. This is
automated deduplication right there.

4. additional metadata for buffers external to buffer text.

5. all powers of sqlite buffer data indexing. I felt the need for
something like this for decades.

6. building buffers from database contents in non-trivial ways (text
chunks, binary data and so on).

If maintainers think this has potential I am happy to develop the
prototype further.

So what do you think?

Thanks

-- 
Regards,

Vladimir Kazanov

buffer-sqlite.el
Description: Text Data