Automatic (e)tags generation and incremental updates

[Dmitry Gutov]

Hi all!

I went back to an old thread from 2018 and updated the patch. Together 
with 'project-files' being faster than previously and a few other 
tweaks, the current state feels surprisingly usable, from small to 
medium projects, up until tags file sizes where code completion starts 
giving an unpleasant latency.

The code lives in the branch scranch/etags-regen and currently it's just 
triggered by etags when you try using the etags xref backend without 
having a tags table visited. I'll also attach a patch against master to 
this email.

This time I took the approach of implementing the meat of incremental 
updates inside Emacs, instead of relying on an external tool. And it 
works better when Emacs knows which exact things it needs to update, and 
which it doesn't need to. E.g., it doesn't need to re-visit the whole 
45MB tags file if it just needs a file re-indexed; it directs etags's 
output to the buffer and then appends it to the file. Completion table 
updates could be made faster this way too, although I think we'd need 
some new data structure for them.

The main question remains how to update information for files that have 
been deleted, or edited from outside Emacs (including by 'git 
checkout'). Two main approaches that I'm thinking of:

- On a timer, re-create the list of project files together with their 
mtimes (for instance, by piping through 'stat -c %Y') and compare with 
the previous saved list. Given a big enough project, it will create 
intermittent stalls in the UI, though, which could be unpleasant. But it 
can be the first approach to be implemented anyway.

- filenotify. I have already been warned here that it's unreliable, 
prone to overflowing due to excessive notifications or file watching 
limits. The current API doesn't allow to want a directory recursively 
either (which would be required to know about new files). There is a 
project walled Watchman, however (https://github.com/facebook/watchman) 
which I have read good things about, and it must use some sort of file 
notification API under the covers. Perhaps if someone here is familiar 
with its architecture, they could advise how to build a better 
abstraction on top of inotify in Emacs as well.

Of course, we can give the users a manual knob as well (in could come in 
the form of enabling/disabling an associated minor mode), but first we 
should try to make it work automatically, at least for projects of up to 
certain size.

Another question I'd like to ask is where the maintainers want to see 
this code: inside etags.el, in a new file near it (etags-regen.el, 
perhaps?), or just in GNU ELPA? It can be a minor mode, or a value in 
some user option like proposed in bug#43086.

Normally I would go the ELPA route straight away, but this kind of 
feature (automatic code indexing) is what Emacs sorely needs OOtB, IMHO. 
It loses here even to Sublime Text 3 released 7 years ago, which is not 
a very "smart" editor. And "democratizing" etags this way should result 
in better adoption, bug reports, feature requests, etc.

Please give it a try and comment.

(Cc'd some folks who went near previous discussions.)

etags-regen.diff
Description: Text Data