Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

> On 11 Oct 2022, at 09:46, Po Lu <luangruo@yahoo.com> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Yes, I will.
>> 
>> It isn't our business to develop language parsers, and we have no hope
>> of keeping up with the technology developments in that area and with
>> new languages being invented, well enough to offer reasonable and
>> modern support in Emacs for editing programs in those languages.
> 
> It isn't quite our business to develop mini-gmp either, yet it's also
> kept in-tree.
> 
>> It's the same reason why we depend on text shaping engines like
>> HarfBuzz for rendering complex scripts and other typography-related
>> features, on image libraries to display images, on GnuTLS to support
>> TLS connections, etc. etc.
> 
> But editing text mostly does not involve editing complex shaped text,
> and likewise, TLS connections and image display is mostly orthogonal to
> the business of editing text.
> 
>> We cannot possibly be experts in all of those fields, we don't have
>> the manpower for that.  In the case of support for editing programs we
>> actually tried for many years, and the result is before our eyes --
>> and it's unsatisfactory at best.  We should advance Emacs towards
>> leveraging existing technologies, not try developing those
>> technologies in-house, or replacing them by cheap hacks and kludges.
> 
> Then why is mini-gmp kept in-tree? Portable multiple precision
> arithmetic is at least as complicated (if not more) than tree-sitter,
> yet mini-gmp is a completely satisfactory solution for people who do not
> have libgmp installed.
> 
> mini-tree-sitter would fill the same role: it would expose the same API
> as tree-sitter, but be small enough for Emacs to include and use when
> tree-sitter is not available.
> 
>> It is completely impractical to expect us to be able to develop this
>> in-house.
> 
> Why?  The tree-sitter runtime library is only about 16,000 lines of C
> source code and headers, which is less than xdisp.c alone, and only
> slightly more than keyboard.c.  In addition, the path to the current
> implementation of tree-sitter is well-trodden, so it can't be that hard
> to walk down that path again.
> 
> Such an implementation would be able to make use of existing and new
> tree-sitter grammars without changes.
> 
>> If tree-sitter becomes defunct (something that doesn't seem like it
>> will happen any time soon)
> 
> Famous last words.

Absolutely true. But I don’t consider that a big enough problem to be worried. 

tree-sitter will just be one way to implement a major-mode. There’s still 
others. There can be even more. 

tree-sitter has momentum outside Emacs and high quality parsers for quite a lot 
of complex languages which are troublesome to implement parsers for in Elisp, 
as evident by our current best-effort parsers. 

While there’s no guarantee it will be maintained forever, like Emacs, it 
doesn’t have to either. 

It just needs to be maintained for as long as the languages whose parsers are 
implemented using it are maintained. 

Once a language stops developing, it shouldn’t be a problem for us that 
tree-sitter based parsers for it stops developing as well. 

If a new, better thing comes along ten years down the line, there’s nothing 
preventing us from adapting that for the languages and major-modes which at 
that time will benefit from it. 

Just like we’re doing with tree-sitter now, for the major-modes and languages 
with complex grammar which will benefits from it. 

Disclaimer: I may be biased here, because I’ve spent the last ten+ years trying 
to maintain parsers for C# and typescript in Elisp (and failing at that). And 
TSX has been a thorough no-go. 

For some of these languages, tree-sitter really is the only realistic option we 
have now if we want to support these languages at the level our users(and 
programmers in general) expect. 

I see little to be gained by being philosophical at this point when the reality 
is as clear and harsh as it is. 

> 
>> , we will have to find a suitable replacement -- like we did with
>> text-shaping engine, when the FLT library stopped being actively
>> developed.  Dealing with these changes in the outside world is an
>> important part of the job of the Emacs maintainers
> 
> Text shaping is, at least IME, immensely more complicated than
> interactive language parsing.
> 
>> The present font-lock and indentation features based on our own code
>> can remain in Emacs forever, if someone wants a safe fallback for when
>> worse comes to worst.
> 
> That logic could also have been applied to bignums, yet there was no
> objection to including mini-gmp.
> 
>> But these implementations are from my POV a dead end: they cannot be
>> developed any further, and for some languages, like C++, are already
>> all but failing to reasonably and efficiently support their complex
>> grammars.  Using expertise of others is simply a logical step based on
>> past experience; the decision to develop this in-house would be a step
>> back and a disaster in the long run, IMNSHO.
> 
> I'm not disputing that fact, which is why I proposed to write an
> *implementation* of tree-sitter for inclusion with Emacs, and not to
> develop an equivalent in-house.