Hi all,
I've accumulated several questions/observations surrounding language embedding:
I would really like to take advantage of treesit range rules to handle embedded Scheme (especially as my current hacky implementation of indentation and font lock is a major performance bottleneck), but it looks to me like treesit expects embedded languages to always require a second parser. LilyPond is an odd case because embedded Scheme lacks a closing delimiter, meaning a full Scheme parser is required to find the end of a Scheme block. As a result, the LilyPond Tree-sitter grammar already builds in a complete Scheme grammar. Is there a way to use treesit range rules for this situation, where I don't need a separate parser, but I still need the ranges themselves?
More generally, I think it would be useful to separate the handling of embedded parsers from the code that updates range overlays based on query rules, as query ranges have many applications beyond language embedding. As one example, LilyPond's variadic music function syntax makes it essentially impossible for a Tree-sitter grammar to distinguish between music and markup input modes. My plan is to combine interactive evaluation via a LilyPond Geiser REPL with treesit queries to generate ranges distinguishing music vs. markup. These would be an entirely separate semantic layer from language ranges, but would have all the same performance and overlay bookkeeping challenges.
LilyPond embedded Scheme is unusual for another reason: it is not semantically separate from the surrounding code. Basically all LilyPond syntax can be substituted with a Scheme _expression_. Arguably, LilyPond could be considered an alternate syntax for a Scheme domain-specific language, rather than a separate language. It's not uncommon to encounter LilyPond files that are 50% or even 90% embedded Scheme.
I mention this because it's really not enough just to emulate the appearance of scheme-mode; since LilyPond *is* Scheme, it really needs the functionality of scheme-mode and Scheme tooling. I've gotten a fair bit of the way toward this goal, but it's been surprisingly difficult to figure out how to do it. My impression is that treesit major modes are expected to support embedded languages by relying on font lock and indentation code, etc. provided by the treesit modes for those languages. But I also have the impression that there is little reason for treesit modes for Scheme and Lisp to ever be developed, since the existing tooling is so good. What then is the intended approach for a treesit major mode to support embedding for a language lacking a ts-mode?
In closing let me say this is my first time posting to this list, so please be kind. In fact, I only started learning/using Emacs about a month before I started this project. I've read a lot of the available documentation but guidance and feedback on lilypond-ts-mode would be very welcome. I'm sure there are all sorts of unwritten conventions I've unwittingly failed to follow.
Saul