Re: Tree-sitter api

[Yuan Fu]

> 
> Why do we need to document the language definitions?  When a Lisp
> programmer defines font-lock and indentation for a programming
> language in the current Emacs, do they necessarily need to consult the
> language grammar?

> […]

> This stuff should be known to TS; the Lisp programmer only needs to be
> aware of the results of lexical and syntactical analysis, in terms of
> their Lisp expressions (Lisp data structures with appropriate symbols
> and fields).

I demonstrate the reason why one needs to consult the source a few messages 
back:

> Tree-sitter has no indentation calculation feature. Major mode writers 
> genuinely need to read the source of the tree-sitter language definition. The 
> source tells us what will be in the syntax tree parsed by tree-sitter, and 
> the node names differ from one language to another. For example, if I want to 
> fontify type identifiers in C with font-lock-type-face, I need to know how is 
> type represented in the syntax tree. I look up the source[1], and find
> 
>    _type_specifier: $ => choice(
>      $.struct_specifier,
>      $.union_specifier,
>      $.enum_specifier,
>      $.macro_type_specifier,
>      $.sized_type_specifier,
>      $.primitive_type,
>      $._type_identifier
>    ),
> 
> This roughly translates to 
> 
> _type_specifier := <struct_specifier>
>                 | <union_specifier>
>                 | <enum_specifier>
>                 | <macro_type_specifier>
>                 | <sized_type_specifier>
>                 | <primitive_type>
>                 | <_type_identifier>
> 
> in BNF
> 
> From this (and some other hint) I know I need to grab all the _type_specifier 
> nodes in the syntax tree, find their corresponding text in the buffer, and 
> apply font-lock-type-face. And type identifiers in another language will be 
> named differently, tree-sitter doesn’t provide an abstraction for semantic 
> names in the syntax tree.


>> And I want to also point out that as Emacs core developers, we can’t 
>> possibly provide a good translation from convention language names to their 
>> tree-sitter name (C# -> c-sharp). Maybe we can do a half-decent job, but 1) 
>> that won’t cover all available languages, and 2) if there is a new language, 
>> we need to wait for the next release to update our translation. It is better 
>> for the major mode writers to provide the information on how to translate 
>> names.
> 
> The database used by the conversion should definitely be extensible.
> But that doesn't mean it should be empty.
> 
> Anyway, we've spent enough time on this issue.  If you are still
> unconvinced, feel free to do it your way, and let the chips fall as
> they may.

I’ll do it the way I see fit. You can always comment in the final review (or 
something). Thanks.

Yuan