Re: Recent updates to tree-sitter branch

[Yuan Fu]

> On Oct 2, 2022, at 10:58 PM, Ihor Radchenko <yantar92@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>>> Also bear in mind that the override flag can only be applied to the whole 
>>>> query, rather than individual captured nodes.
>>> 
>>> How does it change anything? I may be misunderstanding something---can
>>> you provide some illustrative example clarifying whole query vs.
>>> individual notes?
>> 
>> What I meant is that, for font-lock-keywords, one can set override flag for 
>> each individual match:
>> 
>> (string-regex font-lock-string-face t)
>> (function-name-regexp font-lock-function-name-face nil)
>> (class-name-regexp font-lock-type-face t)
>> ...
>> 
>> But for tree-sitter, a query contains many matches and the flag is set for 
>> the query. So if I want to use different override flag for different 
>> matches, I need to split them into two queries:
>> 
>> (treesit-font-lock-rules
>> :language 'python
>> :override 'append
>> '((string) @python--treesit-fontify-string
>>   ((string) @font-lock-doc-face
>>    (:match "^\"\"\"" @font-lock-doc-face))
>>   (interpolation (identifier) @font-lock-variable-name-face))
>> 
>> :language 'python
>> :override nil
>> '((function_definition
>>    name: (identifier) @font-lock-function-name-face)
>> 
>>   (class_definition
>>    name: (identifier) @font-lock-type-face)
>> 
>>   ;; Comment and string.
>>   (comment) @font-lock-comment-face))
> 
>> That means if we use override=nil as default, it is very likely that users 
>> need to explicitly set override to t for the whole query, or split the query 
>> into separate parts. Nothing serious, but it seems less convenient.
> 
> What about allowing (@python--treesit-fontify-string 'append) to specify
> the override?

That’s not impossible but I don’t think it’s worth it.

> 
>> A real use-case for override is how I fontified Python strings above. I have 
>> three matches for (1) all strings (2) docstrings (3) variable names in 
>> string interpolations. IMO it’s intuitive and convenient for later more 
>> specific matches to override earlier more general matches.
> 
> The current convention in font-lock-keywords is exactly opposite -
> earlier matches are more specific, and they are later not replaced by
> later more general matches.
> 
> Also, for reference, I am currently developing parser-based
> fontification for Org.
> 
> I am using a somewhat different approach (closer to font-lock-keywords):
> 
> ((drawer property-drawer) ;; <- match node types
>  (:begin-marker 'org-drawer t) ;; <- apply fontification to :begin-marker 
> field inside 
>  (:end-marker 'org-drawer t)) ;;  <- ...                    :end-marker ....
> ((headline inlinetask)
>  (:title-line
>   (if (org-element-match-property :archivedp) ;; <- Elisp matching of the 
> node properties
>       'org-archived
>     (pcase (org-element-match-property :todo-type) ;; <- ....
>       (`todo (when org-fontify-todo-headline 'org-headline-todo))
>       (`done (when org-fontify-done-headline 'org-headline-done))
>       (_ nil)))
>   t)) ;; <- override
> ((bold italic underline verbatim code strike-through)
>   (:full-no-blank '(face nil org-emphasis t))) ;; <- fontify contents of the 
> matched node
> 
> Also, see 
> https://github.com/yantar92/org/blob/feature/org-font-lock-element/lisp/org-font-lock.el#L574
> 
> From my experience re-implementing the vanilla fontification,
> fontification order is important and may create subtle issues when not
> designed carefully.

Thinking more about it, it is not a tall order to ask to add :override t for 
those who want override by default. And we can keep the consistency between 
font-lock and tree-sitter. I just need to make sure to document it clearly so 
no one misses it.

Yuan