bug#62951: 29.0.90; c-ts-mode: Incorrect fontification due to FOR_EACH_TAIL_SAFE

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 22 Apr 2023 17:28:25 -0700
> Cc: 62951@debbugs.gnu.org
> 
> >> I’m aware of this issue, but the truth is there isn’t a good solution to
> >> it. We need to recognize FOR_EACH_TAIL_SAFE (not hard) and fix arbitrary
> >> code after it (hard). In this case it’s a if statement, with macro calls
> >> and AND operation in it’s condition, it’s already three things we need
> >> to recognize and somehow handle. It can also be a for loop, a switch
> >> case, a function call, a while loop. If we want to fix FOR_EACH_TAIL we
> >> would need to handle every possible thing, at that point we might as
> >> well have wrote a parser :-)
> > 
> > Sorry, I don't understand the difficulties.  The body of FOR_EACH_TAIL
> > and a few similar macros we use could be on of the following:
> > 
> >  . a single simple statement
> >  . an 'if' clause
> >  . a 'while' loop
> >  . a 'do' loop
> >  . a 'for' loop
> >  . a brace-delimited block (this one already works, AFAICS, so we
> >    perhaps need not anything about it)
> > 
> > (In practice, only the first 2 and the last one are used, AFAICS.)
> > 
> > Doesn't tree-sitter tell us enough to figure out which of the above is
> > in the body?  If so, why would we need to write a full parser?
> 
> Ok, the full parser part is a bit exaggeration :-) But my point is it’s a lot 
> of dirty code for a subpar result. Take
> 
> if (NILP (XCDR (tail)) && STRINGP (XCAR (tail)))
> 
> for example, it’s parsed as a function definition and all the tokens in the 
> condition are parsed as weird things like macro declarator, error, function 
> declarator, type, etc. And if the condition changes to something else, say it 
> has another layer of function call, it’ll parse differently.

But the top-level construct is 'if', no?  Isn't that enough?

Also, can we detect the FOR_EACH_TAIL etc. macros themselves, and then
treat their body specially?

> > Please understand: good support for editing Emacs C sources is from my
> > POV imperative for c-ts-mode to gain traction.  One of my gripes about
> > CC Mode was insufficient support for our macro system and for various
> > GCC attributes; that improved recently to some extend, but not enough,
> > and at a price of introducing ugly lists of strings that cause trouble
> > when used in file-local variables.  We must do better in c-ts-mode!
> > 
> > So please make an effort of providing reasonable built-in solutions
> > for these idiosyncrasies of the Emacs C sources, conditioned on the
> > new variable c-ts-mode-emacs-sources-support, at least for those of
> > them that are used widely enough.  It is very important.
> 
> What do you think of extending the parser to support these macros instead? 
> (So we fork tree-sitter-c.)

This goes against the purpose of using tree-sitter and its grammars.
I don't think we should maintain and develop language grammars,
especially since the production of the C sources from them needs
non-trivial system resources and additional tools.

Maybe coming up with a way of extending the C grammar in some more
general way, and then submitting the changes to the tree-sitter-c
developers for inclusion would be OK.

But I very much hope that we could support these macros at a lower
cost, by some tailored Lisp.  Please give it a try.  Even if the
result works only for the cases we actually use in our sources, it
might be "good enough" for us.