Re: Creating a paradigm for leveraging Tree Sitter's power

[Theodor Thornhill]

On 24 December 2022 10:09:17 CET, Yuan Fu <casouri@gmail.com> wrote:
>
>
>> On Dec 23, 2022, at 5:32 PM, Perry Smith <pedz@easesoftware.com> wrote:
>> 
>> I've seen on this list talk about "adding" Tree Sitter concepts to
>> such things as mark-defun and other existing Emacs commands.
>> 
>> I've just spent a few days writing rtsn-mark-method that I intend on
>> adding to ruby-ts-mode which implements all the features of mark-defun
>> but I did it from scratch.  This is mostly out of ignorance on how to
>> leverage existing Emacs features and facilities.
>> 
>> One more concrete reason is mark-defun will include the comments
>> before the defun.  I wanted the same for mark-method but (as far as I
>> can tell) the hooks back into mark-defun and its associated routines
>> are simple regular expressions and that, to me, walks away form the
>> power that Tree Sitter is providing.
>> 
>> I've got rtsn-mark-method working but I plan to rework it over the
>> next few days.  The "arg" as well as the "interactive" features of
>> mark-defun which I also put into rtsn-mark-method I believe can be
>> pulled out to a wrapper routine.  There is actually quite a few
>> features in the code that I did not know about.  For example, if
>> mark-defun is called with -1 (or -N) it marks N back.  Ok.  But now if
>> it is called again with no argument, mark-defun knows to go backwards
>> to add in the previous defun.  The same is true in the forward
>> direction as well.  (Although it does have a subtle bug in my opinion
>> but I digress.)
>> 
>> To repeat myself, I believe these higher level features could be
>> separated into a wrapper function so that all that would be needed for
>> the language specific piece is a routine that would be passed a point
>> and a direction.
>
>I think it makes a lot of sense.
>
>> 
>> I'll call this the "primitive routine".  The routine would be
>> responsible for returning a beginning and end (in a cons cell) and it
>> would be the routine's responsibility to make sure that the beginning
>> and end lie after (in the forward case) or before (in the backward
>> case) the point that is passed in.
>
>You mean beginning and end of (symbol | string | statement | …)?
>
>From my experience implementing defun navigation for tree-sitter, it might be 
>more helpful to return three ranges: the thing before point, the thing at 
>point, and the thing after point, and either one could be nil if there  
>doesn’t exist one. For nested things it can be prev-sibling, parent, 
>next-sibling instead. The point is that the user can move back and forward and 
>make decisions easily with this “field of view”.
>
>> 
>> The wrapper routine would then know how to properly adjust the mark
>> and point, execute counts, add to regions, catch errors, etc.
>> 
>> Broadening the picture: navigation, transpose, and other Emacs
>> commands could likewise call the same primitive routine to provide
>> transpose-method, forward-method, etc.  And, broadening the picture
>> once more: primitive routines could be written not just for methods
>> (defun) but also for statements, arguments, expressions, classes, etc.
>> In all cases, the primitive routine would be relatively simple.  It
>> would be given a point and return a ( begin . end ) cons cell leaving
>> all the harder work of expanding the region, remembering the
>> direction, etc to the wrapper routines.
>> 
>> To be clear, there would be a wrapper routine for mark, a wrapper for
>> forward, transpose, etc.  We would end up with the classic:
>> 
>> For A + B number of routines we would have A x B number of commands.
>> 
>> Does this strike others as a good idea or insanity?
>
>Yeah it sounds pretty good. Probably no one needs/wants all the A x B 
>combinations, but the framework that allows this miss and match is good, since 
>it allows us easily create the few combinations we care about. We need more 
>brains thinking about it and more experiments to fully flesh it out.
>
>We also had a related discussion in another thread “Plug treesit.el into other 
>emacs constructs”.
>
>Yuan

Yeah. One shortcoming of tree-sitter imo is that the parser author decides what 
the nodes are named. So I think we need to create a framework so that every 
mode can map over ast-names to Emacs concepts. The goal must be for the normal 
Emacs things to require little to no changes, but get the benefits from treesit.

I think we should just start doing that immediately on the master branch and 
allow for "big" changes going forward. We should settle on something good for 
Emacs 30, hopefully.

I'm a little worried we feel we need "complete" proposals too soon.  

Let's get all good ideas on the table, implemented and installed, then we can 
consolidate after we discover pain points etc.

I'm working on changing the forward/backward thing and transpose. Not only for 
tree-sitter, but for others as well :)

What do you think?

Happy holidays!

Theo