bug#60127: treesit-end-of-defun: possible bug with clojure grammar

[Yuan Fu]

Danny Freeman <Danny@dfreeman.email> writes:

> Hello,
>
> I have been running into what I believe is a bug with
> treesit-end-of-defun while developing clojure-ts-mode. There are some
> positions in clojure buffers where treesit-end-of-defun will jump to the
> end of the buffer.
>
> Steps to reproduce:
>
> Checkout the emacs-29 branch. I have built mine with commit
> b01d0246d71a7a3fd92b2864a3c0c0bc9367ee0b and tree-sitter version 0.20.7
>
> Clone my clojure-mode fork:
> https://github.com/dannyfreeman/clojure-mode
>
> check out the `end-of-defun-bug` branch (master works too, but I added
> a helpful debug message in this branch)
>
> Clone tree-sitter-module repo:
> https://github.com/casouri/tree-sitter-module
>
> Apply the patch `tree-sitter-module-clojure-support.patch` located in
> clojure-mode repo to the casouri/tree-sitter-module repo.
>
> Build the treesitter parser for clojure with
>
> $ ./build clojure
>
> Start emacs
>
> $ emacs -q
>
> Evaluate the following, replacing the paths with what is relevant to
> your system
>
> ```
> (progn
>   (setq treesit-extra-load-path '("~/path/to/tree-sitter-module/dist"))
>   (add-to-list 'load-path "~/path/to/clojure-mode/")
>   (find-file "~/path/to/clojure-mode/clojure-ts-mode.el")
>   (eval-buffer)
>   (find-file "~/path/to/clojure-mode/test_end_of_defun.clj"))
> ```
>
>
> Once you have opened test_end_of_defun.clj you can see the problem by
> pressing
>
> M-<
> C-M-e
> C-M-e
>
> You should see the cursor move from
>
> ```
> |(def w 1)
>
> (def x 2)
>
> (def y 3)
>
> (def skip-to-here? 4)
> ```
>
> to
>
> ```
> (def w 1)
> |
> (def x 2)
>
> (def y 3)
>
> (def skip-to-here? 4)
> ```
>
>
> then it unexpectedly jumps to the end of the buffer
>
> ```
> (def w 1)
>
> (def x 2)
>
> (def y 3)
>
> (def skip-to-here? 4)
> |
> ```
>
>
> The message buffer shows that when the point is between the forms
>
> ```
> (def w 1)
> |
> (def x 2)
> ```
>
>
> that `treesit-end-of-defun` thinks the current node is
> `(def skip-to-here? 4)`, which does not seem right.
>
> When the cursor is at other points in this buffer, this problem doesn't
> seem to occur. For instance, between the (def x 2) and (def y 3) forms,
> this doesn't happen.
>
> I can also see this happening in the `test.clj` file of the clojure-mode
> repo in a different spot but there is a lot more going on in that file.
> (hit C-M-e in it until the problem occurs if you are curious).
>
> I have the following vars relevant vars set in clojure-ts-mode.
> Different combinations of them yield the same results.
>
> ```
> (setq-local treesit-defun-prefer-top-level t
>             treesit-defun-tactic 'top-level
>             treesit-defun-type-regexp (cons (rx (or "list_lit" "vec_lit" 
> "map_lit"))
>                                             (lambda (node)
>                                               (message "Node: %s" 
> (treesit-node-text node t))
>                                               t)))
> ```
>
>
> The clojure code in question produces an error free parse tree. I can
> see it with `treesit-explore-mode` and by running the file through
> `tree-sitter parse`, which gives the following parse tree:
>
> ```
> (source [0, 0] - [7, 0]
>   (list_lit [0, 0] - [0, 9]
>     value: (sym_lit [0, 1] - [0, 4]
>       name: (sym_name [0, 1] - [0, 4]))
>     value: (sym_lit [0, 5] - [0, 6]
>       name: (sym_name [0, 5] - [0, 6]))
>     value: (num_lit [0, 7] - [0, 8]))
>   (list_lit [2, 0] - [2, 9]
>     value: (sym_lit [2, 1] - [2, 4]
>       name: (sym_name [2, 1] - [2, 4]))
>     value: (sym_lit [2, 5] - [2, 6]
>       name: (sym_name [2, 5] - [2, 6]))
>     value: (num_lit [2, 7] - [2, 8]))
>   (list_lit [4, 0] - [4, 9]
>     value: (sym_lit [4, 1] - [4, 4]
>       name: (sym_name [4, 1] - [4, 4]))
>     value: (sym_lit [4, 5] - [4, 6]
>       name: (sym_name [4, 5] - [4, 6]))
>     value: (num_lit [4, 7] - [4, 8]))
>   (list_lit [6, 0] - [6, 21]
>     value: (sym_lit [6, 1] - [6, 4]
>       name: (sym_name [6, 1] - [6, 4]))
>     value: (sym_lit [6, 5] - [6, 18]
>       name: (sym_name [6, 5] - [6, 18]))
>     value: (num_lit [6, 19] - [6, 20])))
> ```
>
> Any help or advice here is appreciated.
>
> Thank you,

Thank you very much for this detailed report! This seems to be some
problem with either tree-sitter or tree-sitter-clojure, where
treesit-node-first-child-for-pos gives wrong result. We’ve had similar
problem with bash before[1].

At this point there aren’t much we can do in Emacs. If I find some time
I’ll make a reproduce recipe and submit an issue on tree-sitter-clojure.

[1] https://github.com/tree-sitter/tree-sitter-bash/issues/139

Yuan