bug#60856: 29.0.60; C-M-f in python-ts-mode does not work properly with docstrings

[Theodor Thornhill]

Hi there!

Apostolis Bessas <apostolis@bessas.me> writes:

> In python-ts-mode executing C-M-f (forward-sexp) when at the beginning
> of a docstring does not go to the end of the docstring (as happens in
> python-mode), but at the end of the file.
>
> However, if I switch to python-mode and then back to python-ts-mode,
> C-M-f works correctly. See the following example:
>
> - Start emacs -Q
> - Execute:
>
> (add-to-list 'auto-mode-alist '("\\.py\\'" . python-ts-mode))
>
> - Create the file test.py with the contents:
>
> """Some docstring.
>
> The docstring has multiple lines.
> """
>
> x = 1
>
> - Go to the first character in the file (beginning of docstring) and
>   press C-M-f. The point is moved to the end of the file (after the line
>   `x = 1`). Instead, the point should have been moved to the end of the
>   docstring (line 4, last '"' character).
>
> - Switch to python-mode and then back to python-ts-mode and repeat the
>   above step. The point is moved to the end of the docstring.
>

Yeah, I can reproduce this as well.  What happens is that python.el
redefines and remaps lots of commands, and doesn't set the functions.
In addition it uses some internal stuff that if I'm not mistaken isn't
really enabled in python-ts-mode.  If you try to apply the supplied
patch, then execute these steps:

1. M-< ;; go to beginning of buffer
2. M-e
3. M-<
4. M-x forward-sentence

You should see the behavior diverges, right?  That's because the
'treesit-major-mode-setup' sets up 'forward-sentence-function' which
python doesn't set.  A fix could be either to set the functions rather
than remapping, or make sure we at least create equivalents for
python-ts-mode.  I think we inherit too much from the python-base-mode,
so it is a little hard to reason about where functionalities come from.
I added the maintainers of treesit.el and python.el to CC for them to
provide some input.

Theo


diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 21d16db287..973e405e8f 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -6710,6 +6710,10 @@ python-ts-mode
                                               "_definition"))
     (setq-local treesit-defun-name-function
                 #'python--treesit-defun-name)
+
+    (setq-local treesit-sentence-type-regexp
+                '"statement")
+
     (treesit-major-mode-setup)
 
     (when python-indent-guess-indent-offset