bug#46364: regression in lm-commentary

[Matt Armstrong]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> I attach the file in question, diredc.el.  Here's the recipe:
>
> 0. emacs -Q
> 1. (require 'lisp-mnt) C-j
> 2. (lm-commentary "/path/to/diredc.el") C-j
>
> In Emacs 27, the resulting string includes leading semicolons and
> indentation.  In Emacs 28, it doesn't.  Version info follows my
> signature.

Thanks Basil. I investigated this too and filed a new bug with the
information below (because I don't know debbugs well). My investigation
belongs here instead.

Lars asked about MELPA. Answer: MELPA is relevant only because in some
cases MELPA code calls `lm-commentary' to form user-facing description
of a package.

The issue is a behavior change in `lm-commentary' that probably deserves
some consideration, because it seems suboptimal in this example. The
behavior change is most likely caused by commit
963a9ffd66cb29f0370e9a4b854dddda242c54a6.

Prior to that commit, and in Emacs 27.1, the function returns the file's
commentary as an verbatim substring of the elisp source, including the
elisp comment characters, commentary headline, etc. E.g. a string like
this (I'll quote the string for email sanity):

> ";;; Commentary:
> ;;
> ;; This package extends and configures `dired' with features found in
> ;; almost all 'file managers', and also some unique features:
> ;;
> ;;   * Resilient dedicated dual-pane frame.
> ;;     * similar look to 'midnight commander'.
> ;;     * intelligent recovery of manually altered frame configuration
> ;;     * exit diredc/dired cleanly and totally
[...]

After that commit, the function returns a "sanitized" version of the
same content, such as:

> "
> This package extends and configures `dired' with features found in
> almost all 'file managers', and also some unique features:
> 
> * Resilient dedicated dual-pane frame.
> * similar look to 'midnight commander'.
> * intelligent recovery of manually altered frame configuration
> * exit diredc/dired cleanly and totally
[...]

It seems `lm-commentary' now strips all leading whitespace from every
line, as a "sanitization" step, and this has the unsatisfying side
effect of ruining any indentation formatting in the original commentary.

We need not go farther than Emacs' own lisp/align.el to see a similar
problem.

In Emacs 27, (lm-commentary "align.el") returns:

> ";;; Commentary:
> 
> ;; This mode allows you to align regions in a context-sensitive fashion.
> ;; The classic use is to align assignments:
> ;;
> ;;    int a = 1;
> ;;    short foo = 2;
> ;;    double blah = 4;
> ;;
> ;; becomes
> ;;
> ;;    int    a    = 1;
> ;;    short  foo  = 2;
> ;;    double blah = 4;
> 
> "

And in Emacs 28 (mainline):

> "This mode allows you to align regions in a context-sensitive fashion.
> The classic use is to align assignments:
> 
> int a = 1;
> short foo = 2;
> double blah = 4;
> 
> becomes
> 
> int    a    = 1;
> short  foo  = 2;
> double blah = 4;"

Perhaps this should be re-thought?