Re: removing the .IX macro from the ms package in 1.23 breaks old documents

[joerg van den hoff]

On 02.10.24 20:29, G. Branden Robinson wrote:
> [follow-ups set to groff@gnu, a discussion list]
>
> Hi Joerg,

hi branden,

thanks for bothering to reply so thoroughly. I have interspersed a few comments below but up front 
just this: I principally regard changes breaking backward compatibility in software like troff/groff 
as really undesirable. I would never have expected elimination of a macro definition from an ancient 
macro package like ms in a new groff release (that's why it took me non-negligible time: I looked 
everywhere (especially in my scripts massaging the output (resorting the dit output such that TOC is 
moved to the right place and stuff like that) before I looked into s.tmac to inspect the IX 
definition for clues -- which was no longer there...).

but I do not want to get into a lengthy argument about this issue (if only for the reason that 
re-implementing this trivial macro is ... trivial :)). you maintain it, you decide. if you would 
reconsider that decision, I would welcome it, if not, so be it.

> At 2024-10-02T18:12:06+0200, joerg van den hoff wrote:
> > I was not aware of this change (not following groff development
> > closely) and it took me quite a bit of time today to find the root
> > course of why some newly compiled old documents of mine do not contain
> > an index any more.
>
> I am unhappy to learn that you experienced frustration with groff
> 1.23.0.
>
> > the reasoning in the release notes "This undocumented 4.2BSD ms
> > extension was similarly undocumented by groff mm and ms. No documents
> > applying it are attested." seems not to be sound:
>
> Let me begin my defense of the decision right away.  "Soundness" refers
> to the conjunction of two properties of an argument: (1) the truth of
> its premises, and (2) the absence of fallacy from its syllogism or other
> application of inference rules under the logical system in use.
>
> I don't think soundness applies here, because the language you quote
> does not make an argument.  It makes assertions of (purported) fact.
>
> Both were true to the best of my knowledge at the time I made them--the

I do not doubt that you were under that impression at the time but this does not make it "true" 
which ideally is an absolute category, independent of one's own knowledge :). "not sound" referred 
to that circumstance: I read the release note as "because A and B are given it is justified to do 
C". and the stated premises do not hold up to closer scrutiny...

> first still is, and, up until the point I read your email, the second

undocumented via manpages might very well be. I do not recall how I learned about IX approx 30 years 
ago. maybe SunOS did have a ms manpage mentioning it... but as said in my first mail: if there is a 
useful undocumented feature the solution obviously should be to document the feature rather than 
removing it :). also a short search brings up this document

https://www.mail-archive.com/ion-general@lists.berlios.de/msg01838/ms.pdf

(sure available elsewhere, too) which indeed at least mentions the IX macro. so it _was_ "user 
visible" for groff 1.16 around 2004 (even if not "officially" documented, maybe). but as said "not 
documented" would not justify removal in the first place).

> was as well.  I collect ms(7) documents that I find online.  (It's not
> easy to _search_ fruitfully for them thanks to collisions with a common
> abbreviation for "Microsoft", which also has a measurable presence in
> the document formatting business.)

yes, I have been told so :).

> > 1.
> > existence of that macro was sure not a secret
>
> *roff has no means of "information hiding", as object-oriented languages
> term it.  The existence of any macro in a package is visible to anyone
> who inspects its code, or uses the `pm` request to dump the names of
> defined macros/strings/diversions.
>
> If the non-secrecy of macro's existence establishes a contract with its
> users, then no macro can ever be withdrawn or altered; since there is no
> reason not to apply this principle to registers as well, that means that
> no macro package can ever be permitted the luxury of "internals". > To accept 
> such a restriction is to surrender groff to stagnation.  While
> I am aware of at least one person for whom that situation is a
> preference, I claim that the same can be achieved by never upgrading
> groff from the version one finds satisfactory.  In the meantime, those
> who _don't_ want groff to stagnate can benefit from its development.

as said, I really do not really want to argue too much about the issue, but...
I posit that .IX is a "user macro", definitely not intended for package 
"internal" use.

the question in my view is whether it is a good idea to delete an apparently "unimportant", 
"useless", "undocumented" user macro from a central groff macro package like ms.

so your line of thought in my view does at least not apply to .IX.

>
> > and if it really was undocumented in the official docs,
>
> I invite you to consult the references at the end of the groff_ms(7) man
> page and test my claim for yourself.  Lesk and Tuthill's papers are
> widely available.

s. above: there are (inofficial) documents listing it.

> > that could have been fixed instead of just deleting the functionality.
>
> I considered this.  When contemplating how I'd write a paragraph
> specifying its behavior, I came to regard the macro as too simple and
> inflexible to be worth promoting to "official" status.

the question remains: what harm is done if it is kept vs. how much headache does removal cause in 
the field.

> > 2.  regarding "no documents": well, maybe no "official" ones, but mine
> > sure did and those of other users probably too. :).
>
> This is the best argument against the change.  Screwing up the rendering
> of real documents that people write and read is an anti-goal of mine.

good! keep that goal :).

> But it's not the only objective I have for groff development.  Some
> documents depend on (or work around) buggy behavior, and I think groff
> needs the freedom to fix its own bugs.

the presence of .IX in ms is not a bug...

> ---begin digression---
>
> For an example of a case where I choked down bug-compatibility despite
> my conviction (which remains) that it was the inferior choice on grounds
> of API quality and flexibility for users, see
> <https://lists.gnu.org/archive/html/groff/2022-06/msg00026.html>.
>
> Warning: it's a long post in a long thread and it gets very deep into
> the weeds of _undocumented_ semantics.  Ultimately, the problem, I feel,
> was a design flaw in the `it` request.  Like a couple of other
> nroff/troff requests (`ce` and `ul`), it takes a count of "productive[1]
> input lines".  Unlike those two, it doesn't suppress incrementation of
> the count upon encountering a `\c` escape sequence on a productive input
> line.  I cannot think of any reason for this asymmetry.  I assume it was
> an oversight.
>
> The gist of this case is that I had concrete evidence before me of man
> pages that rendered correctly under Seventh Edition Unix troff (and
> older groff versions, like 1.22.3) that would not with this change to
> groff.  To be frank, available evidence suggests that the impact would
> have been limited--David Korn apparently located these same weeds and
> found a furry reward therein, but few other man page authors proved as
> resourceful (or as attentive to points of typographical detail).  On the
> other hand, ksh has seen many forks and reimplementations and so it

in fact ksh93 (the new korn shell maintained by Korn until about 10 years ago) very fortunately now 
has a very dedicated and competent maintainer

https://github.com/ksh93/ksh

(formally a fork from the now frozen official ATT repo)

who goes to extreme length to not introduce breaking changes -- which is a very 
important thing...

> seemed likely that his sly trick would get copied around among that
> shell's man pages--even if no others.
>
> Exhibit:
>
>    .SS Field Splitting.
>    After parameter expansion and command substitution,
>    the results of substitutions are scanned for the field separator
>    characters (those found in
>    .SM
>    .B IFS\^\c
>    )
>    and split into distinct fields where such characters are found.
>
> The `B` macro, like `SM`, uses the `it` request. The input trap is for
> the benefit of callers who give the macro _no_ arguments so that it
> affects the next text line.
>
> Because `it` instead of `itc` (a GNU troff extension) is used, and
> because the `B` macro formats its own arguments as a text line when any
> are present, in the foregoing specimen the trap established by `B`
> springs immediately, ending the changes to font face and size, and
> setting the closing parenthesis in roman at normal size.
>
> groff 1.22.4 had switched `B` (and `I`, and `SM`) to use `itc` instead
> of `it`, so it mis-rendered the closing parenthesis as smaller and bold.
> This was the "right" thing to do (because `it` itself should have been
> behaving as `itc` does all along), but, acknowledging that `it`'s
> semantics were baked into the operation of _documented_ (and, insofar as
> `B` and `I` go, widely used) man(7) macros in place since 1979, I felt
> honor-bound to revert the change.
>
> Maybe I shouldn't have.  Maybe one of the reasons man page authors have
> stuck so fervently to horribly ugly font selection escape sequences in
> their man(7) documents is that it became folk wisdom to do so, to avoid
> the problems of poor macro composability thanks to the irritating
> inconsistency in `it`'s behavior.  `\f` escape sequences did just what
> you wanted and no more, whereas `B`, `I`, and `SM` didn't compose in a
> perfectly predictable manner with `\c`, and there are times when `\c` is
> terribly useful and even indispensable if you wish to avoid writing your
> own macros, a subject to which I return below.
>
> London and Reiser got their hands deeply into troff in 1978 for the
> Unix/32V port.  Kernighan got his hands even deeper into it a year or
> two later to refactor it for device independence.  I really wish they'd
> fixed `it` to count lines just like `ce` and `ul` while they were there.
>
> Alternatively, McIlroy could have avoided use of input traps in the
> man(7) package in the first place.  :)
>
> But none of those eventuated, so here we are.
>
> ---end digression---

will have to read that more thoroughly later. bookmarked.

> To get back to ms(7), I have considered withdrawing support for another
> undocumented macro, `RT`, for similar reasons as `IX`.  Do you use that
> one?

not that I can recall. I also do not find a definition in s.tmac?

> > I have now reimplemented the macro as a fix/workaround (it really was
> > a one-line macro in the orignal ms package anyway, I guess)
>
> Yes.  This is the correct solution.  In hindsight I should have added
> the macro definition I deleted to the release notes so that affected
> users wouldn't have to hunt it up.  Neglecting to do so was a poor

yes that would be principally desirable. but would not have helped me, since 
looking at the release
note was the very last thing I did (after looking in vain for .IX in s.tmac...

> choice on my part.  I apologize for that.  I can't retroactively fix the
> release announcement, but I can update the NEWS file (in groff's source
> distribution) upon which that announcement was based; possibly some ms
> users will be spared the larger part of the frustration you suffered
> today.

hopefully. but as said, the real problem is that "nobody expects the spanish 
inquisition"
https://en.wikipedia.org/wiki/The_Spanish_Inquisition_(Monty_Python)

not many people will think immediately "oh maybe the macro is no longer part of the package" if a 
hickup occurs. not with ms being functionally unmodified around for, what? >40 years or so.

> > but my simple question: why was it deemed necessary to delete the
> > macro in the first place. I cannot see any benefit from doing this
> > (except ca. 3 LOC less) but it sure did cost me some lifetime today to
> > find out what is going on ...
>
> "Necessary" is a strong word here.  It wasn't "necessary".  It was a
> choice.  I believe I laid out the reasons for my choice at length above.

your general approach/attitude, yes. why the .IX macro? I am not sure, really, beyond that you 
personally do not find it useful (and I also can admit to it's limited (but non-zero) usefulness ;)).

> > ps: yes, I know that IX is not a full fledged indexing facility and
> > that there are superior solutions including makeindex. but it sure is
> > easy and sufficient for many purposes.
>
> Certainly.  By the same token, the cost to your documents of a one-line
> definition is not high.  (The cost was incurred in not initially being
> aware of the change, and the bug hunt that followed.)

agreed. in this special case, yes. not much harm done (apart from time burning). but as a general 
approach I really would prefer an extremely conservative approach. coming back to ksh: if you have 
time at your hand look at what happened with ksh a couple of years ago when the official(!) ATT repo
was still under control of a maintainer who was intent on modifying ksh beyond recognition 
("improving" and "bug fixing" in his view :)). the story can be found in the ATT ksh github repo in 
assorted threads/issues. in my personal view ksh (and any ksh user) was extremely fortunate that a) 
the ATT team came to a decision to freeze the repo after rolling back to ksh93u+ (the last official 
ATT ksh93 release) and b) https://github.com/ksh93/ksh happened afterwards. otherwise ksh would have 
been done for I am sure.

of course I honestly by *no means* imply that the situation with groff is even remotely comparable. 
but the ksh story is a cautionary tale what happens if backward compatibility is considered to be of 
secondary relevance and things that used to work for a long time start to break...

> And if you find that you need a fifth argument to `IX`, or to change the
> page numbering style under certain conditions, you will have more
> flexibility to do so.

yes, sure. all true. but no reason to remove the thing in the first place. of course I could 
redefine it. as it stands, it just was good enough for my modest means more or less.

> My understanding of ms(7)'s audience and user base is that it is a
> package for *roff users who are willing to develop some sophistication
> with macro programming--meaning, you _may_ not need to write your own
> macros, but where you require a facility that the package doesn't offer,
> you're willing to start.  mm(7) can be better for users who are more
> wary of macro programming--it provides numerous conveniences for those
> who are--and historical forces (the many divergent implementations of
> "man2html") have pushed man(7) in the opposite direction.
>
> Do you feel that characterization of audience describes you accurately?

quite so (regarding ms at least -- mm I do not know anything about). still, I 
really prefer to
keep "extensions" beyond existing functionality in ms at a minimum as far as 
possible.

_and_ I would hope for _keeping_ all existing functionality provided in such a 
"legacy package" and
avoid any break of backward compatibility without very good reasons.

best,

joerg

> Regards,
> Branden
>
> [1] AT&T troff documentation never bothered to say "productive".  But
>      input lines consisting of the empty request, register definitions
>      like `.nr a 1`, or even a 1,000-line macro definition, don't count
>      toward the limit established by the request.  If I sound cross with
>      Joe Ossanna and the Bell Labs CSRC for not fixing this wart in the
>      1970s when they could have done it at relatively low cost--I am!  I
>      liken it to Stuart Feldman's decision not to change the use of
>      invisible tabs for the inset of make(1) recipes despite knowing it
>      had problems that would hurt more over time, because he had a whole
>      12 users or so.  Well, I'd probably have a stroke too if I glanced
>      up to find Ken Thompson looming in my office doorway with a scowl.

:)

>      I also understand giving weight to those users who actually exist
>      and will talk to you.  Neither Ossanna nor Feldman likely
>      anticipated by how many orders of magnitude their user bases would
>      grow.  Nor how each of their projects would come to be stewarded by
>      others.