Re: removing the .IX macro from the ms package in 1.23 breaks old documents

[G. Branden Robinson]

[follow-ups set to groff@gnu, a discussion list]

Hi Joerg,

At 2024-10-02T18:12:06+0200, joerg van den hoff wrote:
> I was not aware of this change (not following groff development
> closely) and it took me quite a bit of time today to find the root
> course of why some newly compiled old documents of mine do not contain
> an index any more.

I am unhappy to learn that you experienced frustration with groff
1.23.0.

> the reasoning in the release notes "This undocumented 4.2BSD ms
> extension was similarly undocumented by groff mm and ms. No documents
> applying it are attested." seems not to be sound:

Let me begin my defense of the decision right away.  "Soundness" refers
to the conjunction of two properties of an argument: (1) the truth of
its premises, and (2) the absence of fallacy from its syllogism or other
application of inference rules under the logical system in use.

I don't think soundness applies here, because the language you quote
does not make an argument.  It makes assertions of (purported) fact.

Both were true to the best of my knowledge at the time I made them--the
first still is, and, up until the point I read your email, the second
was as well.  I collect ms(7) documents that I find online.  (It's not
easy to _search_ fruitfully for them thanks to collisions with a common
abbreviation for "Microsoft", which also has a measurable presence in
the document formatting business.)

> 1.
> existence of that macro was sure not a secret

*roff has no means of "information hiding", as object-oriented languages
term it.  The existence of any macro in a package is visible to anyone
who inspects its code, or uses the `pm` request to dump the names of
defined macros/strings/diversions.

If the non-secrecy of macro's existence establishes a contract with its
users, then no macro can ever be withdrawn or altered; since there is no
reason not to apply this principle to registers as well, that means that
no macro package can ever be permitted the luxury of "internals".

To accept such a restriction is to surrender groff to stagnation.  While
I am aware of at least one person for whom that situation is a
preference, I claim that the same can be achieved by never upgrading
groff from the version one finds satisfactory.  In the meantime, those
who _don't_ want groff to stagnate can benefit from its development.

> and if it really was undocumented in the official docs,

I invite you to consult the references at the end of the groff_ms(7) man
page and test my claim for yourself.  Lesk and Tuthill's papers are
widely available.

> that could have been fixed instead of just deleting the functionality.

I considered this.  When contemplating how I'd write a paragraph
specifying its behavior, I came to regard the macro as too simple and
inflexible to be worth promoting to "official" status.

> 2.  regarding "no documents": well, maybe no "official" ones, but mine
> sure did and those of other users probably too. :).

This is the best argument against the change.  Screwing up the rendering
of real documents that people write and read is an anti-goal of mine.
But it's not the only objective I have for groff development.  Some
documents depend on (or work around) buggy behavior, and I think groff
needs the freedom to fix its own bugs.

---begin digression---

For an example of a case where I choked down bug-compatibility despite
my conviction (which remains) that it was the inferior choice on grounds
of API quality and flexibility for users, see
<https://lists.gnu.org/archive/html/groff/2022-06/msg00026.html>.

Warning: it's a long post in a long thread and it gets very deep into
the weeds of _undocumented_ semantics.  Ultimately, the problem, I feel,
was a design flaw in the `it` request.  Like a couple of other
nroff/troff requests (`ce` and `ul`), it takes a count of "productive[1]
input lines".  Unlike those two, it doesn't suppress incrementation of
the count upon encountering a `\c` escape sequence on a productive input
line.  I cannot think of any reason for this asymmetry.  I assume it was
an oversight.

The gist of this case is that I had concrete evidence before me of man
pages that rendered correctly under Seventh Edition Unix troff (and
older groff versions, like 1.22.3) that would not with this change to
groff.  To be frank, available evidence suggests that the impact would
have been limited--David Korn apparently located these same weeds and
found a furry reward therein, but few other man page authors proved as
resourceful (or as attentive to points of typographical detail).  On the
other hand, ksh has seen many forks and reimplementations and so it
seemed likely that his sly trick would get copied around among that
shell's man pages--even if no others.

Exhibit:

  .SS Field Splitting.
  After parameter expansion and command substitution,
  the results of substitutions are scanned for the field separator
  characters (those found in
  .SM
  .B IFS\^\c
  )
  and split into distinct fields where such characters are found.

The `B` macro, like `SM`, uses the `it` request. The input trap is for
the benefit of callers who give the macro _no_ arguments so that it
affects the next text line.

Because `it` instead of `itc` (a GNU troff extension) is used, and
because the `B` macro formats its own arguments as a text line when any
are present, in the foregoing specimen the trap established by `B`
springs immediately, ending the changes to font face and size, and
setting the closing parenthesis in roman at normal size.

groff 1.22.4 had switched `B` (and `I`, and `SM`) to use `itc` instead
of `it`, so it mis-rendered the closing parenthesis as smaller and bold.
This was the "right" thing to do (because `it` itself should have been
behaving as `itc` does all along), but, acknowledging that `it`'s
semantics were baked into the operation of _documented_ (and, insofar as
`B` and `I` go, widely used) man(7) macros in place since 1979, I felt
honor-bound to revert the change.

Maybe I shouldn't have.  Maybe one of the reasons man page authors have
stuck so fervently to horribly ugly font selection escape sequences in
their man(7) documents is that it became folk wisdom to do so, to avoid
the problems of poor macro composability thanks to the irritating
inconsistency in `it`'s behavior.  `\f` escape sequences did just what
you wanted and no more, whereas `B`, `I`, and `SM` didn't compose in a
perfectly predictable manner with `\c`, and there are times when `\c` is
terribly useful and even indispensable if you wish to avoid writing your
own macros, a subject to which I return below.

London and Reiser got their hands deeply into troff in 1978 for the
Unix/32V port.  Kernighan got his hands even deeper into it a year or
two later to refactor it for device independence.  I really wish they'd
fixed `it` to count lines just like `ce` and `ul` while they were there.

Alternatively, McIlroy could have avoided use of input traps in the
man(7) package in the first place.  :)

But none of those eventuated, so here we are.

---end digression---

To get back to ms(7), I have considered withdrawing support for another
undocumented macro, `RT`, for similar reasons as `IX`.  Do you use that
one?

> I have now reimplemented the macro as a fix/workaround (it really was
> a one-line macro in the orignal ms package anyway, I guess)

Yes.  This is the correct solution.  In hindsight I should have added
the macro definition I deleted to the release notes so that affected
users wouldn't have to hunt it up.  Neglecting to do so was a poor
choice on my part.  I apologize for that.  I can't retroactively fix the
release announcement, but I can update the NEWS file (in groff's source
distribution) upon which that announcement was based; possibly some ms
users will be spared the larger part of the frustration you suffered
today.

> but my simple question: why was it deemed necessary to delete the
> macro in the first place. I cannot see any benefit from doing this
> (except ca. 3 LOC less) but it sure did cost me some lifetime today to
> find out what is going on ...

"Necessary" is a strong word here.  It wasn't "necessary".  It was a
choice.  I believe I laid out the reasons for my choice at length above.

> ps: yes, I know that IX is not a full fledged indexing facility and
> that there are superior solutions including makeindex. but it sure is
> easy and sufficient for many purposes.

Certainly.  By the same token, the cost to your documents of a one-line
definition is not high.  (The cost was incurred in not initially being
aware of the change, and the bug hunt that followed.)

And if you find that you need a fifth argument to `IX`, or to change the
page numbering style under certain conditions, you will have more
flexibility to do so.

My understanding of ms(7)'s audience and user base is that it is a
package for *roff users who are willing to develop some sophistication
with macro programming--meaning, you _may_ not need to write your own
macros, but where you require a facility that the package doesn't offer,
you're willing to start.  mm(7) can be better for users who are more
wary of macro programming--it provides numerous conveniences for those
who are--and historical forces (the many divergent implementations of
"man2html") have pushed man(7) in the opposite direction.

Do you feel that characterization of audience describes you accurately?

Regards,
Branden

[1] AT&T troff documentation never bothered to say "productive".  But
    input lines consisting of the empty request, register definitions
    like `.nr a 1`, or even a 1,000-line macro definition, don't count
    toward the limit established by the request.  If I sound cross with
    Joe Ossanna and the Bell Labs CSRC for not fixing this wart in the
    1970s when they could have done it at relatively low cost--I am!  I
    liken it to Stuart Feldman's decision not to change the use of
    invisible tabs for the inset of make(1) recipes despite knowing it
    had problems that would hurt more over time, because he had a whole
    12 users or so.  Well, I'd probably have a stroke too if I glanced
    up to find Ken Thompson looming in my office doorway with a scowl.

    I also understand giving weight to those users who actually exist
    and will talk to you.  Neither Ossanna nor Feldman likely
    anticipated by how many orders of magnitude their user bases would
    grow.  Nor how each of their projects would come to be stewarded by
    others.

signature.asc
Description: PGP signature