Re: removing the .IX macro from the ms package in 1.23 breaks old documents

[G. Branden Robinson]

[replying only to the lists since you're clearly subscribed, or
otherwise following them]

Hi Joerg,

At 2024-10-13T12:36:23+0200, joerg van den hoff wrote:
> On 13.10.24 11:24, G. Branden Robinson wrote:
> it seems it would probably be best to just agree to disagree and move
> on, but ...

Where would the fun be in that?

> > At 2024-10-13T11:03:52+0200, joerg van den hoff wrote:
> > > but I really believe the line should never have been deleted in
> > > the first place (I think the working hypothesis should better be
> > > "even if I do not see why it is there and even if it seems
> > > superfluous to me, it is there for a reason"
> > 
> > I think a better hypothesis is "have good automated test coverage",
> > so that you can delete stale code and otherwise refactor.
> 
> no, that is not true, simply because you can't test everything.

Non sequitur.  For any non-trivial system, you can't test everything,
_and_ you should have good automated test coverage.  A system that is
not amenable to worthwhile automated test coverage is not a system at
all, but chaos.

> you removed the .ll -8n as "doing nothing in all my trials"

I didn't test aggressively _enough_, simple as that.  "Sufficient
testing" is a better alternative to "insufficient testing" than "no
testing".

> and if you have no idea why is there in the first place you hardly can
> test reliably whether its deletion harms ;).

Fair.  I understood the ms package less well back then.  I've learned.

> more seriously: test coverage is never 100% (far from it).

It is a vastly greater task to formally verify groff.  To say nothing of
infrastructure for doing so being lacking, and the large resource
demands of contemporary verification systems like Isabelle/HOL or Coq.

This leaves aside the elephantine question of whether GNU troff or its
macro packages are specified well enough to be amenable to formal
verification in the first place.

> removing code simply because it does not affect existing tests,

Straw man.  I didn't remove it "simply" because it didn't have an impact
on the (inadequate) experiments I conducted, but to reduce the amount of
perturbation the macros caused to formatter state.  Formatter state,
while it may sound abstract, is something every user relies upon.

> however comprehensive, sure is not a good idea.

If the `XA` macro had had something something like this in it...

.tr aBcDeFgH

you, as an ms user, would call for its removal whether existing tests
caught the problem or not.

That is of course a contrived example.  The rub is that relatively few
bugs are so obvious (except in software that the developer doesn't even
attempt to use themselves, which is more common than it should be).

Let's consider something subtler.  Add this to end of the `XA`
definition and see if you can spot any difference.

'ti +0.1m

> so the approach "when in doubt, don't touch" seems really the
> preferable one to me.

I didn't have doubt.  I had confidence.  I was wrong.  C'est la vie.

> at least with stuff like this.

Spend more time thinking about the boundaries and parameters of your
application of the foregoing principle.

> > Our test coverage may have been inadequate here; I'll see what I can
> > do about it.
> 
> that's always good, sure.

It's not a luxury; it's a priority.  Sometimes things are too difficult
or too laborious to test, and sometimes there isn't anything _to_ test.
The `IX` macro's removal is an example of the latter, and the `XA` line
length change is a good _counterexample_ of the former.

You can observe what I expect to become the basis of a regression test
for `XA` in Savannah #66328.[2]

> > What is `IX`'s interface on the following ms implementations?  By
> > "interface", I mean not only "what arugments does the macro accept?",
> > but "what output or result can the macro user rely upon?".
> > 
> > 1.  DWB 3.3
> > 2.  Solaris 10
> > 3.  Solaris 11
> > 4.  groff 1.22.4
> 
> I do not know. and I am not sure where this argument should go and
> what should follow from it.

In that case the most salient point is that groff--and ms(7)--has users
that aren't you.  Some of them have documents that were prepared on
other systems, using other _ms_ implementations.  Some of them will be
as fixed and rigid in their expectations as you are.

What's a maintainer to do, then, when faced with one user who says:

"I use makeindex(1), which has supported troff for longer than some of
your have been alive, and it supports up to nine arguments, which it
emits space-separated and with an 'IX: ' prefix."[1]

...and another who says:

"No, the output has no prefix, the arguments are _tab_-delimited, and
you can supply `IX` at most _four_ arguments and that's all it will
output--I use `.am IX` to add logic to handle a fifth argument in a
special way because my document has multiple indexes and I have an AWK
script to read stderr and ensure the index entries end up in the right
place so YOU'RE BREAKING MY DOCUMENT.  _I_ am absolutely right."

Here are the answers to the pop quiz you didn't care about, or were wary
of answering.

1.  DWB 3.3 ms had no `IX` macro at all.  Neither does neatroff ms or
    Plan 9 troff ms.

2.  Solaris 10 ms's `IX` looks like this.

.       \" IX - index words to stderr
.de IX
.tm \\$1\t\\$2\t\\$3\t\\$4 ... \\n(PN
..

3.  Solaris 11 ms's `IX` looks like this.

.\" indexing
.de IX
.tm \\$1\t\\$2\t\\$3\t\\$4 ... \\n[PN]
..

4.  groff 1.22.4 ms's `IX` looks the same as Solaris 11's.

(Solaris 11 ships groff 1.22.2 instead of AT&T troff.)

> my position/experience basically is: groff (and legacy troff before)
> is "user software".

That's not a term with a rigorous definition in my experience, but okay.

> macro packages are there to help "simple" users along.

No.  This is incorrect and ahistorical.

First of all, the only *roff users who can get by without a full-service
macro package (see groff_tmac(5)) at all either prepare only very
simple documents, perhaps of a single page, or are not merely
ill-described as "simple" users, but are *roff experts.  Most users,
simple or otherwise, have a poor command of formatter features like
environments, diversions, and traps.  But you need all of these just for
reasonable footnote support, let alone sophisticated typesetting
applications like drop caps or multiple columnation.

ms(7) is as far along the spectrum toward an "expert mode" full-service
macro package as any I'm aware of.

At the other end, man(7) and mdoc(7) are designed for a limited
application domain, and the former in particular has a sorry history of
coming under attack by ad hoc parsers written by people with little
understanding of *roff.  Thus both the groff and mandoc projects
discourage users of these packages from writing their own macros or
exercising formatter features directly.

me(7) and mm(7) are in between; they try to help the user to a
significant extent but acknowledge that the user may sometimes indulge
what (these days) man(7) and mdoc(7) discourage, hence the idiom of
listing formatter requests that "may be used with impunity".  (I think
this phrase originates with me(7) author Eric Allman and got borrowed by
the authors of later editions of the mm(7) manual [among others]--but I
could have the chronology backwards.)

ms(7) was, and to a large extent has remained, oriented toward relative
*roff experts, like its initial audience at the Bell Labs CSRC.  One of
the reasons the Holmdel Bell Labs site developed mm(7) was, apparently,
because they saw ms(7) as promising too heavy a support burden.  The
research team in Murray Hill weren't in the user support business at all
in any conventional sense.  Their attitude seems to have been, "use any
formatter request you like, and be prepared to cope with the
consequences".  If you got really stuck, Joe was right down the hall in
the early days, Brian in later ones, and Mike the whole time.)

> if there is a macro that was around 25 years ago (.IX), even if never
> well documented, never doing something fancy but only doing something
> helpful to some users some of the time,

It's doing something actively unhelpful for people who are dealing with
diagnostic messages mixed in with the index entries.  In, say, a book of
150 pages or so, the index entries can easily be so numerous that
they'll scroll off the screen.  That is why other groff macro packages
emit index entries and similar to the standard error stream only when a
dedicated register or string is set, as with groff mm's `Ref` and
mom(7)'s `LABEL.REFS`.

Have you ever checked your indexes to see if they had any error message
buried in them?  If you run groff with the `-ww` option, does that
change?

> than there is no good incentive (or none at all...)

I've enumerated several incentives to you; that you discard them all as
unconvincing or simply irrelevant to your personal needs doesn't cause
me to have unsaid them, or alter the needs of others.

> to just delete it 25 years down the road and cause poor users like me
> some headache ;).

As I've said before, I'm not happy to have caused a headache, but groff
is living, maintained software.  There will be changes.  That is why
it has a "NEWS" file.  groff has been in the practice of changing things
in user-visible and -impacting ways for the entirety of its history,
which is a decade longer than the 25+ years you cite in your message.

> in any case, your line of reasoning and justification for deletion of
> .IX seems to boil down to "not official, not documented, not canonical
> part of ms, useless".

I've expounded upon my reasons and justifications at length.  If you
wish to distort or disregard them for the sake of shoring up your
confidence in your own opinion, that's your business.

> my reasoning is "worked for 25+ years (which, let's face it, is quite
> a long time ;))

So did a 32-bit time_t.

> in groff-ms, I found it because documented in some ms-related
> tutorials

If you'd used System V troff you'd have had a different experience with
`IX` and possibly wonder what sort of drug was being smoked by the
authors of the tutorials.  groff has users who aren't you.

> and I deemed it useful and used it.

Terrific!  Include the definition in your documents or inaugurate an
"sjoerg" package that you use to produce them.  This is the sort of
thing ms(7) in particular was _designed_ to support.

> 25 years later current maintainer decides to do away with it. bad."

Worry not--I'm getting the "bad maintainer" message clearly.

> bottom line: no, you will not convince me that your decision is the
> correct one in this respect no matter what no matter what arguments
> you personally consider valid justification

All right.  I know where you stand.

> keeping it does zero harm (maybe beyond hurting you personal feelings
> about source code economy and tidiness ;)),

Economy and tidiness, yes.  Let me suggest that you have some things to
learn about what makes software maintainable.  You could do a lot worse
than to start with the book described here:

https://en.wikipedia.org/wiki/The_Practice_of_Programming

> deleting it does non-zero (however small) harm (has done, matter of
> fact).

Non-zero impact on users, I grant.  One of the ways we know that a field
admits the practice of engineering is that tradeoffs have to be made.
When a solution exists that is a global optimum in all parameters, there
are no such decisions to make.

> but I of course take note of the fact that you disagree and, as a
> maintainer, have the last word (sort of...).

It's not nearly as final as that.  I could die, lose interest and
resign, be replaced by a revolt of my colleagues, the project could be
forked, you could tell your packaging system to "pin" groff at a version
you found satisfactory for your needs, or you could build groff 1.22.4
from source and install it in /usr/local where (with a properly
configured $PATH) it would supersede whatever insanity of mine comes
down the "official" pike.  Likely there are other possibilities.

Or you could define `IX` locally to behave _exactly_ how you want for
all time.  One place you could do so is the "troffrc-end" file.

> so I have to work around your decisions when+where they get into my
> way

I regret that that is sometimes the case.  I generally take pains to
document my rationale when making user-visible changes to groff.
(In truth, I tend to motivate the ones that _aren't_ user-visible, too.)
Where my explanations are inadequate or ill-premised, I seek to amend or
correct them.

But if you've ruled out being persuaded of the worth of a change ("you
will not convince me that your decision is the correct one in this
respect no matter what no matter what arguments you personally consider
valid justification"), then understand that you are offering me no
incentive to make such emendations or corrections, but relying on others
to do so.

> (to the extent that my modest knowledge of groff suffices: I actually
> use groff just as a tool and want to write/format stuff,

I consider that a good description of groff's primary audience.  Our
impasse seems to lie in the fact that groff is used as a tool to
write/format stuff by people other than you, and who may have somewhat
different needs.

Here's an example of another ms user at work:

https://github.com/g-branden-robinson/retypesetting-mathematics

> not play or fight with groff).

I don't mind if groff is a programming environment that is rewarding to
"play" in; play can lead to learning.  But I'm not happy if the user
struggles--that is why I have placed such emphasis on correcting,
revising, and expanding groff's documentation, and on making its
diagnostic messages more informative and comprehensible.

> but this does not change my principle opinion: like with removal of
> the `.ll 8n' line from the .XA definition such removal of (really
> only) apparently stale/useless code risks to (and indeed does:
> exhibits A and B are .IX and .XA) break decade old user documents.
> this is a too high price to pay just for the sake of "cleaner" code:
> better to unnecessarily keep some "noise" in the sources (really stale
> code) than to inadvertently destroy some "signal" (relevant stuff).
> 
> coming back to "agree to disagree" -- let's probably draw a line here:
> at least the `.ll -8n' is back (good)

It isn't, yet.  What you have seen is a Savannah bug report about it.[2]
It was filed anonymously.  ("Who _was_ that masked man?"  Dave Kemper, I
reckon.)  However, as you may have seen by now, I did decide to restore
it.  I had to convince _myself_.  (Tadziu's messages didn't steer me one
way or the other, since he pointed out the flexibility of the macros'
design.)  I did so by consulting historical implementations.

If you care only about what you want, this is a happy outcome for you.
But then consider that you played less of a role than you could have in
bringing it about.

> and you seem to refuse to restore .IX.
> (unfortunate for affected users (>= 1)).

And I've said why.  For me personally, the riot of spew to stderr when
the macro is used is the most compelling reason.  I had forgotten about
it because none of groff's own documents use `IX`, and neither do any of
the Seventh Edition Unix Programmer's Manual documents to which I have
access (and whose rendering I try not to damage).

But when I threw some `IX` calls into a sample document to format with
groff 1.22.4, I got a rude reminder.

> that's that, then...

You win some, you lose some.

Regards,
Branden

[1] "man makeindex" if you don't believe me.  Or, if you don't have that
    installed:

    https://linux.die.net/man/1/makeindex

[2] https://savannah.gnu.org/bugs/?66328
[3] https://www.zdnet.com/article/eric-raymond-how-ill-spend-my-millions/

signature.asc
Description: PGP signature