Re: Man page customization after 1.23?

[G. Branden Robinson]

Hi Ronan,

At 2023-07-08T21:03:03+0000, Ronan Pigott wrote:
[LESS_TERMCAP_*]
> I wrote:
> > Today I learned such a feature exists. Despite Mark Nudelman having had
> > up to 39 years to document it, apparently the only place the feature is
> > described is on StackExchange.[...]
> 
> Indeed, it went many years undocumented in less. In more recent
> versions there is a nod to this feature in the less(1) man page:
> 
>    LESS_TERMCAP_xx
>        Where "xx" is any two characters, overrides the definition
>        of the termcap "xx" capability for the terminal.

Ah.  I'm still on less 581.2, and haven't looked at anything later.
I'm glad something's shown up!

> > I'd say more that it is an unforeseen consequence of a deliberate
> > change. However, it appears to be that you can get the feature back
> > by switching grotty(1) back to legacy/overstriking output as opposed
> > to SGR output.
> 
> Ok. For now I've decided to export MANROFFOPT="-c" along with MANPAGER.

One potentially annoying caveat is that if you ever use man(1)'s `-T`
option to format for something _other_ than a terminal device, you will
need to clear the "-c" out of that environment variable, because other
output drivers like grops(1) and gropdf(1) don't understand it.

My understanding is that relatively few people ever use man(1) to do
anything but render man pages to a terminal, though.  Which is kind of a
shame...

> > SGR output was _already_ the default in groff 1.22.4 (and for some
> > time before that), so this change was not actually recent. But if
> > you have replaced a distributor's groff build with your own from
> > source, or if your distributor has decided to accept the upstream
> > default use of SGR, then that could explain why the LESS_TERMCAP_*
> > trick suddenly stopped working.
> 
> Are you certain about this?

Some distributors have overridden this upstream default.  As far as I
know, Debian was the first to do so; Ubuntu inherited it (or perhaps the
other way around, as the same person has long maintained the groff
package for both), and then other distributions cargo-culted it into
their own groff packages.

This has already stung one distribution, and it came up on this list in
the past few days.

https://lists.gnu.org/archive/html/groff/2023-07/msg00057.html

> I don't see any corresponding change in
> https://gitlab.archlinux.org/archlinux/packaging/packages/groff.

I do:

https://gitlab.archlinux.org/archlinux/packaging/packages/groff/-/blob/main/site.tmac

As of this writing, the most recent commit to that file has a hash of
8bc5cd9c; given that it now causes warnings, I would expect it to go
away in the near future.

[In the course of writing this email, we received a bug report against
groff because apparently some hack to make Vim work as a man pager
assumes that grotty(1) will operate in overstriking mode.  My money's
on it being this same problem of a local patch slipping in a device
control command.]

> I have been blindly using this Arch package for years. IIUC the
> invocation from /usr/bin/man is:
> 
>     groff -Tutf8 -mtty-char -mandoc

This is a sound invocation.  You can add a `-Z` option to this and see
if the 'x X sgr 0' command appears in troff(1)'s output.

> and I can tell that the groff provided by 1.22.4 has overstricken
> output when I reinstall the older package, while 1.23.0 groff does
> not.

If I have inferred correctly, you have had the GROFF_SGR environment
variable set, then switched from using Arch's groff 1.22.4 package to
Arch's groff 1.23.0 package.  In the former, this patch was present and
effective.  In the latter, the patch is still present, but does no good
because the grotty(1) output driver no longer honors the device control
command (and warns about it).  So it no longer switches to legacy output
format.

Does this explanation seem to cover the facts?

> > One of the things I'd like to do for groff 1.24 is make grotty(1)
> > use terminfo(3) to employ terminal capabilities, instead of assuming
> > a terminal is either a typewriter ("-c") or an ISO 6429 terminal
> > (default). https://savannah.gnu.org/bugs/?63583
> > 
> > If that is implemented, a side effect is that you could achieve this
> > color customization simply by redefining the corresponding terminfo
> > entries. You could then do so globally, affecting all uses of bold
> > and italic (as such, not counting applications that blindly write
> > ISO 6429 escape sequences as groff, GNU ls, and Bash do) to the
> > terminal, for instance, not just those that grotty writes.
> 
> Are you certain about those too? AFAIK ls color usually comes from
> LS_COLORS, which has the values embedded. They might contain dumb
> escapes but there's no reason LS_COLORS couldn't be generated based on
> a terminfo db.

That's what I was trying to get at.  I think LS_COLORS's use of raw
escape sequences is questionable design.  My guess is that it is due to
historical factors; the history of termcap, terminfo, and (n)curses is
messy, with specters of (1) proprietary AT&T Unix hanging over terminfo
and much curses development; (2) BSD fanaticism which held that termcap
was inherently better due to not being AT&T code, when in fact terminfo
had superior features and flexibility;[1] (3) an early choice by GNU
(circa 1990?) to back termcap (and fork its own implementation) instead
of terminfo; (4) a free implementation of terminfo getting coupled with
ncurses, the maintenance of which suffered from several problems in the
1990s as the Linux kernel and "open source movement" enjoyed a huge
surge of momentum.[2]  By the time ncurses was in a steady pair of
hands, many developers of Free Software terminal-based applications
(Vim, Bash, GNU coreutils [though it was not yet distributed under that
name] had already arc-welded themselves to the termcap API.

By the year 2000 and the dot-com bubble, all the paying jobs in Free
Software were in Web stuff or, for a few, in "making Linux ready for the
desktop [sic]".  The closer one gets to the money spigots, the clearer
the utter disregard of the industry for console/terminal applications
becomes; this is particularly visible in the persistently incomplete
Linux virtual console implementation[3] and the struggles the kernel has
seen to find _anyone_ to attentively maintain its terminal driver.[4]
Evidently the fat wallets with stock ticker symbols refuse to pay for
engineering in this area; and if you're not paid by a fat wallet with a
stock ticker symbol (or an employee of the Linux Foundation), your
chance of getting a patch merged into the Linux kernel is meager.[5]

While I'm shooting gouts of flame at sundry targets, I'd be remiss to
leave out the "S-lang" library, which over the entire span of the best
time window to date for improvement of the terminal experience on free
Unices, chronically over-promised and under-delivered, and the
limitations of which were overlooked by a rabble of fans who were
possessed neither of programming experience nor critical judgment.

Relatedly, in the late 1990s, some highly placed decision makers, like a
former Debian Project Leader, who also at the time led the development
of the Debian installer, announced that he'd never use ncurses for
anything again, out of loyalty (at the time), to its former maintainer.
Red Hat made a similar choice with the S-lang based "newt", which was
about as diligently documented as the library it laid atop.  (This was a
characteristic choice, as Red Hat has long been of the opinion that its
own staff are the only people who have any business modifying, or even
understanding, its code.)

Termcap should be taken out behind the barn and shot.  And in terms of
_implementation_, it has been.[6]  ncurses has long provided an
emulation layer for it, but that doesn't escape the limitations of the
termcap interface itself.

> Also, I think bash/readline uses ncurses impl of termcap's tgetstr to
> get at terminfo db.

Right.

> A user's bashrc certainly might contain hard coded escapes, but as
> they are user supplied they will presumably be correct for their
> chosen term at least.

One hopes.

> There is room for user error in both cases, but they are at least
> adjustable.  Any software that emits such escapes without
> configuration or terminfo is probably an outlier today.

That claim feels optimistic to me, but I hope you're right.

> Yes, I think using a terminfo db is a good direction for groff.  I
> admittedly don't really grok groff all that well but maybe it would be
> possible for man to change the rendering of man pages by redefining
> some macros like .B and .I?

It is absolutely possible.  One can redefine these macros in the
"man.local" file.  The usual caveat applies--if you break it, you get to
keep both pieces.  :-O

> With either that or a custom terminfo db, man-db has both $MANOPT and
> a user configuration file that could then be used to configure this
> hypothetical feature.

One thing I should clarify is that I anticipated a refactoring of
grotty(1) to use terminfo(3) as enabling us to kill off the
"GROFF_NO_SGR" variable, whereas I also said in reply to you that I had
no plans to kill off the variable.  This sounds contradictory.

What I did not talk about was time scales.  Given the grief around this
issue, I think it might be a good idea to retain "GROFF_NO_SGR" for a
release, at least to throw a diagnostic about how grotty(1) uses
terminfo(3) now, and if people want Western Electric Teletype Model 37
emulation they should set $TERM accordingly.

Regards,
Branden

[1] I recommend John Strang's _termcap and terminfo_ book for a good
    survey and comparison.  He doesn't editorialize much; termcap's
    ridiculous limitations veritably leap off the page.

[2] https://invisible-island.net/ncurses/ncurses-license.html
[3] 
https://lore.kernel.org/linux-man/20220320160217.gws42lklp6ishzub@localhost.localdomain/
[4] https://lwn.net/Articles/343828/

[5] This piece doesn't come right out and back up my claim, but as it's
    under the Linux Foundation's banner, it's worth having a look at
    what that organization is willing to live with respect to PR.

    
https://www.linuxfoundation.org/blog/blog/jonathan-corbet-on-linux-kernel-contributions-community-and-core-needs

[6] As far as I know, the various BSDs have either adopted or forked/
    reimplemented ncurses, including its termcap compatibility
    interface.  GNU termcap has not been maintained for over 20 years.

signature.asc
Description: PGP signature