[bug #61439] [tmac] test "an_inner-footer-abbreviation-works.sh" fails if package defaults changed

[G. Branden Robinson]

Update of bug #61439 (project groff):

                 Summary: [tmac]: test "an_inner-footer-abbreviation-works.sh"
fails => [tmac] test "an_inner-footer-abbreviation-works.sh" fails if package
defaults changed

    _______________________________________________________

Follow-up Comment #3:

Hi, Bjarni,

[comment #2 comment #2:]
> 
> [comment #1 comment #1:]
> [...]
> 
> > I need more to go on.  What is in the test log file?  It should be in your
build tree:
> > 
> > tmac/tests/an_inner-footer-abbreviation-works.sh.log
> > 
> 
> Yes, you need more to go on.
>   You need information (and thinking) but primarily from
> yourself.

Another groff contributor has recently pointed out how you shouldn't hold your
breath waiting on me to think correctly.
<https://lists.gnu.org/archive/html/groff/2021-11/msg00009.html>  Explain
issues so that many people can understand.

> > I have a suspicion that your personal groff branch is setting the
CHECKSTYLE register to a nonzero value, perhaps in tmac/man.local or
tman/an.tmac.
> > 
> 
>   The change is in "git/groff/tmac/an.tmac" among other additions of
> mine.

Okay.  All is well.  We've been around this block before--maybe Dave can
remember in which ticket(s).  The groff test suite is not a generic test
harness for *roff implementations in general.

That would be a wonderful thing to have but it's out of scope for the groff
project itself.  As I understand it, Bertrand put the infrastructure and first
few tests in because groff had no automated regression testing at all and it
was sorely needed, given the project's size and complexity.

Some of the tests are "white box"[1] tests, coded with knowledge of
implementation internals.  A general *roff test suite would probably take a
"black box" approach, caring about nothing but generated, user-facing output.

One of the consequences is that the test suite is coupled to the branch that
it's in.  That's why I say, if you're going to make changes to the
implementation, you may have to modify the tests as well.

> > If you've made such a change, the responsibility is similarly on you to
change the tests to update their expectations.
> > 
> 
>   No, it is not.  The defect is in the test code, not in the
> environment of the user.  It is your expectation that is wrong, not
> that of any other user.

If someone later changes the default CHECKSTYLE to be "truthy", it likely will
break at least one test.  That's a consequence of another test design goal: I
want freedom to tweak the language of diagnostic messages, and I don't want to
have to complicate the tests by pattern-matching diagnostic output,
_especially_ when it's not the point of the test.

The test at issue is written to deliberately expect zero output to the
standard error stream.  If you hack up your tmac/an.tmac to instrument it, it
and other tests will fail.  This happens to me when I troubleshoot our man(7)
macros.  Once I figure out the macro programming problem that has bedeviled
me, I back out the instrumentation and write a test for root-cause problem I
discovered.  And if I can't figure it out, as with hyperlink macros in
paragraph tags
<https://git.savannah.gnu.org/cgit/groff.git/commit/?id=b5461ca30b0f91a3efdead93aa7d5c2dca112d27>
recently, I certainly don't leave the instrumentation in and rewrite all the
failing tests to expect it.

I make no claim to be the world's greatest software tester but I do think the
approach I'm using helps keep the tests short, focussed, and relatively easy
to understand (notwithstanding the big readability penalty that the POSIX
shell language imposes).

>   If a test fails by returning a value different from zero, it
> may be expected that the observer investigates the cause, if he knows
> how, in my case by issuing

Yes, "sh -x".  Often I find myself editing the test to just dump the output of
the offending test ('echo "$output"').  Frequently I can tell quickly by
visual inspection what has gone wrong, and there is little other output to
distract me.  "sh -x" can be chatty.
 
>   Your issue now is similar to the bug report
> 
>  #58164] First html-test fails when a non-utf-8 charmap is used
> 
> where you deny a general solution, but then later did change the code
> to fulfil your own constraints.

Yes.  I had access to a Mac OS X build environment and, curious, I built groff
on it to see what would happen.

You should note that the test was altered to _skip_ itself if the environment
didn't prove adequate.

> ---cut---
> (commit
> f51cf27208e70ccb60afc4a245f293a0146f77ed
> Author: G. Branden Robinson <g.branden.robinson@gmail.com>
> Date:   Thu Oct 28 00:00:33 2021 +1100
> 
>     [tests]: Fix portability problems.
> 
> [...]
> 
>     * src/roff/groff/tests/smoke-test_html_device.sh: Stop trying to set
>       $LC_CTYPE to "C.UTF-8"; some systems don't support this
> expediency.
>       Skip the test if the tester hasn't configured the environment
>       adequately.
> ---cut---
> 
> 
>   An update of my proposal was put forward in
> 
> https://lists.gnu.org/archive/html/groff/2020-11/msg00138.html
> (Subject: Testing the test)
> 
>   I later reduced the cases to be tested in my repository.
> 
>   Fact is: that test does not need any locale, "groff" handles that.
> 
>   My current example (removing irrelevant lines):
> 
> echo "testing -k -Thtml" >&2
> printf '\303\241' | "$groff" -KUTF-8 -Thtml | grep -qx '<p>&aacute;</p>'
> 
> # We test compatibility-mode HTML output somewhat differently since
> # preconv only emits groffish \[uXXXX] escapes for non-ASCII codepoints.
> echo "testing -C -k -Thtml" >&2
> printf "\('a" | "$groff" -C -k -Thtml | grep -qx '<p>&aacute;</p>'
> 
> ###
> 
>   "The problems of the real world are primarily those you are
> left with when you refuse to apply their effective solutions."

The lesson here is that real world experience with a Mac OS X environment
proved more persuasive to me than your occasionally context-light and
sometimes hectoring reports, particularly when you sprinkle them with
quotations as evangelists do with their favorite texts.

There is much philosophical literature than we can read for wisdom, certainly
including EWD, but quoting it at length in an attempt at persuasive discourse
suggests that your are not molding your arguments to your context and
interlocutor, but expecting your counterparty to reshape themselves to fit
your dogma.  Ultimately, that is an authoritarian approach, not a collegial
one.

In simpler language, you need to work harder to convince me using fresh
language appropriate to the scenario.  Dropping scripture in front of people
and inviting them to proof-text it in order to find their way into agreement
with you is not likely to succeed.

(Okay, I brought up proof-texting
<https://carlsweatman.wordpress.com/2010/08/04/problems-with-proof-texting-1/>.
 Sometimes I fail to achieve simpler language.)

Normally I'd close the ticket as invalid based on the customization
revelation, but since I got all philosophical I'll leave it open and invite
you (and others) to try the approach to persuasion that I claim is more
effective.

Admittedly, I may remain a stick in the mud nevertheless, and thereby
similarly fail to be an effective advocate for my principles.

Regards,
Branden

[1] "Clear box" would probably be a better metaphor but it's not one I've seen
in use.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?61439>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/