[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Testing groff. Was: GNUism in groff tests

From: John Gardner
Subject: Re: Testing groff. Was: GNUism in groff tests
Date: Mon, 6 Jan 2020 12:40:42 +1100

> and they don't have to be unit tests.  In some ways, that's the worst
> option because they tend to be trivial functions being tested

Nobody's saying they should *all* be unit-tests. It's better to test
components in isolation first, verify they do what they're supposed to,
then follow up with more elaborate tests. That way, a regression is easier
to spot and narrow down when a test failure occurs. BDD-style testing can
help — I find tests easier to follow when they're written like plain English

  When a file entry is displayed
    When it matches an icon
      ✓ it shows the icon next to its name
      ✓ it removes the built-in icon class

  When a zip-file is opened
    ✓ it shows icons for each of its contents
    When a file is displayed
      ✓ it shows an icon next to its filename
      ✓ it tests path-sensitive rules

Once the simple stuff is covered, you can move on to running pixel-diffs
against corpora of documents (which don't necessarily have to use a
BDD-flavoured approach to testing).

I'm something of a pedantic prick when it comes to airtight test coverage,
so a lot of this mail probably stems from personal bias.

On Mon, 6 Jan 2020 at 05:20, Ralph Corderoy <address@hidden> wrote:

> Hi Ingo,
> > Werner wrote:
> > > I think the proper way for testing groff would be to make it run
> > > with a fuzzer,
> >
> > Yes, that would no doubt be useful.
> ...
> > is kind of orthogonal to developing a test suite, though.
> ...
> > A test suite, on the other hand, is most useful for making sure no
> > regressions creep into the functionality,
> I think that's groff's pressing need: stopping regressions.  Given what
> I imagine is the part-time nature of contributions, to a large code
> base, that implements a complex system of parts, e.g. the troff
> language, I suspect a fix to more delicate bug may be quite high risk of
> introducing a regression, and often this will be only apparent in the
> PostScript, PDF, or pixels on the page being different; no warnings,
> errors, or dumped core.
> That bug I referred to earlier is a good example.
>     The footer printed with the -mm macros has changed position between
>     groff versions 1.17.2 and 1.18
>     ...
>         .\" groff -mm -ma4 >
>         .\"
>         .PGFORM 21c-2i 29.7c-1i 1i 1
>         .PH "'hl'hc'hr'"
>         .PF "'fl'fc'fr'"
>         .\"
>         .S +1
>         .P
>         Footer not in a constant place across groff versions.
>     If the `.S +1' is commented out the problem doesn't occur.
> Fixing known bugs, finding more bugs with fuzzing, implementing new
> features, or adding something from another troff, are all less fun to do
> if one's constantly concerned that there's not a good safety net to stop
> faults getting released.  And a buggy groff version or two means
> documents have to start testing the troff implementation and its version
> to implement workarounds.
> So I think regression tests come first, and they don't have to be unit
> tests.  In some ways, that's the worst option because they tend to be
> trivial functions being tested; it's the interplay of the whole system
> that matters.  A corpus of documents and golden eyeballed output, say
> ditroff, PostScript, or PNG to suit, would be a good start.  Even if
> it's just a script that compares renderings of all the documents shipped
> with groff between two built-but-not-installed groffs.
> Do any of the other troffs have regression tests to our benefit?
> --
> Cheers, Ralph.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]