bug#20715: tap-driver.sh 'missing test plan' message contradicts the TAP Standard

[Arthur Schwarz]

On 2015-06-02 09:40 -0700, Arthur Schwarz wrote:
> [Nick Bowler wrote]
>     The plan line has to be mandatory for this feature to work as
>     intended (i.e., for the TAP consumer to determine whether a
>     producer has run to completion or not).  An optional plan would
>     be useless.
> 
> Sorry to be such a pest.
> 
> Your statement that "... determine whether a producer has run to
completion
> or not" is puzzling. The TAP Standard specifically states that if the test
> plan is 1..N and the number of test lines are k < N, then the k+1 .. N
> missing test lines are to be considered as skipped.

Where does it say that?  It is not correct to treat missing test results
as skips.

   From  http://testanything.org/tap-version-13-specification.html

   Skipping a few

   The following listing plans on running 5 tests. However, our program
decided to not run tests 2 thru 5 at all. To properly report this, the tests
are marked as being skipped.

   TAP version 13
   1..5
   ok 1 - approved operating system
   # $^0 is solaris
   ok 2 - # SKIP no /sys directory
   ok 3 - # SKIP no /sys directory
   ok 4 - # SKIP no /sys directory
   ok 5 - # SKIP no /sys directory

It is an error for the number of actual test results to differ from the
plan.  This is the whole point of the plan: the consumer always knows
how many test results are expected.  This enables the consumer to tell
the difference between a producer that crashed before completing all the
tests, and a producer that ran normally.

   See above.

> There is one issue not addressed in the TAP Standard and not addressed in
> the Automake Manual. What happens if k > N?

If k > N, then the number of test cases does not match the plan, so
this is an error.

  TAP allows k < N (see above). The question of what happens if k > N seems
reasonable. As an extension, the Automake Manual does not state what will
happen if k > N (nor if k < N). This also reflects of the TAP Protocol
statement that the test plan can be at the beginning or end of the test. The
issue is easy to address if the test plan is at the beginning of the
document but will require that an implementation use deferred test line
posting if the test plan is at the end of the document. I think that the TAP
Protocol is poor in this regard.

art

Regards,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)