[Nmh-workers] test framework clobbers failed data files

[Lyndon Nerenberg]

Trying to figure out why test-attach was failing pointed out an issue with the 
test framework.

Currently, the names of the test input/output files are hard-wired into the 
scripts, and use $$ to somewhat "uniquify" the names to avoid collisions.  When 
a script runs multiple tests it looks like the same naming template is used 
throughout.

In this specific case, check diffed the files (which failed), then the 
following test clobbered the files.  Not having the files from the failed test 
at hand made it difficult to see just what the underlying failure was.  I had 
to hardwire an 'exit 1' at the appropriate location in test-append in order to 
examine the data files.

It would be nice if a failed test would leave its data files around for a 
post-mortem.

There are a couple of options:

1) Modify all the test scripts to use unique filenames for each test inside the 
script (and ensure check() doesn't remove them on failure).

2) Modify run_test() and check() (and perhaps others?) to internally generate a 
unique filename prefix for each test invocation.  This also requires modifying 
all the test scripts.

3) Modify check() to abort in the face of failure, leaving the intermediate 
test files behind.

(1) and (2) seem like a lot of busy-work, and (2) seems needlessly 
bureaucratic.  Given that any test failure is a build failure, potentially 
skipping a couple of follow-on tests in the same scripts doesn't seem all that 
harmful.  If there are no loud objections, I'll go ahead and implement option 
(3).

--lyndon

signature.asc
Description: Message signed with OpenPGP using GPGMail