Re: run BISTs for all installed packages

[Andrew Janke]

On 3/5/19 7:18 PM, Mike Miller wrote:
> On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
> > I've put together a list of references to bugs and Octave maintainers
> > mailing list threads related to enhancing the test suite: 
> > https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md
> >
> > Any major ones I'm missing?
>
> Looks good, thank you for collecting all of these topics together.
>
> I agree with your project goals. 

Thanks!

> I also hope you'll preserve some of the
> current test suite features I think are important, including being able
> to run the test suite pre-install or post-install, read-only, optionally
> print full test failures in auto build or CI environments.

I'm gonna preserve ALL of these features, *ALL* of them, ha ha ha!

Though not necessarily with the same invocation syntax. :)

More seriously, my intention for Testify is to be a:
a) more-or-less strict functional superset of existing BIST 
functionality, with
b) nicer output and more convenient calling forms, and
c) maybe more structured control over how the BISTs are invoked and 
interact with each other and are collected together in summary objects

This Testify project isn't some big idea to make a whole new testing 
layer for Octave[1]. I mean, originally all I wanted to do was provide a 
concise summary output of failed tests at the end of the 
__run_test_suite__() output so you didn't have to scroll back and copy & 
paste multiple times to get a list of your failed tests 
(https://savannah.gnu.org/bugs/index.php?55522). But all this code is 
intertwingled, and one thing led to another, so...

[1] ...although, maybe, I'll throw in a port of the xUnit testing 
framework, given some spare time and user interest...

> If I have my own list of annoyances / wishlist things I'd like to see
> improved, would you rather see them reported on savannah or on your
> project?

I'd prefer them reported on my project (one issue report per 
annoyance/wishlist item, please, and no annoyance too small), because a) 
GitHub has nicer issue/task-tracking tools than Savannah IMHO, and it'll 
be easier to grind them down in to separate pieces of work there, b) I'd 
probably consider these in-scope for this particular project, and could 
iterate on them in GitHub independent of the Octave project management 
because Testify is an independent project that's not in Octave Forge 
yet, and c) I think in terms of tooling, it's actually easier for us to 
work on an independent octave project using git/GitHub and `pkg install` 
tooling? Once you get to multiple commits, forks and branches are easier 
to manage than Hg changesets stored behind bug attachment URLs.

You should view Testify as "Andrew's wacky idea of where Octave BIST 
support should go in the next release or two", and if you submit a 
gripe/wishlist item there, I'm likely to pick it up and have an Idea and 
do something about it. And then hopefully eventually we can get back 
together with core Octave developers, find a way that a compromise 
version of Testify can be merged back in to Octave core, do that, and 
everybody benefits.

> Are you aware of the 'doctest' package [1] to run tests from Octave help
> strings? I think it's mostly orthogonal to your goals, just wanted to
> make sure you knew it was out there.
>
> [1]: https://github.com/catch22/octave-doctest

Uhh, no! Thanks for the heads-up. I'll look in to it.

At first glance, this actually doesn't look *that* different from BISTs, 
it just has fancy ways of specifying the strings it's using as 
"expected" values in its assert() logic...

> I also maintain a separate shell-based test suite [2] to test Octave's
> command interface, command line options, exit status, and so on, also
> unrelated to your project goals.

This is interesting. Looks like you're mostly concerned with the 
interface between the `octave` command or runtime and the external 
system. This is probably something I should learn about some day. :)

> But at some point I'd like to see these two supported by Octave to some
> extent.
>
> [2]: https://gitlab.com/mtmiller/octave-test-suite