Re: [Libcdio-devel] Test suite

[Rocky Bernstein]

 >What distinguishes executables in test/ and test/drivers? They both
seem to test the drivers.

tests in drivers are what are often called "unit tests" or I like to call
them "transparent-box" tests. You look at the specific code and devise
tests given that specific code.

For example suppose my code is


for (i=0; i<n; i++) {
  /* something ... */
}

Then the driver test should have test for when n less than or equal to zero
as well as a case for when n is greater than 0. This test will help me
pinpoint more precisely where in that code things might go wrong, because
it is based on inspecting how that particular code works.  Unit tests tend
to be fast.  But tests for code like this are also fragile.

The tests in the parent directory, are more end-to-end kinds of tests and
they may use top-level tools like cd-info. They are less fragile because
they cover larger units of code. But they are slower and when there is a
problem my not pinpoint the error as much.


> Now my biggest question. Why are all the test sources and scripts
generated from .in templates? Don't we have `make' for source files?

Look for @..@ in the templates and you will see that those values are
custom for things specific to you setup. A common thing that is customized
for your setup is the full path of a data files that are needed to run a
test. Or where uninstalled C include headers are located.

> For the scripts its even worse because the generated files are not
necessarily executable.

Not "worse" but "better" ;-)  the script is not executable because you are
not supposed to execute it. Or modify it by hand. Instead values get
substituted in there such before it can be executed.

As for AM_TESTS_ENVIRONMENT I'm not aware of it, so it is probably
something newer. If it addresses the above concerns, by all means use it.


On Wed, Apr 13, 2016 at 3:53 AM, Leon Merten Lohse <address@hidden>
wrote:

> Good morning,
>
> to prevent further regressions in the CD-Text code, I have decided to
> write a dedicated test case for it and not hide it within the cue test
> case anymore.
> I takes me a great deal of effort to understand what is actually
> happening in the test directory. This is partially because I am
> fairly new to autotools but also because of the overall complexity of
> the test suite.
>
> Let me try to express, what I learned so far and comment on it.
>
> There are 4 places for tests:
> 1) scripts in test/
> 2) executables in test/
> 3) executables in test/drivers/
> 4) executables in examples/
>
> Although all executables in 4) get executed, all but 3 return SKIP when
> no disc is inserted. Looking into the sources in examples/, it does not
> look like those are actually meant to be tests. The result merely
> indicates if execution fails or not.
>
> There is a test/data directory. Still, much of the data lies in the
> test/ directory.
> What distinguishes executables in test/ and test/drivers? They both
> seem to test the drivers.
>
> Now my biggest question. Why are all the test sources and scripts
> generated from .in templates? Don't we have `make' for source files?
> For the scripts its even worse because the generated files are not
> necessarily executable. I found numerous attempts to make them
> executable in different places. Can we not use AM_TESTS_ENVIRONMENT to
> define the environment used by the scripts? That would also save the
> trouble with the executable bits.
> Is it a standard practice to use .in templates for such purposes at
> all? Is there a reason for all this or is it simply `historic'?
>
> Please excuse my naive questions.
>
> Best regards
> Leon
>
>