Re: master 262d0c6: Mark some tests as expensive

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Yes, if they take more than a second to run.
>
> So how do we make sure some change didn't introduce a bug in those
> features?

We either say "make check-expensive", or we live on in uncertainty, as
with all those other functions we've marked as expensive.

>> We've already made this decision with all the tests previously marked as
>> expensive, so there's nothing new here.
>
> There's a large gap between what is currently marked as "expensive"
> tests and having entire packages not tested at all.  The latter sounds
> too radical to me.  E.g., auto-revert is an important feature, used by
> many people.  Not having it in regression testing sounds like a step
> backward to me.

It's still being tested -- just not as often.

> I'm talking about a balance here.  Losing the tests of complete
> features just because we want tests to finish quickly sounds
> sub-optimal to me.  Can we make a smarter balance?

Sure, there's a balance, and I think the current balance is a bit too
skewed towards having too many slow tests in the "make check".  I think
that, ideally, "make check" should be so fast that people run it as
their standard workflow before pushing a change, and we're not quite
there.

Having people do "make check" as a matter of routine, and running 97% of
the tests is, in my opinion, better than people doing a "make check"
seldom, but running 98% of the tests.

Where the cutoff is a matter of balance, yes, but I think a test that
takes a second is way too slow to be run in the routine case.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no