Re: Can the automated tests be included in the source tarball?

[Shlomi Fish]

Hi Ole!

Sorry for the late reply.

On Tue, 13 Oct 2020 13:56:39 +0200
Ole Tange <ole@tange.dk> wrote:

> On Thu, Oct 1, 2020 at 5:09 PM Shlomi Fish <shlomif@shlomifish.org> wrote:
> > On Tue, 29 Sep 2020 01:18:31 +0200
> > Ole Tange <ole@tange.dk> wrote:
> >  
> > > On Thu, Sep 24, 2020 at 8:37 AM Shlomi Fish <shlomif@shlomifish.org>
> > > wrote:  
> :
> > > Unfortunately some of them are heavily dependent on running on the
> > > same equipment as mine due to timing issues (and every time I change
> > > hardware or upgrade software, I have to tweak the tests or results a
> > > little).  
> >
> > I see. I suppose you are aware of running on a pristine CI system, e.g:  
> 
> I had a talk with a guy who used Jenkins. Jenkins is all good and fine
> for tests that have predictable outcomes no matter the environment.
> 
> It is worse when you have 2 tests: One has to run while the machine is
> swapping and one has to run when the machine is _not_ swapping. In
> other words they cannot be run in parallel.
> 
> Or: A test has to run on a machine with high CPU load, while another
> test has to run on an idle machine.
> 
> Or: A test has to be run on Centos 3, because other distributions do
> not show the error.
> 
> I guess around 20% of my tests are in that category. The rest could
> probably be run by a normal CI.

I see. I tend to agree.

> 
> The current home-made test system also has the advantage that if a
> test fails, it will be pushed to the top, so when I run the test suite
> again, it will be the first test to run.
> 

This is possible to do in CI setups as well, with a (hopefully) small amount of
scripting.

> > > If you want to help, a good start would be to have
> > > testsuite/REQUIREMENTS ported so it would work on more systems. It
> > > (tries to) install and set up all required software to run the
> > > testsuite.  
> >
> > Wow, it looks intimidating. And I thought some of my FOSS projects had too
> > many prereqs!  
> 
> The problem is that errors are often seen in the interaction with
> other tools - sometimes only as race conditions.
> 

I see - we can try modularising it.

Regards,

        Shlomi

> 
> /Ole
> 



-- 

Shlomi Fish       https://www.shlomifish.org/
Summer Glau Facts - https://shlom.in/sglau-facts

Hacker sees bug. Hacker does not want bug. Hacker fixes bug.
    — https://www.shlomifish.org/humour.html

Please reply to list if it's a mailing list post - https://shlom.in/reply .