[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: mark expensive tests
From: |
Eli Zaretskii |
Subject: |
Re: mark expensive tests |
Date: |
Mon, 04 Jan 2016 17:47:56 +0200 |
> From: John Wiegley <address@hidden>
> Date: Sun, 03 Jan 2016 12:09:12 -0800
> Cc: address@hidden
>
> Yes, we will have three test intensity levels:
>
> 1. Sanity tests just make sure that the environment works, and the whole of
> them should run in <30 seconds on typical hardware. Some shops call these
> 'smoke tests', as they are just there to make sure that Emacs can
> function in the most basic ways.
>
> 2. Regular tests do not incur intensive CPU or memory costs, and so can be
> run on any hardware. These should finish on a scale of minutes, likely
> <10 minutes at the most.
>
> 3. Extensive tests are given free reign, and may not even be able to
> complete on systems with CPU or memory constraints. These should feel
> free to take up to an hour, maybe even beyond.
>
> 4. Selective tests are never run automatically, but exist to stress test
> some particular area of the system. These could take days, it doesn't
> really matter what their requirements are.
I'm fine with this gradation, but here's some reality check:
. The current test suite takes 3.5 min for a full run, including
compilation of all the *.el files, 2.25 min if the *.el files are
already compiled
. For comparison, the test suite of the latest version of Texinfo I
just built takes 4.5 min on the same machine, even though a lot of
tests are skipped (due to some required infrastructure not being
installed)
So it sounds like we have no candidates for #3 and #4, and the
division between #1 and #2 is questionable, since 2 min is not such a
long time to wait, IMO.
> I'd especially like the file-notify tests to move to #3, since these are what
> consistently bog down my testing environment.
file-notify-tests takes about 25 sec here, so I'm unsure why it should
be in #3. Maybe that's because of remote tests (which are skipped on
my system), in which case perhaps the remote tests should be separated
into a separate test file and run on demand only.
- Re: mark expensive tests, (continued)
- Re: mark expensive tests, John Wiegley, 2016/01/08
- Re: mark expensive tests, Eli Zaretskii, 2016/01/04
- Re: mark expensive tests, Eli Zaretskii, 2016/01/04
- Re: mark expensive tests, Paul Eggert, 2016/01/04
- Re: mark expensive tests, Eli Zaretskii, 2016/01/04
- Re: mark expensive tests, Paul Eggert, 2016/01/04
Re: mark expensive tests,
Eli Zaretskii <=
- Re: mark expensive tests, Michael Albinus, 2016/01/04
- Re: mark expensive tests, Eli Zaretskii, 2016/01/04
- Re: mark expensive tests, Michael Albinus, 2016/01/04
- Re: mark expensive tests, Eli Zaretskii, 2016/01/04
- Re: mark expensive tests, Michael Albinus, 2016/01/04
Re: mark expensive tests, Michael Albinus, 2016/01/04