#1749: Parallelize CHICKEN core tests

[Chicken Trac]

#1749: Parallelize CHICKEN core tests
--------------------------------+-------------------------------
            Reporter:  siiky    |       Type:  enhancement
              Status:  new      |   Priority:  not urgent at all
           Milestone:  someday  |  Component:  build system
             Version:           |   Keywords:  tests
Estimated difficulty:  medium   |
--------------------------------+-------------------------------
 = What?

 This ticket is meant to be the place to discuss everything related to the
 parallelization of the (current) CHICKEN core tests (`make check`). It's
 not a "please do this" ticket! I'm willing to work on it.

 I'd like to know if there's any interest in it, so please share your
 opinions, thoughts, and ideas!

 = Why?

 The tests take the majority of the time of the whole build process. On my
 desktop:

  * Build: ~3m32s (1 CPU); ~2m25s (2 CPUs); ~2m12s (3 CPUs); ~2m8s (4 CPUs)
  * Test: ~7m38s
  * Install: ~2m19s

 On my RBPi2 the whole process takes hours -- I usually leave it running
 over night. On my RBPi4 it takes around 45min, which is not so bad, I'm
 still impressed with it, but "I want it all, and I want it now"! This is
 actually my biggest incentive, because of the salmonella tests.

 megane said on IRC that there's been some talk about rewriting the tests
 in Scheme, but that it's not for the near future, which I can understand.
 However, I think parallelizing the current tests should be pretty
 maneagable, and quicker than rewritting them in Scheme (a guess before
 actually putting my hands to work, of course), so we could take advantage
 of those in the meantime.

 = How?

 Assuming that there's interest, I'd like to know what others think --
 ideas on how to implement, and how not to implement the tests. Related,
 I'd also like to know what's OK, and what isn't -- file structure, file
 names, number of new files in the repository, files created by the tests,
 etc.

 Now, my current idea/plan: Keep everything contained in the `tests/`
 directory, as it is now. Have a script for each test -- one for
 "repository search path", one for "types.db consistency", etc. Have an
 "environment" script that sets up the environment variables (`CHICKEN`,
 `CHICKEN_PROFILE`, ...), commands (`compile`, `compile_r`, ...), etc, to
 be used by each of the previously mentioned scripts. Have a Make target
 for each test, so the actual parallelization is managed by Make. Each test
 script should return the usual 0 for OK, and !0 for !OK, so that Make
 works correctly.

 The first obvious problem is the tests' output. Right now, since they're
 sequential, everything is printed to stdout, and all is fine. But running
 in parallel that's not an option. My first idea to solve this was to save
 the output of each test script to a test-specific output file.

 And the last thing I'm wondering about is whether having a test-specific
 directory inside `tests/` is worth it or not. This is mainly to unclutter
 the `tests/` directory, due to the tests' output files.

 That's all I have for now! Hopefully I didn't miss anything stupidly
 obvious...

-- 
Ticket URL: <https://bugs.call-cc.org/ticket/1749>
CHICKEN Scheme <https://www.call-cc.org/>
CHICKEN Scheme is a compiler for the Scheme programming language.