Re: [PATCH v2 1/8] tests/qemu-iotests/testrunner: Allow parallel test invocations

[Hanna Reitz]

On 11.02.22 17:00, Kevin Wolf wrote:
> Am 11.02.2022 um 14:53 hat Thomas Huth geschrieben:
> > On 11/02/2022 10.29, Kevin Wolf wrote:
> > > Am 09.02.2022 um 11:15 hat Thomas Huth geschrieben:
> > > > If multiple tests run in parallel, they must use unique file
> > > > names for the test output.
> > > >
> > > > Suggested-by: Hanna Reitz <hreitz@redhat.com>
> > > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > > ---
> > > >    tests/qemu-iotests/testrunner.py | 2 +-
> > > >    1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> > > > index 0eace147b8..9d20f51bb1 100644
> > > > --- a/tests/qemu-iotests/testrunner.py
> > > > +++ b/tests/qemu-iotests/testrunner.py
> > > > @@ -259,7 +259,7 @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
> > > >            """
> > > >            f_test = Path(test)
> > > > -        f_bad = Path(f_test.name + '.out.bad')
> > > > +        f_bad = Path(f'{os.getpid()}-{f_test.name}.out.bad')
> > > >            f_notrun = Path(f_test.name + '.notrun')
> > > >            f_casenotrun = Path(f_test.name + '.casenotrun')
> > > >            f_reference = Path(self.find_reference(test))
> > > Hmm... It does make sense, but nobody ever cleans those files up.
> > > Currently, the next run of the test will just overwrite the existing
> > > file or delete it when the test succeeds. So after running the test
> > > suite, you have .out.bad files for every failed test, but not for those
> > > that succeeded.
> > >
> > > After this change, won't the test directory accumulate tons of .out.bad
> > > files over time?
> > True ... but we certainly want to keep the file for failed tests for further
> > analysis instead of immediately deleting them ... maybe it would be enough
> > to encode the image format (qcow2, qed, vmdk, ...) into the output name,
> > instead of using the PID, so that "make check SPEED=thorough" works as
> > expected here?
> It depends on what the supported use case for test suites running in
> parallel is. If it's just for testing multiple formats at the same time,
> then this would work, yes.
>
> I could think of more test runs that you might want to do in parallel,
> like different protocols, different image format options, maybe even
> different host file system. I'm not sure if all (or any) of these are
> relevant, though.
>
> Supporting only things that "make check" uses might be a good
> compromise.

Personally and originally, I wrote that diff to allow me to actually run 
the very same test many times in parallel.  If an error occurs only very 
rarely, then I like running like 24 loops of the same test with exactly 
the same configuration (just different TEST_DIRs, of course) in parallel.

The fact that the .out.bad files tend to accumulate is why I haven’t 
sent it upstream so far.  Personally, I like Vladimir’s idea to put them 
into TEST_DIR, but probably just because this works so well for my usual 
case where TEST_DIR is on tmpfs and I thus don’t have to clean it up.

Hanna