Re: [PATCH v4 3/3] parallel-tests: allow each test to have multiple results

[Stefano Lattarini]

On Monday 20 June 2011, Ralf Wildenhues wrote:
> Hi Stefano,
> 
> * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:26:06PM CEST:
> > On Monday 20 June 2011, Ralf Wildenhues wrote:
> > > Why not just split the whole documentation change into a followup patch
> > > then?
> > >
> > Because that would only postpone, not avoid, the continous tweaking and
> > amending the documentation; what I'd like instead is to improve it
> > organically and incrementally, in separate patches; i.e., commit a sketchy
> > but correct (even if incomplete) documentation first, and then improve it
> > with follow-ups (maybe handling one concept or one part at the time).
> 
> Well, then please split the incomplete doc part of your patches into
> separate patches, so I can say "no" to them and "yes" to the other ones
> easily.  ;-)
>
This can be done, yes.  I'll keep the documentation part of the first
patch, tough, as that's complete as-is (not perfect, but without TODOs
or FIXMEs, nor obvious shortcomings).

[PATCH v5] series coming up soonish ...

> > > > > This will be quite fork expensive, if done in real-world code.
> > > > >
> > > > But this is in a script used only for testing.  I don't think
> > > > it's worth optimizing it.
> > > 
> > > No, it's not, but your real scripts won't look all that different.
> > > Besides, why not do it right the first time?
> > >
> > I still don't see the point this honestly, but I've thrown in a couple
> > optimizations for bash, zsh and XSI shells (see attached squash-in).
> > Is that enough?
> 
> Just leave that out.  You are right that doing such micro optimizing
> will not at all be a good strategy, if you do it unorganized and without
> planning.  Leave it for later, but still keep half an eye on the stuff
> not getting like 4 times slower overhead.
>
OK.

> > > I don't actually care much about micro optimizations like the above at
> > > this point, but I do care when the whole set of code changes will
> > > introduce a factor of 2 slowdown in the test suite overhead.  It looks
> > > like it may eventually, judging from your measurements done, and that's
> > > what I am trying to prevent.  On w32, that would cause real pain.
> > >
> > But maybe it would be worth trying to instead optimize stuff like
> > $(am__check_pre) and $(am__vpath_adj_setup), where we could trim
> > extra forks in the case of XSI shells or bash.
>
Addendum: your (incomplete) patches about GNU make specific optimizations
could help with this too.

> 
> Doesn't sound like it would bring your project forward at this point.
>
It wouldn't -- and I wasn't planning to do it now; I only pointed out
that it would be possible.

> I'm sorry I brought this topic up before.  I shouldn't have.
> 
> > I.e., optimize an
> > existing and tested implementation instead of holding back a
> > promising design due to *possible* future performance problems.
> 
> Well, I don't like this attitude.  If something will have a performance
> problem, then maybe it was not all that promising after all.  I'm not
> claiming your approach has, however.  All I'm suggesting is that you do
> keep an eye on it.
>
Of course, sorry if I made it look I'm disregarding the problem.  I'm
not, and I'll try hard not to let it fall off the radar either.

> > Also, "execing" the test driver in check2.am instead of "spawning" it
> > could avoid an expensive fork.  But we should then test at configure
> > time that $SHELL can gracefully handle such "execing" w.r.t. the use
> > of $(TESTS_ENVIRONMENT); i.e., that an usage like:
> >   "9>&2 foo=bar exec sh -c 'echo $foo >&9'"
> > does the expected thing (hint: it does with dash, bash, zsh, NetBSD
> > /bin/sh and Debian ksh; it doesn't with Solaris /bin/ksh and /bin/sh).
> 
> Such changes should only be done after demonstrating that they actually
> cause measureable speedups.
>
Of course -- that's why I'm not doing them right now, and keeping them
for follow-up patches.  While the changes might be simple, proving that
they are usful might be harder, and certainly more time-consuming.

> And has no semantic changes (which I am not so sure of).
>
That's why we should need extensive configure-time tests.

> For example the parallel BSD makes tend to reuse shells for running
> the recipe commands;
>
But only for the commands in the same recipe, right?

> I'm not so sure they like if their shells go away with exec.
>
In this case, not a problem I think, since the invocation of the
testsuite driver is the last command in the recipe.

Regards,
  Stefano