emacs-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: call for more ert tests


From: Stephen J. Turnbull
Subject: Re: call for more ert tests
Date: Tue, 25 Jun 2013 11:31:50 +0900

Eli Zaretskii <address@hidden> writes:

 >> Well, requiring tests for new features or fixed bugs, and confirmation
 >> that there are no regressions from the existing tests.
 >
 > I believe it's good, obviously.  The problem is with introducing it
 > without losing too many contributors.

It's not obviously good.  It helped kill Bazaar development, for one
thing.  While that may be an exaggeration, it's clear that the high
standards for getting patches applied discouraged many potential
contributors, and the "patch pilot" program was extremely useful in
getting contribution rates up.  Whether a patch pilot program would be
a good idea for Emacs is a separate question, but note that the core
dev on patch pilot duty spent 25% of their time (full-time employees
of Canonical, most of them) on that task.  Not to mention the time the
contributors spent on meeting the Bazaar quality standards.

Does Emacs really need that high quality?  After trying to get
standards raised at XEmacs (where, being Number 2, we have to try
harder), I've come to the conclusion that for Emacsen, worse is
better.  The factors that led Richard Gabriel to that conclusion are
quite strong in this area of application.  I think rather than
pressure people about testing, code reviews are more important.  Teach
people to habitually write good, maintainable code.  That's the cake,
tests are the icing.

That said, I encourage the core developers to lead by example,
creating tests, even trivial ones, whenever possible.  Today's general
programming culture is very sympathetic to testing, even to the point
of "write tests first".  So new developers will pick up the idea from
what the leaders do.  At XEmacs, regulars (even those not associated
with core development) do submit tests regularly without being asked.
When people post a good replication recipe, we try to put it in the
test suite immediately, crediting the poster, not the committer.  (At
Emacs this might occasionally raise assignment issues, but remember,
it's all for freedom.)  So this practice works.

I also encourage you to get your buildbots green.  It's hard to stay
green in an active project, and as somebody mentioned about the XEmacs
bot for Gnus, some platforms will stay red for a long time because
nobody cares enough.  But I would guess that in XEmacs modal
turnaround for 'bot failures is 3-6 hours, with the tail stretching
out because of sleep and $DAYJOB periods.  (Part of this is because
you get lazy: testing only one platform, or committing "obviously
correct" fixes without testing locally, and the 'bot makes your
mistake obvious.  But laziness gets corrected by social pressure when
there are lots of active committers.)  Once the bots get green,
developers take pride in keeping them that way.

Óscar Fuentes writes:

 > I wonder about the possibility of making Emacs testing an "interesting"
 > task. Something like a test framework which is complete and expressive
 > enough to allow describing testable conditions on a very high level way.

That's going to be a very hard task.  Testing is inherently boring
work, exploring the things that shouldn't happen, and the burden is
heavy, because writing good tests requires good specs, which we rarely
have.  So designing those specs (and maybe even writing them down) is
a hidden burden of testing.

 > Having Lisp and an almost unlimited capability for state introspection
 > looks like a good start.

Lisp is good.  Unlimited state introspection sucks.  It's very easy to
write a test that fixes the implementation in stone when you use
introspection.  Good tests require that you distinguish between
language essentials (externally visible behavior) and implementation
accidents (internal state).  Sometimes you do want to test
intermediate results in an implementation to help debug maintenance
errors on the implementation, but these tests should be marked as
"subject to internal API changes".  This is the kind of thing that
makes testing a hard problem.

John Weigley writes:

 > One thing that would help with that is to allow structural pattern
 > matching.

It helps a little, but as Oscar says, we have Lisp.  Extracting values
from a structure should be part of the API for the structure, and
fixed patterns don't help when fuzzing or iterating over all values in
a sledgehammer test -- constructing the pattern becomes a PITA and
unreadable, even with backquotes.

Also, testing that reasonable input produces correct results (the
common case for such pattern-matchers) isn't so important.  More
important is providing facilities to test safety, that is to express
tests for correct behavior under stress (ie, whether Emacs signals an
erroneous condition, and whether the right signal is raised), and for
helping the buildbot stay green in the presence of bad behavior whose
fix is unknown and stays that way for a while (an ExpectedFailure
condition that people see as a failure when running the test, but the
'bot perceives as a pass == SNAFU).



reply via email to

[Prev in Thread] Current Thread [Next in Thread]