[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: testing framework and package.el

From: Christian Ohler
Subject: Re: testing framework and package.el
Date: Wed, 13 Oct 2010 01:19:43 +1100

 On 12/10/10 14:02, Stephen J. Turnbull wrote:
Christian Ohler writes:

  >  AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the
  >  same repository.  Are you saying this has proven to be a bad idea?

Not bad; suboptimal.  There are two kinds of tests, tests of the LISP
language (including Emacs extensions described in the Lispref), and
tests of the internal implementation.

The former should be used for all versions of X?Emacs Lisp, and
therefore are conceptually separate from the sources.  They often need
to be versioned, but usually a boundp or fboundp test is good enough
for that.

The latter, of course, is implementation-specific and tied to the

In XEmacs practice, the two are mixed together, and as bugs are fixed
and features are added the tests for different versions have diverged,
even where the features are the same.  This means that, for example,
it's possible (and not uncommon, in fact) for 21.4 to be missing
regression tests that are present in 21.5 for 21.4 features.

I see, thanks for clarifying. Is it really mainly ELisp language features where this applies? Any test that I write now could make sense for the previous Emacs version, or across all Emacsen, if it's a shared feature. Probably we don't care about compatibility or regressions of specific UI features too much, so sharing such tests is perhaps not worth the trouble. Still, do you think there's a strict boundary between shared tests and implementation-specific tests, or a continuum? If it's a continuum, perhaps `ert-deftest' should have a :since keyword that takes some kind of structured value that specified the Emacs versions that are supposed to support the feature being tested. OTOH, if tests really mostly fall in one of two categories, a simpler solution would be sufficient.

  >  a way that these actions are as easy as possible.  I expect that
  >  running the current tests against older Emacs versions is something
  >  that we will want to do much less frequently, so we should not
  >  optimize for this use case at the expense of the others.

What's so hard about prepending "../" to a few Makefile variables?

I'm not sure I understand your question. I was trying to say that it would be easier to write and maintain tests if the tests for X.el were in X-tests.el in the same directory (or perhaps in a subdirectory tests/X.el), rather than in ../../test/*/*/X-tests.el, which is quite a bit further removed, and clumsy to navigate to (you can provide Emacs commands that help with this, but you really also need shell aliases, etc., and it's never going to be as simple as having code and tests next to each other). While browsing and editing the tests, you would benefit from having the tests right where the source code is; and since running them against an older Emacs version is comparatively rare, the more complicated find expression that you need to collect the test files makes little difference.

You're right that changing the convention is not too hard for the
framework, but getting tests to be backward compatible is going to be
annoying if you don't plan for it from the beginning.  In practice it
very likely won't happen, and that's unfortunate.

I think this is a good time to discuss concrete proposals to solve this problem.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]