[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: testing framework and package.el

From: Christian Ohler
Subject: Re: testing framework and package.el
Date: Tue, 05 Oct 2010 01:06:35 +0930

 On 4/10/10 6:22, Sebastian Rose wrote:
A structure like that could be enforced later
if needed.  We will start with a very small project.  I want to make it
easy to jump to the test-file for a given source-file and execute tests
for current defun automatically.  This could be done based on such a
structure (i.e. conventions).

I guess reading more of ERT's code I'll find other ways to find the file
a certain test (or tests matching a selector) is defined.
I just started experimenting with a directory layout where ERT's code is in lisp/emacs-lisp/*.el and the tests are in test/lisp/emacs-lisp/*.el. It is indeed very clumsy to jump back and forth between the tests and the code under test. (Putting the tests in test/emacs-lisp/*.el would make no difference.)

I suspect it would be easier if the convention was that a file /some/path/foo.el had its tests in /some/path/test/foo.el. The convention of having a top-level src/ and test/ split makes sense in Java because of its package file layout and visibility rules, but Emacs Lisp has no such constraints. Has anyone else thought about the pros and cons of these different conventions? Is there a deep reason why Emacs currently has test/ as a separate top-level directory?

Lennart's code is already integrated in ERT, obsoleting ert2.  I remember that I
wasn't quite happy with his design, but didn't have time to fix it, and since it
solved an important problem, I included it as an experimental addition in
OK, good.  Found that, thanks.  From skimming over that file I guess
that's what I was looking for and maybe even a bit more.
I just rewrote the test buffer functionality. The design is now more orthogonal and flexible, but reading input from files will require an explicit call to `insert-file-contents' or similar.

Org's exports to XHTML, LaTeX etc. We could compare them to control
files.  The publishing mechanism might create quite big files and even
entire file/directory trees.  Currently we publish our stuff and detect
regressions when clicking a link that points nowhere or validating a
XHTML page (spot checking).
I see. Comparing directory trees may be important to org-mode, but overall, it seems like a feature that would be used by the tests of only very few packages. I think you're better off writing some utility functions on top of ERT, as you have already started doing for other things. Not every test-related utility function belongs in ERT itself; it's fine to build stuff on top.

The `ert-info' feature that I just implemented allows you to add custom context information that ERT will display on test failures. It's not documented in the manual yet, but it makes it easier to build the kind of test utilities that you may want, so please check it out.

One limitations of my approach is that we really need some kind of
"with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
for inspection if the test fails, but this is item 2.) on your list already.
Another limitation is that the explainer for `equal' is not as pretty as ediff,
but OTOH, as soon as you want to compare text properties, ediff can't help
anymore (AFAIK), while there is an explainer for
Also, I believe ediff is too interactive for a simple test run (diff's
exit status is 0 if no difference is found - don't know yet if and to
how ediff could be used non-interactively).
I have a rough idea now how to extend ERT's "explanation" feature to allow interactive invocation of ediff to visualize differences; this seems like a good way to solve this. Tests would simply use (should (equal string1 string2)), and the explainer for `equal' would take care of the rest, putting a button in the results buffer that allows the user to invoke ediff. string1 and string2 could be the result of calling `buffer-string' on different buffers, of course.

Text properties (e.g., font lock properties) are important to many kinds of
tests.  Both ediff and using files are problematic in such cases because they
don't support text properties.  Do text properties never come up in your
Some so some so I guess.  I didn't start to write actual tests yet.
This will be done by many people (I hope) and there will be many
different types of tests (I hope).  Some will involve text properties,
OK, that makes sense. It is important to write actual tests, though; ideas for improvement that come from experience with real-world problems carry much more weight than thought experiments.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]