Re: testing framework and package.el

[Christian Ohler]

 On 4/10/10 6:22, Sebastian Rose wrote:
> A structure like that could be enforced later
> if needed.  We will start with a very small project.  I want to make it
> easy to jump to the test-file for a given source-file and execute tests
> for current defun automatically.  This could be done based on such a
> structure (i.e. conventions).
>
> I guess reading more of ERT's code I'll find other ways to find the file
> a certain test (or tests matching a selector) is defined.
I just started experimenting with a directory layout where ERT's code is 
in lisp/emacs-lisp/*.el and the tests are in test/lisp/emacs-lisp/*.el.  
It is indeed very clumsy to jump back and forth between the tests and 
the code under test.  (Putting the tests in test/emacs-lisp/*.el would 
make no difference.)

I suspect it would be easier if the convention was that a file 
/some/path/foo.el had its tests in /some/path/test/foo.el.  The 
convention of having a top-level src/ and test/ split makes sense in 
Java because of its package file layout and visibility rules, but Emacs 
Lisp has no such constraints.  Has anyone else thought about the pros 
and cons of these different conventions?  Is there a deep reason why 
Emacs currently has test/ as a separate top-level directory?


> > Lennart's code is already integrated in ERT, obsoleting ert2.  I remember that I
> > wasn't quite happy with his design, but didn't have time to fix it, and since it
> > solved an important problem, I included it as an experimental addition in
> > ert-exp.el.
> OK, good.  Found that, thanks.  From skimming over that file I guess
> that's what I was looking for and maybe even a bit more.
I just rewrote the test buffer functionality.  The design is now more 
orthogonal and flexible, but reading input from files will require an 
explicit call to `insert-file-contents' or similar.


> Org's exports to XHTML, LaTeX etc. We could compare them to control
> files.  The publishing mechanism might create quite big files and even
> entire file/directory trees.  Currently we publish our stuff and detect
> regressions when clicking a link that points nowhere or validating a
> XHTML page (spot checking).
I see.  Comparing directory trees may be important to org-mode, but 
overall, it seems like a feature that would be used by the tests of only 
very few packages.  I think you're better off writing some utility 
functions on top of ERT, as you have already started doing for other 
things.  Not every test-related utility function belongs in ERT itself; 
it's fine to build stuff on top.

The `ert-info' feature that I just implemented allows you to add custom 
context information that ERT will display on test failures.  It's not 
documented in the manual yet, but it makes it easier to build the kind 
of test utilities that you may want, so please check it out.


> > One limitations of my approach is that we really need some kind of
> > "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
> > for inspection if the test fails, but this is item 2.) on your list already.
> > Another limitation is that the explainer for `equal' is not as pretty as ediff,
> > but OTOH, as soon as you want to compare text properties, ediff can't help
> > anymore (AFAIK), while there is an explainer for
> > ert-equal-including-properties'.
> Also, I believe ediff is too interactive for a simple test run (diff's
> exit status is 0 if no difference is found - don't know yet if and to
> how ediff could be used non-interactively).
I have a rough idea now how to extend ERT's "explanation" feature to 
allow interactive invocation of ediff to visualize differences; this 
seems like a good way to solve this.  Tests would simply use (should 
(equal string1 string2)), and the explainer for `equal' would take care 
of the rest, putting a button in the results buffer that allows the user 
to invoke ediff.  string1 and string2 could be the result of calling 
`buffer-string' on different buffers, of course.


> > Text properties (e.g., font lock properties) are important to many kinds of
> > tests.  Both ediff and using files are problematic in such cases because they
> > don't support text properties.  Do text properties never come up in your
> > context?
> Some so some so I guess.  I didn't start to write actual tests yet.
> This will be done by many people (I hope) and there will be many
> different types of tests (I hope).  Some will involve text properties,
> too.
OK, that makes sense.  It is important to write actual tests, though; 
ideas for improvement that come from experience with real-world problems 
carry much more weight than thought experiments.

Christian.