[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: testing framework and package.el

From: Christian Ohler
Subject: Re: testing framework and package.el
Date: Sun, 03 Oct 2010 21:51:18 +1100

 On 2/10/10 13:53, Sebastian Rose wrote:
1.) Make testing a matter of one key-stroke
To ease testing, I've written two little commands, as you can see on

   `org-test-test-current-defun' and

The idea is to search the directory structure from the current source
file upwards for a directory named "tests/" if it exists.  Else ask the
user.  Similar to what `add-change-log-entry' does.

Below that directory, a tree like the source tree exists:

    +-- lisp/
    |     +-- a.el
    |     `-- b/
    |         +-- b.el
    `-- tests/
          +-- a.el/
          |     +-- tests.el
          |     `-- a-defun.el
          `-- b/
              +-- b.el/
                    +-- tests.el
                    `-- b-defun.el

If this setup exists, when editing defun-x in lisp/a.el,
`M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
(fallback: tests.el there) and execute all tests with selector
`M-x org-test-test-buffer-file' in that same source file will load all
*.el files in tests/a.el/ and execute all ERT tests for selector "^a".

This would   a) recommend a certain structure for testing (what ever the
structure will be) and   b) make testing a key-stroke.
This structure looks like it is optimized for many test files for each file of non-test code. Right now, Emacs has 1735 .el files, and 35 files total in the tests/ directory. This ratio is not likely to invert any time soon. The structure that you describe could work, but isn't it overkill?

2.)  Create temp buffers for output

One thing I miss is a way to create output buffers automagically (I
didn't look deeper into the sources yet).

I know Lennart Borgman has written something for nXhtml.  It's called
ert2.el and part of the nXhtml distribution (but I did not comprehend
what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
will help here or even care for that addition?
Lennart's code is already integrated in ERT, obsoleting ert2. I remember that I wasn't quite happy with his design, but didn't have time to fix it, and since it solved an important problem, I included it as an experimental addition in ert-exp.el.

3.)  Compare output to control files

ediff the contents of those output buffers with control files.  We need
this feature to test org's exports.  No one will find the time to sit
there and search hundreds of output files for errors "by hand" (once
tests exist).
What do you mean by "sitting there and searching hundreds of output files for errors"? Do you have an example of tests where ERT's error reporting is too unspecific to be useful?

I remember writing some tests roughly like this:

(ert-deftest foo ()
    (insert "initial buffer contents")
    (... invoke code under test ...)
    (should (equal (ert-filter-string (buffer-string) ...)
                   "expected final buffer contents"))))

Are you sure that you want the initial/expected buffer contents in their own files? Such a split between the inputs and the invocation of the code under test seems cumbersome. I guess if your inputs/expected outputs are complex enough, moving them out into files can make sense. Can you point me to some examples where you think the data is too complex to be in-line?

One limitations of my approach is that we really need some kind of "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around for inspection if the test fails, but this is item 2.) on your list already. Another limitation is that the explainer for `equal' is not as pretty as ediff, but OTOH, as soon as you want to compare text properties, ediff can't help anymore (AFAIK), while there is an explainer for `ert-equal-including-properties'.

Text properties (e.g., font lock properties) are important to many kinds of tests. Both ediff and using files are problematic in such cases because they don't support text properties. Do text properties never come up in your context?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]