Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <address@hidden> writes:
> On 28/09/10 8:57, Stefan Monnier wrote:
>>> is there a chance to get a testing framework on elpa.gnu.org on the
>>> short run?
>>> Ert [1] is used in nxhtml, simple to use and documented.
>>
>> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
>> itself, and then try and move the few tests we already have
>> scattered about to use ERT.  But I haven't heard much about it recently.
>
> Incidentally, I finished a major overhaul of ERT's manual a few days ago.  
> Over
> the next few weeks, I should be able to find some time to put together a patch
> to integrate it into Emacs.  If anyone is interested, you can already review 
> the
> code at http://github.com/ohler/ert and send me comments to speed up the
> integration process.
>
> I think it makes more sense to have ERT in Emacs proper rather than as an ELPA
> package.  This way, tests for Emacs itself can use ERT.  Ulf Jasper's tests in
> the test/ directory would be the first candidates.
>
> Christian.


Yes, that would make sense.


Christian, I have a few ideas/feature requests.


1.)  Make testing a matter of one key-stroke

To ease testing, I've written two little commands, as you can see on
http://github.com/SebastianRose/org-test

  `org-test-test-current-defun' and
  `org-test-test-buffer-file'

The idea is to search the directory structure from the current source
file upwards for a directory named "tests/" if it exists.  Else ask the
user.  Similar to what `add-change-log-entry' does.

Below that directory, a tree like the source tree exists:

project
   +-- lisp/
   |     +-- a.el
   |     `-- b/
   |         +-- b.el
   |
   `-- tests/
         +-- a.el/
         |     +-- tests.el
         |     `-- a-defun.el
         `-- b/
             +-- b.el/
                   +-- tests.el
                   `-- b-defun.el

If this setup exists, when editing defun-x in lisp/a.el,
`M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
(fallback: tests.el there) and execute all tests with selector
"^a-defun".
`M-x org-test-test-buffer-file' in that same source file will load all
*.el files in tests/a.el/ and execute all ERT tests for selector "^a".

This would   a) recommend a certain structure for testing (what ever the
structure will be) and   b) make testing a key-stroke.



2.)  Create temp buffers for output

One thing I miss is a way to create output buffers automagically (I
didn't look deeper into the sources yet).

I know Lennart Borgman has written something for nXhtml.  It's called
ert2.el and part of the nXhtml distribution (but I did not comprehend
what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
will help here or even care for that addition?


3.)  Compare output to control files

ediff the contents of those output buffers with control files.  We need
this feature to test org's exports.  No one will find the time to sit
there and search hundreds of output files for errors "by hand" (once
tests exist).



Thanks for all your efforts

   Sebastian