Re: [Orgmode] A testing framework

[Martin Pohlack]

Hi Carsten, all,

On 21.03.2010 08:26, Carsten Dominik wrote:
> we had an earlier threads about testing frameworks:
> 
> http://orgmode.org/worg/org-tests/index.php
> 
> http://thread.gmane.org/gmane.emacs.orgmode/8759/focus=8775
> 
> http://thread.gmane.org/gmane.emacs.orgmode/8743/focus=8743

Very interesting, I just went through all this.

>From above material, two general approaches have been considered for
testing org-mode:
* Lisp based unit testing.
* Example-file based feature testing.

I am targeting the second approach here for now as I think it may
provide more benefit per time spent and would allow more people to
participate easily.

I hope that every emacs user will be able to write tests with my
framework.  Basically only an example test case and the expected
result should be need for tests, together with a minimal description
of the setup.

If only an example file triggering a bug is available, a corresponding
test can be crafted with minimal effort.


For now I have a minimal framework running with two tests
demonstrating expected behavior for folding and a bug in the version
I'm using.

I would like to enquire comments regarding the following open issues,
where I have no satisfying solution yet:

* I would like to start the test in something like a temporally stable
  setup, such that generated timestamps would be somewhat predictable
  (basically with the system clock set to 1.1.2005 20:00, or something
  like that).

  There are several levels where this could be achieved, ranging from:
  1. Experimental patches to the Linux kernel,
  2. Library wrappers using LD_PRELOAD around emacs to virtualize the
     time and gettimeofday system calls,
  3. Advised lisp functions for obtaining time stamps, and
  4. Post processing the test output to generalized time stamps.

  Currently, I'm going with 4, 3 may make sense if someone could
  identify the hopefully 2-3 functions to be advised.

  2 and 1 are probably too Linux specific and I'm not sure if 2 works
  with emacs.

  Comments?

* Currently I use a list of lisp functions for specifying the behavior
  to be executed in a test, for example:

  (org-todo)

  or

  (org-end-of-line)
  (org-cycle)

  This is not very intuitive for users.  I would rather like something
  like a recorded macro to execute.

  For example:

  "end, tab"

  Does anyone here has a tip how to specify a list of user-input
  actions in a file and play that back?

* For visual comparison I currently simply save the visible part of
  buffers.

  One may get a step further here and also capture properties, faces and
  the cursor position.  I remember vaguely that there is such a
  facility somewhat related to rich text in emacs, but wasn't able to
  find it again.  Do you guys know of such a thing?

> I think it would be great to have a testing framework and a
> lot of tests, but I do not have time to make this happen.
> Maybe you can try to get the people who were active in those
> threads and energize them?

I assumed everyone still interested in contributing to org-mode is
still lurking in this list.  So speak up if you are interested in
contributing to the tests :-).

Cheers,
Martin