[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Orgmode] Re: How you can help

From: Sebastian Rose
Subject: Re: [Orgmode] Re: How you can help
Date: Fri, 24 Oct 2008 17:39:23 +0200
User-agent: Mozilla-Thunderbird (X11/20080724)

Hi Richard,

Richard Riley wrote:
Sebastian Rose <address@hidden> writes:

Hi Richard,

Richard Riley <address@hidden> writes:
Added this one to the Clippboard section on new org-tests/index.org in
Worg.git. (this section will be temporary...)
Something like the above should only be a link (at most) to the emacs
manual. Reproducing standard info is bad in the long run in case things
in the base product (emacs) change for example.

On the long run, yes. I just added this section, to start the page. And
since it's just STARTing... this was in the first or second reaction
that came in. Feel free to edit ('go wild' ;-))!

I wouldn't bother to have several pages like this one will be (it's
supposed to be an index on the long run), each covering another way of

I just meant to take _ACT_. Scepticism is a good thing, as long as it
doesn't stop you from doing.

Of course not.

Glad you replied. While I speek a little English, I'm not alwas shure
how it _feels_ for native speakers. For example this one about

Some kind of regression testing framework would be awesome.  Org-mode is
large enough that this is almost a necessity to keep things stable and
It's big and feature-RICH.
The nature of OSS means that the community using the product keep it
stable and bug free. I dont think the efforts to produce meaningful
regression tests would be beneficial in an ever morphing product like
org-mode. Clearly my humble opinion on that one :-;

We do test our software by using it. But the bug in the HTML-exporter
that Carsten has fixed two days ago, was introduced in early September
and would be in 6.10, which is supposed to be in the emacs 23 release.

A very simple test plan would have revealed it.

The question is : define simple. I am not against test plans, but the
nature of such means a process in place where such test plans are
carried out. Carsten knows his SW inside out and would find it hard to
apply test cases every time a bug is fixed for time alone. Bugs happen
in SW and the nature (excuse the repetition) of OSS is that we, the
users, are kind of the testers and users.

OK, simplicity.
That's an important point I'd like to make clear. I _want_ simplicity
in two ways:

  a) This here is _not_ meant to make Carsten work _more_, but to _help_
     Carsten and all contributors. We must keep that in mind, while
     searching for the right way to test.
     Not Carsten, but _we_ collect the bugs, _we_ write the
     tests (whithout interfering with Carstens code), _we_ setup the
     testenvironment, and _we_ finally _do_ the testing.

  b) I _want_ tests, that everyone can do. Nobody knows, who of us is
     still here in a year or two. I like the idea of automated testing,
     but I'd also like to get every org-user to test. Imagine to
     have org-test.el in contrib (split into modules maybe).

     (setq org/test-html-export-base-dir "~/org-test/data/")
     (setq org/test-html-export-publising-dir "~/public_html/org-test/")
     M-x org/test-test-html-export

     The whole thing automatically executed once a week/day/mounth,
     after every git-pull, <SCHEDULED +1d>, sending a backtrace on

Once tests exist, we will find out how the interaction with Carsten
works best / where neccessary. I know it will be hard to adjust all
those tests to 'movin target' called Org.

Maybe something can be put together from the git testing framework and
use of emacs -batch to process test org files and verify the output is
as expected (with diff or some other tool).

Hey, diff is a good idea!!

I didn't take the verification of the output into account yet :-)

I just pushed a change of Worgs start page, and added a directory
'org-tests'. I've placed an index.org there, which now is just a
collection of ideas (I'm on my day job, so I can't really work on it

Don't know how often the git repo is published.
Bernt and Ben, are you 'worgers' allready?

Do you think it makes sense to add snippets and ideas to the new page in
Worg? I think while the list great to exchange ideas, it's good to have
a place, where all those ideas are destilled to one-liners.
I must say I am dubious about this. It means, for the tests to be
meaningful, that the output must be  a fixed format in base org.

Why? If the test bails out 'ERROR', I will have to look for the
reason. If the format changed in a legal way => adjust the test.

What is an error? And for whom? That the categories are one more tab to
the right?

In this case:
if I have a project defined in org-publish-projects-alist,
and :base-dir contains a file /foo/bar.org,
and, when publishing,
this file goes to /bar.html (instead of /foo/bar.html),
it is definitively an ERROR. At least in my eyes :-)

> I know I am probably sounding a bit of a killjoy here but
> having been involved in closed source SW development with dedicated QA
> and testing teams it was hard enough then. Maybe I am just being a bit
> of an old curmudgeon here :-(

It will _never_ be that hard, since we are who we are :-D

I doubt
this will ever be the case. The presentation will fluctuate while the
core information (dates, schedules periods etc) will remain pretty much

The majority of bugs that I see are often down to people misusing or
using things in the base which are not fully explored. No amount of
regression testing can cover things like that unless the regression
tests include everyones customisations.

Yes, the customizations:

   M-x org/test-test-html-export

That's why I wrote (in another mai), we shouldn't urge users to use a
bug tracker. We would just fill the bug tracker with lots of useless
information (from the bug trackers point of view). I love the bug
tracking on the list the way it is.

Yes, because Carsten add features en masse :-)
I see the testing differently. In the first place we need THINK of

I think we all do. And the main contributors do too.

I didn't.
I didn't even recognize the bug for a while, until I found, that one of
the published org files had the wrong contents. A simple test would have
revealed the bug.

But it is
unrealistic to expect a full regression test for each and every release
unless there is someone willing to step up to the mark and take the time
to perform them. I personally feel that enough take the "cutting edge"
and report back before the "official" cuts. We are all beat testers. Its
the nature of being an emacs and OSS user in the first place :-;

There will always remain parts of the system untested. It'd be
funny to calculate all possible combinations of emacs configuration,
emacs versions and OS version Org-mode is supposed to run on, org mode
configurations and invariants. Seems like a big big number :-D

But the number of invariants itself is not endless! Differences
between OSs are, from an elisp view, minor. Not all the possible combinations of configuration values are sensible. Those configs
could be added as needed (or simply left up to the users).

Shure, tests are just one piece of the puzzle.

But I'm so happy to see this thread grow, see new names here, see
people editing Worg, learn something myself - that makes me hope, we can
make the best out of it.

New Org-revision out?
Ahh, OK, I have to the HTML-exports, I want to be working.

To do this, I need several different setups, several different data
directories (Org-files) and an easy way to test, that doesn't eat my
time and gives a result. The quality may vary, but ERRORs will be
detected for shure.

Not one or mounths later - immediately.

If you don't find it immediately then how often do you use the feature?
Now I know I probably sound negative. I assure I am not.  Maybe the
earlier word "sceptic" is more applicable ..

I use that feature nearly every day. But the project I export is quite
big and I happend to not visit one of the affected files - hm, maybe I
did, but the contents where not updated (the exporter doesn't delete
files in :publishing-dir - which is OK btw - a test should do that).

When I finally found something was wrong, I did an

rm -r ~/public_html/or-notes
rm ~/.org-timestamps/*

and exported again, I found it was a desaster. Many files missing or in
wrong directories (I think about 20%).

HTML export worked like a charme 6 mounth for me. Suddenly it doesn't
and none of us even noticed. A test for this should be easy to write.

If someone installs emacs 23, tries to export to HTML and gets an

I have been there too. In fact you found a bug that I left hanging for
ages because I found a work around ( dont publish from anywhere other
than the root of the project) - so thanks for your perseverance on that.

I think the best thing people can do is keep a stable version and also
test the CVS (oops) GIT version too very now and again.

Do I think regression testing is important? Yes - in certain
environments. But every time Carsten, you, myself or anyone else fires
up org-mode we are already doing just that.
Yes, but we can do better, easier and more complete.

Possibly. I look forward to seeing proposed solutions  and would gladly
lend some time to applying them.

Ahh, Great! Thanks!


reply via email to

[Prev in Thread] Current Thread [Next in Thread]