Re: [Orgmode] Re: How you can help

[Sebastian Rose]

Hi Richard,


Richard Riley wrote:
> Sebastian Rose <address@hidden> writes:
>
> > Hi Richard,
> >
> >
> > Richard Riley <address@hidden> writes:
> > > > Added this one to the Clippboard section on new org-tests/index.org in
> > > > Worg.git. (this section will be temporary...)
> > > Something like the above should only be a link (at most) to the emacs
> > > manual. Reproducing standard info is bad in the long run in case things
> > > in the base product (emacs) change for example.
> >
> > On the long run, yes. I just added this section, to start the page. And
> > since it's just STARTing... this was in the first or second reaction
> > that came in. Feel free to edit ('go wild' ;-))!
> >
> > I wouldn't bother to have several pages like this one will be (it's
> > supposed to be an index on the long run), each covering another way of
> > testing.
> >
> > I just meant to take _ACT_. Scepticism is a good thing, as long as it
> > doesn't stop you from doing.
>
> Of course not. 

Glad you replied. While I speek a little English, I'm not alwas shure
how it _feels_ for native speakers. For example this one about
scepticism.


> > > > > Some kind of regression testing framework would be awesome.  Org-mode is
> > > > > large enough that this is almost a necessity to keep things stable and
> > > > > bug-free.
> > > > It's big and feature-RICH.
> > > The nature of OSS means that the community using the product keep it
> > > stable and bug free. I dont think the efforts to produce meaningful
> > > regression tests would be beneficial in an ever morphing product like
> > > org-mode. Clearly my humble opinion on that one :-;
> >
> > We do test our software by using it. But the bug in the HTML-exporter
> > that Carsten has fixed two days ago, was introduced in early September
> > and would be in 6.10, which is supposed to be in the emacs 23 release.
> >
> > A very simple test plan would have revealed it.
>
> The question is : define simple. I am not against test plans, but the
> nature of such means a process in place where such test plans are
> carried out. Carsten knows his SW inside out and would find it hard to
> apply test cases every time a bug is fixed for time alone. Bugs happen
> in SW and the nature (excuse the repetition) of OSS is that we, the
> users, are kind of the testers and users.

OK, simplicity.
That's an important point I'd like to make clear. I _want_ simplicity
in two ways:

  a) This here is _not_ meant to make Carsten work _more_, but to _help_
     Carsten and all contributors. We must keep that in mind, while
     searching for the right way to test.
     Not Carsten, but _we_ collect the bugs, _we_ write the
     tests (whithout interfering with Carstens code), _we_ setup the
     testenvironment, and _we_ finally _do_ the testing.

  b) I _want_ tests, that everyone can do. Nobody knows, who of us is
     still here in a year or two. I like the idea of automated testing,
     but I'd also like to get every org-user to test. Imagine to
     have org-test.el in contrib (split into modules maybe).

     (setq org/test-html-export-base-dir "~/org-test/data/")
     (setq org/test-html-export-publising-dir "~/public_html/org-test/")
     M-x org/test-test-html-export

     The whole thing automatically executed once a week/day/mounth,
     after every git-pull, <SCHEDULED +1d>, sending a backtrace on
     failure.


Once tests exist, we will find out how the interaction with Carsten
works best / where neccessary. I know it will be hard to adjust all
those tests to 'movin target' called Org.



> > > > > Maybe something can be put together from the git testing framework and
> > > > > use of emacs -batch to process test org files and verify the output is
> > > > > as expected (with diff or some other tool).
> > > >
> > > > Hey, diff is a good idea!!
> > > >
> > > > I didn't take the verification of the output into account yet :-)
> > > >
> > > > I just pushed a change of Worgs start page, and added a directory
> > > > 'org-tests'. I've placed an index.org there, which now is just a
> > > > collection of ideas (I'm on my day job, so I can't really work on it
> > > > now).
> > > >
> > > > Don't know how often the git repo is published.
> > > > Bernt and Ben, are you 'worgers' allready?
> > > >
> > > > Do you think it makes sense to add snippets and ideas to the new page in
> > > > Worg? I think while the list great to exchange ideas, it's good to have
> > > > a place, where all those ideas are destilled to one-liners.
> > > I must say I am dubious about this. It means, for the tests to be
> > > meaningful, that the output must be  a fixed format in base org.
> >
> > Why? If the test bails out 'ERROR', I will have to look for the
> > reason. If the format changed in a legal way => adjust the test.
>
>
> What is an error? And for whom? That the categories are one more tab to
> the right?

In this case:
if I have a project defined in org-publish-projects-alist,
and :base-dir contains a file /foo/bar.org,
and, when publishing,
this file goes to /bar.html (instead of /foo/bar.html),
it is definitively an ERROR. At least in my eyes :-)


> I know I am probably sounding a bit of a killjoy here but
> having been involved in closed source SW development with dedicated QA
> and testing teams it was hard enough then. Maybe I am just being a bit
> of an old curmudgeon here :-(

It will _never_ be that hard, since we are who we are :-D


> > > I doubt
> > > this will ever be the case. The presentation will fluctuate while the
> > > core information (dates, schedules periods etc) will remain pretty much
> > > constant.
> > >
> > > The majority of bugs that I see are often down to people misusing or
> > > using things in the base which are not fully explored. No amount of
> > > regression testing can cover things like that unless the regression
> > > tests include everyones customisations.

Yes, the customizations:

   M-x org/test-test-html-export

That's why I wrote (in another mai), we shouldn't urge users to use a
bug tracker. We would just fill the bug tracker with lots of useless
information (from the bug trackers point of view). I love the bug
tracking on the list the way it is.

> > Yes, because Carsten add features en masse :-)
> > I see the testing differently. In the first place we need THINK of
> > testing.
>
> I think we all do. And the main contributors do too.

I didn't.
I didn't even recognize the bug for a while, until I found, that one of
the published org files had the wrong contents. A simple test would have
revealed the bug.


> But it is
> unrealistic to expect a full regression test for each and every release
> unless there is someone willing to step up to the mark and take the time
> to perform them. I personally feel that enough take the "cutting edge"
> and report back before the "official" cuts. We are all beat testers. Its
> the nature of being an emacs and OSS user in the first place :-;


There will always remain parts of the system untested. It'd be
funny to calculate all possible combinations of emacs configuration,
emacs versions and OS version Org-mode is supposed to run on, org mode
configurations and invariants. Seems like a big big number :-D


But the number of invariants itself is not endless! Differences
between OSs are, from an elisp view, minor. Not all the possible 
combinations of configuration values are sensible. Those configs
could be added as needed (or simply left up to the users).

Shure, tests are just one piece of the puzzle.


But I'm so happy to see this thread grow, see new names here, see
people editing Worg, learn something myself - that makes me hope, we can
make the best out of it.



> > New Org-revision out?
> > Ahh, OK, I have to the HTML-exports, I want to be working.
> >
> > To do this, I need several different setups, several different data
> > directories (Org-files) and an easy way to test, that doesn't eat my
> > time and gives a result. The quality may vary, but ERRORs will be
> > detected for shure.
> >
> > Not one or mounths later - immediately.
>
> If you don't find it immediately then how often do you use the feature?
> Now I know I probably sound negative. I assure I am not.  Maybe the
> earlier word "sceptic" is more applicable ..

I use that feature nearly every day. But the project I export is quite
big and I happend to not visit one of the affected files - hm, maybe I
did, but the contents where not updated (the exporter doesn't delete
files in :publishing-dir - which is OK btw - a test should do that).

When I finally found something was wrong, I did an

rm -r ~/public_html/or-notes
rm ~/.org-timestamps/*

and exported again, I found it was a desaster. Many files missing or in
wrong directories (I think about 20%).

HTML export worked like a charme 6 mounth for me. Suddenly it doesn't
and none of us even noticed. A test for this should be easy to write.


> > If someone installs emacs 23, tries to export to HTML and gets an
> > error.... 
>
> I have been there too. In fact you found a bug that I left hanging for
> ages because I found a work around ( dont publish from anywhere other
> than the root of the project) - so thanks for your perseverance on that.
>
> I think the best thing people can do is keep a stable version and also
> test the CVS (oops) GIT version too very now and again.
>
> > > Do I think regression testing is important? Yes - in certain
> > > environments. But every time Carsten, you, myself or anyone else fires
> > > up org-mode we are already doing just that.
> > Yes, but we can do better, easier and more complete.
>
> Possibly. I look forward to seeing proposed solutions  and would gladly
> lend some time to applying them.


Ahh, Great! Thanks!


  Sebastian