Re: [Suggestion] GNUstep-test for quality control

[Stefan Urbanek]

On 2003-10-15 22:58:38 +0200 Lars Sonchocky-Helldorf 
<Lars.Sonchocky-Helldorf@bbdo-interone.de> wrote:

> On 15.10.2003 20:58:37 Stefan Urbanek wrote:

<snip>

> > Simplier solution: What about lots of users as a 'testing framework and
> quality
> > controll'? Advantage of 'user based testing framework' is that it tests 
> > continualy. That 'framework' should also contain users, that were marked
> as
> > 'kind of users that gnustep does not need' in a gworskpace thread few
> days ago
> > on this list. More users means more testers and it does not matter what
> kind of
> > users. At this time and state,  gnustep needs any kind of users. More
> eyes see
> > more.
> >
> > I think, that if at least half of the effort devoted to writing test
> suites (i
> > am not against them) is also devoted to make gnustep more attractive, it
> can
> > also help to remove bugs and have a quality controll.
>
> That's a circular dependency: less bugs mean more users, more users mean less 
> bugs. But the flipside is also true: lots of bugs mean less users, less users 
> mean lots of bugs :-(. Currently the situation tends more to the latter...

Yes, and we need to get out of it.

> The problem with a "unorganised user based testing" approach is that it needs 
> committed, disciplined users to work. That's something you can't take for 
> granted. My experience tells me instead that far most users simply turn away 
> if things don't work out of the box: You even can be thankful if about one 
> percent fire their mail client up and send you just complaints, the users 
> that are able to give you a qualified bug report you can count with the 
> fingers of both hands. Another result introduced by the nature of your 
> approach is that you'll get many reports for easy bugs you already know, but 
> virtually nobody is willing to invest a lot of time to track down a bug which 
> a.) occurs irregulary or b.) involves a lot of work. In the end it is like in 
> any chaotic multi agent system: the path of least resistance is chosen. The 
> already long history of GNUstep is another proof for this.

True, users go away if it does not work. So what? We should at least have the 
first-user-experience parts bug-free, to let them go in and start play with the other 
stuff. Like a trap :-) You need a trap, hook or attraction to get users. Is there any in 
GNUstep? "The great framework desing with no alternative? Yes, but ... it does not 
work (as expected), so what?" You get the point.

About the discipline and organisation: you need that and you do not. Point is, that 
currently there is no need to fix bugs. Why? Because noone wants them to be fixed, or 
there are just a few people that want them to be fixed. So why waste an energy? More 
users means more pressure, and perhaps a chance that there is someone who will likely 
want to fix it. This is something about that famous "it works on my machine, so why 
bother?". True, why to bother, if there are just one or two users with this 
behaviour and noone else complains?

> You make a mistake when you think that the "effort devoted to writing test 
> suites" is basically lost time and done just for respectability. Let me tell 
> you: it is different. Any amount of time invested in a proper test pays off. 
> For a long time I also was a opponent of unit testing, I just saw no benefit 
> from it, I resisted doing it and left it to the other guys here at work. But 
> one day I *had* to do it and it came out to be much less complicated then I 
> had feared. In fact I realized that I was so reluctant towards it because it 
> was unknown to me. Today I have the habbit to test early and often. 
> Unit-testing just helps you to make rapid progress while coding. I can tell 
> you that I discovered lots of bugs, even 'thinkos' in my code doing testing 
> (often borderline cases where some special treatment for certain values was 
> needed) that would have gone undiscovered and would have caused bugs hard to 
> track down later on (if you build on a faulty foundation you building will 
> most likely be faulty to). I wouldn't start a project (even a private) 
> without unit testing today.

No, I haven't said that the effort devoted to writing test is lost time, just that at 
least half of the time should be devoted to gaining more users. This is not a commercial 
software for particular requestor where you already have a customer, this is free 
software where  "customers" are not certain. And if you write a SW that is 
really perfect (according to tests), why is it good for if it is for noone just those who 
developed it? Then the SW will die. I'm saying that testing by users can be complimentary 
to testing with a testing framework. GNUstep needs both. Now a question is, what is 
easier to get at this time?

Btw., to paraphrase your last question ... Would you start a project without 
users (even potential) today? Are there going to be any future users with this 
state and attitude of GNUstep?

Moreover, there are functionalities that can be tested only by users, so saying 
that users are not good for testing is not a way to go.

To sum it up:
- user testing is complimentary to unit testing
- more users will result in greather pressure/need of bug fixing
- to get more users, there should be a trap/hook/attraction - is there any?
- without users, project dies with its developers

Have a nice day,

Stefan Urbanek
--
First they ignore you, then they laugh at you, then they fight you, then you 
win.
- Mahatma Gandhi