Re: quality assurance (was: Re: GNUstep Base 1.11.0)

[Riccardo]

Hello,

On Saturday, July 23, 2005, at 07:29 PM, Richard Frith-Macdonald wrote:

> 1. ask people to volunteer to test the odd configurations they have the 
> machines for.  If we had a list of people who actuyally used the less 
> popular hardware, operating systems, and configuration/build options, 
> we could do an email reminder asking them to test the latest cvs code 
> just before a release (even better if they could do automated 
> builds/tests on a daily basis, I think someone was working on that)

I think I qualify here for this... given my large number of stranger 
platforms I try to run gnustep on. I would like to clarify a couple of 
points.

- it is time consuming, especially on odder configurations which may 
require some tweaks (it reminds me of openbsd which needs a 
additional_include hack everytime to find png.h) or plainly the platform 
I have is slow because it is old
- not reporting a bug is time consuming too. something simple like a 
typo is obvious, but if things don't work at runtime for example, just 
reporting "it doesn't work" is useless often. When I do that I try to 
provide further data, like if the previous release worked fine or what 
triggers it... this is sometimes just a pain or sometimes out of my 
technical reach and thus I need to try to contact some more expert 
people on IRC and there may be TZ differences...

the above are excuses for the "slowness" of the process but here are 
some additional challenges:
- if it is broken on a strange platform and the problem is not trivial, 
it may be difficult to fix for someone that hasn't access to them. I may 
not know how to fix the problem myself so these kind of bugs may remain 
in the bugtracker forever (I think of some openbsd bugs for example)
- currently some platforms are broken for me at low level: -base isn't 
working due to ffcall problems for example on netbsd/sparc. Thus I can't 
test that box at all. Hurd is in a similar situation with threads.
- the particular platform I use to run a certain OS may have problem on 
my setup. SInce I upgrade my CPU on my PPC box, NetBSD has become highly 
unreliable and crashes often. It is almost certainly a netbsd kernel 
problem and thus not "gnsutep's fault" but it renders my workstation 
less usable and thus I use other platforms. Being ppc and sparc the two 
boxen I run netbsd on, netbsd testing lessened a lot since I gather few 
other people use it. The same goes for my gnu/hurd setup that evidently 
decided to kill my filesystem... doing that I discovered that fsck for 
ext2 is buggy and can segfault is the disk is in bad shape. so one bug 
discovers another, this is life in free software. But of course it 
steals time and being gnustep for me only a hobby... the consequences 
are obvious.

the above problems are also due to the small size of our community. If 
we had more testers and developers, things were "on average" in better 
shape.

a last comment si that tracking CVS is sometimes dangerous since it may 
render your system unreliable or not usable. Thus while I decided that 
my netbsd/ppc box is a "test" box living dangerous, the experience with 
the FreeBSD breakage will make me more cautious about blind updates even 
if they seem trivial like minor changes in the build system.


I just wanted to expose some experiences and clarify some problems. It 
is easy to say "we should, we have to" and I too say this sometimes 
about QA, but implementing it may be more difficult in our small gnsutep 
community; cheers
   R