Re: quality assurance (was: Re: GNUstep Base 1.11.0)

[Richard Frith-Macdonald]

On 2005-07-27 02:02:56 +0100 Alex Perez <aperez@student.santarosa.edu> wrote:

> Lars Sonchocky-Helldorf wrote:

> > How about using the donation money GNUstep got for buying some testing 
> > machines which would go into the same place as the GNUstep web servers are 
> > and could then be used for testing/developing GNUstep onto some more 
> > "exotic" platforms: Some *BSD, some Solaris boxen (be it x86 or SPARC), a 
> > Mac (Mac mini should be enough), maybe some HPUX box should be enough for 
> > the start. Those machines don't necessarily need to be new hardware, I 
> > guess we could get them used somewhere for cheap. Then put them online 
> > with 
> > special ssh accounts (maybe some VNC too) for the core devs and there you 
> > go: compilation and testing is possible remotely for those who need it.
>
> In a word, no. I think that would be a huge waste of the money, because we 
> can get access to these kinds of exotic machines through places like 
> sourceforge et al. who have large compile farms for EXACTLY THIS REASON.
>
> Why is their offering insufficient for us? (Seriously)

I know nothing about what sourceforge offer, so I'm talking in pretty 
general terms here ... but IMO lack of man(woman)power is our biggest 
problem ... and to make good use of  more machines/operating systems we 
would need to be able to work with them in a very automated way.  We don't 
have the time to manually log on to lots of machines and build/check every 
code change on all systems/configurations even if we have the hardware and 
operating systems available to us.

Of course, they would be useful for people to log on to and debug system 
specific problems, but in order to screen for problems in the first place I 
think, we would need to develop scripts which would run 
automatically/periodically to ....

1. retrieve the latest CVS code.
2. build it and run tests on it in various configurations.
3. in the event of a success, log it somewhere so that our website can 
display status.
4. in the event of a failure of a build or a test failure in any 
configuration, wait a while and try CVS again
5. after some period in which failures are consistent, email out alerts.

I imagine this could keep the a machine occupied a lot of the time (if we 
are going to have it testing a lot of different configuration options).  My 
guess is that it would not be acceptable usage for a public shared resource 
at sourceforge.