[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Fsfe-france] Savannah future
[Fsfe-france] Savannah future
Fri, 5 Oct 2001 14:34:39 +0200
In the past months various discussions about the future of
Savannah designed a solution to the problems we currently have:
- code impossible to maintain/upgrade
- no internationalization
- no packaging
The core of the problem comes from the unmaintainable state of
the SourceForge code. Re-writing the whole thing is a lot of work, too
much if you ask me ;-) However, there is phpGroupWare
(phpgroupware.org). It implements most of the functionalities we need
and use on Savannah. In addition it provides an xmlrpc interface that
will be a relief for people who think that HTML based interaction
rms once told me : "why is there enough manpower to re-write
Savannah when we lack the manpower to do simpler things such as adding
a bug tracking facility ?". Apparently nobody is willing to invest
time on the current code base. When we implemented the first instance
of Savannah at the beginning of the year the goal was simple: reduce the
workload of address@hidden to nothing by allowing project
maintainers to manage the membership themselves. In that respect it is
a success: in the past 6 month noone had to manually add/remove
accounts for CVS access purpose, nobody had to wait for days for an
overloaded cvs-hackers to spend the necessary 5 minutes to handle the
Although no official announcement was made about Savannah, the
ambitions slowly grew and people are now expecting it to be a fully
functional development hosting facility. In addition, numerous
development hosting facilities were started and an increasing number
of people, companies, universities and governments were willing to
have their own. I think we now have a plausible working plan satisfy
everyone and the necessary software complete an implementation and
install it on Savannah by the end of the year. It can be done by a few
(4) motivated individuals, it can become truly general purpose if all
people I've been talking to actually contribute to it, it can become a
killer application if a large number of contributors come on board.
When thinking about this plan I searched for solutions that
require a minimal work. Nobody I know has enough energy to undertake a
major implementation for a single component, i.e. more than six month
full time. Each task can be implemented within a few days, assuming
someone has the underlying knowledge (learning xmlrpc may take a while
The first bits of this project are available at:
deb http://france.fsfeurope.org/debian/ ./
deb-src http://france.fsfeurope.org/debian/ ./
IRC channels where related things are discussed:
I'll be using address@hidden as the primary mailing list,
cc'ing mails when appropriate when discussions occur elsewhere.
Loic Dachary http://www.dachary.org/ address@hidden
24 av Secretan http://www.senga.org/ address@hidden
75019 Paris Tel: 33 1 42 45 09 16 address@hidden
GPG Public Key: http://www.dachary.org/loic/gpg.txt
Any given machine running &NAME; can host a list of
projects (with pointers to the machines hosting the
projects), read-only copies of projects (with pointers
to the machines hosting the read-write instance of the
project), read-write projects.
The read-only projects are mirrored from the machine
hosting the read-write instance of the project or other
machines hosting a read-only copy. The format used is
based on XML and defined by CoopX (http://coopx.eu.org).
The number of read-only and read-write projects hosted
on a given machine is controlled by the maintainer of the
machine, depending on the available resources. For instance,
an average machine with lots of bandwidth could host at most
one hundred read-write projects and ten thousand read-only
At a given time, only one machine hosts a read-write
project. It is the responsibility of the project
maintainers to ensure this. Should a machine become
unavailable for some reason, the administrators of the
project can toggle the read-only flag of one of the
read-only mirrors and start using it as a
Such a decentralized setup is not the most efficient
replication system but it is definitely simple and
requires very little effort.
A single machine may not be powerful enough for someone
willing to host a large number of projects. Planning a
multi machine setup in advance is mostly impossible
because many strategies can be implemented depending on
the needs. For instance a site like SourceForge will not
implement the same strategies than a site like Geocities
because the usage pattern is not the same.
A Debian package repository will be used to store the
packages that implement various strategies. They can all
coexist together. Let's say, for instance, that the
package &NAME;-single implements a single machine setup
and depends on exim, cvs and bind. There can also be a
&NAME;-multi-cvs, &NAME;-multi-exim, &NAME;-multi-bind,
each installed on a separate machine and that configure
the corresponding service for a multi-machine setup.
Every bit is packaged for Debian. Although it may be
desirable to package it for other GNU/Linux distributions,
this is not our goal. The cooperative nature of Debian and
the package dependency management are essential. When the
set of packages reach a stable state it will be possible
to contribute them to Debian without a need to find an
agreement with a company. The dependency management allows
to consistently use a large set of packages.
The packages will be produced in a source specific to the
project. This source (currently
http://france.fsfeurope.org/debian/) contains three types
of packages: copy of existing Debian packages slightly modified
for the needs of &NAME;, packages of unfinished software written
for &NAME;, packages whose sole purpose is to bind a set of
packages together and configure them.
The modified copy of existing Debian packages are only stored
while waiting for the Debian maintainer or the upstream of the
software to integrate the change. It is never meant to be
permanent. Communication is always established with the Debian
maintainer and the upstream before the modified copy is built.
Should we fail to do that, &NAME; will become a concurrent Debian
distribution and this is to be avoided at all cost.
The software specific to &NAME; is packaged at a very
early stage. Basically the package is done whenever the
software performs at least one task. Properly packaging a
software takes a lot of time and failing to do it from the
very beginning is asking for troubles. It also allows
developpers to use the software and contribute to it in an
organized way. Finally, since &NAME; is a collection of
many packages it allows to debug and architecture their
interdepencies while writing code.
Templates and internationalization.
User visible content is stored in tempates. The preferred
format for HTML pages is XHTML.
Every string displayed is stored using gettext files and
obeys the system locale.
Each release provides an upgrade procedure from the
previous version. The upgrade may imply a planned
down-time, &NAME; is commited to provide an upgrade
facility that would work without stopping the service. The
XML dump of the database will be used to upgrade the data
using XSL files. The database structure itself is upgraded
with a set of SQL orders. The associated services (smtp,
cvs, etc.) are reset to their initial state (empty) and
the backend scripts are run on the upgraded database to
restore the desired state.
Export/Import using CoopX.
CoopX provides a set of XML based schemas to represent all
the data related to a project. This includes the description
of a user (P3P), contact information (XMLvCard) and custom
made schemas for the CVS tree and the bug tracker information
The goal of CoopX is not to promote a standard but to use
them where possible. The bug tracker information, for
instance, is codified using a DTD with some documentation
to understand it. There are no plans to submit it to W3C
or other similar organizations.
The CoopX format is used for upgrades and for communication
between distinct &NAME; platforms.
The motto is: keep it simple, stupid. Inventing complex
solutions, over-generalization, implementing features for
future use are common pitfalls that lead to unnecessary
complex software. Finding a simple solution often requires
more efforts than accepting a complex one.
The myth of a perfectly modular software is hard to
kill. Modularity is an advantage and a burden. Deciding
that a feature will live in a module implies that we have
to define and maintain an interface for it. This is a
significant effort that has to be rapidly compensated by
the actual use of various modules. If the perspective of
using more than one module is too far in the future, we
might as well not bother about it. We need modules but we
should not abuse them.
The following items are implemented, all optionals except for
the authentication system and the import/export functions.
Authentication: anyone can create an account with
associated personal data such as full name and public keys
(P3P). A logged in user can create a group (or project but
in term of authentication it means creating a group to
which users can be added). A set of permission bits is
granted to each application by the authentication system,
the application decides which semantic a specific bit
has. A set of users within a group have the ability to
modify the permission bits. (priority 0).
Export and import projects using the CoopX XML based
format that contains all the data necessary to move
projects from one platform to another. (priority 0).
CVS with ssh/kerberos/gserver/pserver access (priority 1).
WebCVS, ViewCVS (priority 1).
Tracker declined in support request, bug tracking, patch manager
and task manager (priority 1).
Forum (priority 2).
Shell account (priority 2).
Poll (priority 2).
HTML pages managed from CVS (priority 1).
HTML pages managed from shell account (priority 2).
Mailing lists (priority 1).
Calendar (priority 2).
Contact database (priority 2).
Statistics (priority 1).
DNS management (priority 1).
FTP (priority 1).
rsync (priority 1).
SQL database access (priority 2).
php (priority 2).
The core is the server that implements interactions with
the user and backend scripts. It can be used thru a web
based interface or an xmlrpc interface.
phpGroupWare is used as a code base. It provides
templates, authentication, interface to SQL database,
xmlrpc, session management, permission management with
groups and acl. The development version of phpGroupWare
(0.9.13) is preferred and close interaction with the
development team is maintained.
A set phpGroupWare applications are developped or
re-used to implement the &NAME; core. Architectural
decisions that require changes of the phpGroupWare code
base are first discussed with the development team. The
changes are carefully evaluated and reduced to the
minimal. The most important source of changes comes from
the fact that &NAME; is project oriented where
phpGroupWare is user oriented.
The &NAME; application implements the user registration,
project registration, project membership, project moderation,
activation/deactivation of other applications for the
project, the home page, searching projects and members,
export/import of projects, read-only/read-write toggle of
Each functionality available to a project (CVS, WebCVS
etc.) is implemented in the core by an existing
phpGroupWare application where possible, with a new application
if nothing is available. Forking an existing application is
not an option : coordination with the maintainer of the
application allows to implement the missing functionalities.
If an application is of general interest, its inclusion in
the phpGroupWare project is discussed before the application
Quick start, reference manual, user guide, implementor
The quick start is ten pages at most, including screen
shots of the installation process. The intended audience
is a GNU/Linux system administrator with basic knowledge
of the various services installed and an understanding
of the purpose of &NAME;.
The reference manual documents every component of &NAME;
and how they work together. It is the ultimate source of
information for anyone willing to operate, grow or fix a
machine running &NAME;. The intended audience is a
GNU/Linux system administrator with good knowledge of
the various services installed and advanced user of
The user guide explains in detail how to interact with a
machine running &NAME;, how to setup a project on it,
and how to manage the project. The intended audience is
a GNU/Linux user with basic understanding of the
GNU/Linux commands or a developer with basic
understanding of the GNU/Linux development tools.
The implementor guide won't have a fixed structure and
provides basic pointers to help volunteers find their
way in the project. Only when the project becomes mature
enough will the implementor guide be re-worked in a
structured way. The intended audience is a skilled
GNU/Linux system administrator and developer. All
documentation is based on DocBook and also available in
texinfo and man pages formats.
<title>Back end scripts</title>
Bind the authentication information in the database to
the system files/services. Import and export projects
automatically or upon request. Update, configure system
services for CVS, WebCVS/ViewCVS, trackers, HTML over
CVS, mailing lists, statistics, DNS, rsync, FTP,
according to the content of the database.
Backend scripts have two options to interact with phpgroupware.
Either thru the XML dump of the projects or thru xmlrpc.
Queries are never run directly on the phpgroupware database.
Services with priority two are not meant to be usable in
the first stage. Scripts are non intrusive as far as
authentication is concerned. It means that user accounts
and groups unrelated to &NAME; may be created manually
by the system administration and are not overriden by
All backend scripts are designed to be installed
independently, the only mandatory scripts being
authentication and import/export. Libraries common to
all scripts is developed to factorize and normalize
interaction with phpgroupware. All scripts are designed
to be able to run on a machine that is not the same
machine as other services.
Each system service is installed and configured so that
it can be used by the backend scripts. This may involve
to modify/customize the original distribution in some
cases such as WebCVS/ViewCVS, mailing lists, kernel to
allow more group by users + services coping with glibc
lossage, DNS to access the database instead of a script,
chroot'ed environment for security, reverse proxy
ability for FTP service etc.
Such modifications are packaged and generalized in such
a way that they can be accepted by the Debian developer
responsible of the package and the upstream maintainers.
Interaction with the Debian developer and the upstream
maintainer is done before modifying the software.
The whole set of packages needed to run a &NAME;
instance on a specific machine can be installed on a
Debian GNU/Linux unstable distribution using apt-get
install task-&NAME; and answering debconf questions on a
maximum of five screens. Prior to this, a line has to be
added to the sources.list files so that the necessary
modified packaged are found if needed.
The task-&NAME; package is included in the unstable
branch and must therefore be able to operate consistently
on the basis of unmodified packages.
The applications added to the code base, each backend
script and modified services have their own package. All
packages based on phpgroupware use xmlrpc to configure
phpgroupware so that no web based interaction is required
from the system administrator.
The rationale behind creating a source in addition to
the standard unstable is that it can contain packages
modified in a way that is completely specific to
&NAME;. Although this should be kept to the minimum it
won't be possible to avoid all cases. For instance a
kernel patched with NGROUPS_MAX raised to 512 may be of
no interest to Debian in general. Or a chroot'ed MySQL
server with a quota patch.
|[Prev in Thread]
||[Next in Thread]|
- [Fsfe-france] Savannah future,
Loic Dachary <=