[Savannah-cvs] [331] MailingLists: add new page and update existing pages

[assafgordon]

Revision: 331
          
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=331
Author:   agn
Date:     2017-04-18 00:50:30 -0400 (Tue, 18 Apr 2017)
Log Message:
-----------
MailingLists: add new page and update existing pages

Modified Paths:
--------------
    trunk/sviki/ListServer.mdwn
    trunk/sviki/MailSystem.mdwn

Added Paths:
-----------
    trunk/sviki/MailingListsInternals.mdwn

Modified: trunk/sviki/ListServer.mdwn
===================================================================
--- trunk/sviki/ListServer.mdwn 2017-03-04 22:39:37 UTC (rev 330)
+++ trunk/sviki/ListServer.mdwn 2017-04-18 04:50:30 UTC (rev 331)
@@ -1,18 +1,38 @@
-List Server
-===========
+Mailing Lists Server
+====================
 
+All gnu and nongnu mailing lists are processed on `lists.gnu.org`
+using the Mailman package. This machine is managed by FSF admins.
+Savannah admins have only non-root access and are able to perform
+limited mailing-list related administration tasks.
+
+This page describes commands to manually manage mailing lists
+by savannah administrators. See also:
+
+* [[MailSystem]] - various email and mailing-list related details
+  in Savannah and gnu systems.
+* [[MailingListsInternals]] - the interface between savannah's php
+  frontend and Mailman.
+
+
+General
+-------
+
 Some Mailman admin tasks need to be performed on the actual machine
 where Mailman is installed, while others, such as changing the password
 for a list, can be done via the web interface and the site-wide admin
-mailman password. See [[MailSystem]] for architecture details.
+mailman password.
 
-In order to do this you need to have shell access to the machine that
-runs the master Mailman installation.
+In order to do this you need to have shell access to `lists.gnu.org` -
+the machine that runs the master Mailman installation.  If you require
+such access, ask one of the other savannah admins to add your ssh
+public key.
 
 Once your ssh public key has been added, you can log in as the user
 'list'. You will be greeted with the bash prompt:
 
-    address@hidden list]$
+    $ ssh address@hidden
+    lists:~ $
 
 From here you can perform all sorts of Mailman administrative tasks,
 such as those discussed below.  See also: [[ResetListPassword]].
@@ -27,8 +47,8 @@
 -------------
 
 Project admins can create their own mailing lists through the normal web
-interface.  (What that web interface does is ssh to the list server and
-run `newlist`; see the [[MailSystem]] page.)
+interface.  (This results in an ssh connection to `lists.gnu.org`
+and `newlist` being run there; see the [[MailSystem]] page.)
 
 For lists whose names do not follow an allowed pattern, savannah-hackers
 must manually run `~/mycreate_list.pl` on lists, and then attach the

Modified: trunk/sviki/MailSystem.mdwn
===================================================================
--- trunk/sviki/MailSystem.mdwn 2017-03-04 22:39:37 UTC (rev 330)
+++ trunk/sviki/MailSystem.mdwn 2017-04-18 04:50:30 UTC (rev 331)
@@ -1,7 +1,7 @@
 MailSystem
 ==========
 
-See also [[ListServer]].
+See also [[ListServer]] and [[MailingListsInternals]].
 
 The principal GNU (and FSF?) email server is lists.gnu.org, managed by
 FSF sysadmin.  Savannah people and programs do not have root access.
@@ -36,27 +36,13 @@
 -   public archives are managed by mharc: `~mharc/mbox` and
     `~mharc/html` (check [[ImportMailingListArchive]]).
 
-Mailing list creation
----------------------
 
-I (Sylvain) basically installed sv_mailman on internal.sv.gnu.org,
-modified it to support mailman virtual hosts (and use the proper admin
-mail instead of address@hidden).  Then I wrote a fake
-`/usr/sbin/newlist` that connects to address@hidden via SSH and call
-an helper script there (`savannah_wrapper.pl`).  This happens via a
-Savannah public key in address@hidden's `authorized_keys` with a
-`command=` restriction.
+Mailing-List interface to Savannah's frontend
+---------------------------------------------
 
-There is also an empty `/usr/sbin/config_list` on internal that does
-nothing; we don't want what Savane currently does there.
+See [[MailingListsInternals]].
 
-The cron job runs on internal every five minutes, file
-`/etc/cron.d/sv`. The script being run is named sv_mailman.
 
-On lists.gnu.org, there is a script `mycreate_list.pl` which can be used
-to create lists by hand.  It duplicates what `savannah_wrapper.pl` does
-for Savannah actions.  (The scripts must be kept in sync by hand.  They
-should be merged so there is only one.)
 
 Valid sender
 ------------

Added: trunk/sviki/MailingListsInternals.mdwn
===================================================================
--- trunk/sviki/MailingListsInternals.mdwn                              (rev 0)
+++ trunk/sviki/MailingListsInternals.mdwn      2017-04-18 04:50:30 UTC (rev 
331)
@@ -0,0 +1,345 @@
+Mailing Lists Management in Savannah
+====================================
+
+All gnu and nongnu mailing lists are processed on `lists.gnu.org`
+using the Mailman package. This machine is managed by FSF admins.
+Savannah admins have only non-root access and are able to perform
+limited mailing-list related administration tasks.
+
+This page describes the interface between savannah's php frontend
+and the Mailman package running on `lists.gnu.org`. See also:
+
+* [[MailSystem]] - various email and mailing-list related details
+  in Savannah and gnu systems.
+* [[ListServer]] - manually running mailing-list related commands.
+
+
+
+Overview
+--------
+
+Savannah provides mailing lists to hosted projects.
+Project admins can create as many mailing lists as needed through
+the PHP web frontend.
+
+The general workflow is as follows:
+
+1. A project admin on savannah creates a new mailing list.
+2. Mailing list information is updated in the `mail_group_list` table
+   in the `savane` MySQL database.
+3. A cronjob on `mgt0` reads the `mail_group_list` table and determines
+   which operations are needed (e.g. list-creation, password-reset).
+4. The cronjob runs local scripts (on mgt0) for each operation.
+5. Each script proxies the commands over SSH to `lists.gnu.org`
+6. The script `scannah_wrapper.pl` on `lists.gnu.org` processes the commands
+   and runs the appropriate Mailman scripts locally on `lists.gnu.org`.
+
+
+
+Database
+--------
+
+The `mail_group_list` table contains the following fields:
+
+    mysql> select * from mail_group_list where list_name = 'sed-devel' \G
+    *************************** 1. row ***************************
+    group_list_id: 3272
+         group_id: 6711
+        list_name: sed-devel
+        is_public: 1
+         password: NULL
+       list_admin: 829
+           status: 5
+      description: Development discussion for GNU Sed
+
+To see the mailing-lists for a savannah project:
+
+    mysql> select list_name from mail_group_list,groups
+           where groups.unix_group_name = 'coreutils'
+           and groups.group_id = mail_group_list.group_id ;
+    +---------------+
+    | list_name     |
+    +---------------+
+    | bug-coreutils |
+    | coreutils     |
+    +---------------+
+    2 rows in set (0.00 sec)
+
+The meaning of the `status` field:
+
+    Status 0: list is deleted (ie, does not exist).
+    Status 1: list is marked for creation.
+    Status 2: list is marked for reconfiguration.
+    Status 5: list has been created (ie, it exists).
+
+The `password` field means:
+
+    NULL: default value
+    '1':  A password-reset was requested for this mailing list.
+
+The password field should not contain any real passwords,
+though the database does contain values besides NULL and '1'
+(TODO: find out how/why):
+
+    mysql> select count(*) from mail_group_list where password is not NULL and 
password != '1';
+    +----------+
+    | count(*) |
+    +----------+
+    |     1674 |
+    +----------+
+    1 row in set (0.00 sec)
+
+
+
+PHP web frontend code
+---------------------
+
+Mailing list administration is implemented in
+[./frontend/php/mail/admin/index.php](https://git.savannah.gnu.org/cgit/administration/savane.git/tree/frontend/php/mail/admin/index.php).
+
+The PHP code updates *only* the `mail_group_list` table in the MySQL
+database, and queues future actions by changing the `status` and
+`password` fields in the table.
+
+No other actions are directly performed by the PHP frontend code.
+
+The PHP code contains these additional comments:
+
+    This frontend php script sets status to:
+         0 if user deletes a list before the backend ever actually created it.
+         1 if user adds a list
+         2 if user reconfigures an _existing_ list (ie, status was 5)
+
+    The backend sv_mailman.pl script sets status to:
+         0 when a list is actually deleted
+         5 when a list is actually created
+
+    - when we create an alias, which mean someone was able, according to
+      group type restriction to add to his project a list that was already
+      inside the database, we add the list inside the database with a status
+      of 5, so sv_mailman does not try to recreate it.
+      In the worse case, if two persons creates the same list at the same
+
+    The field password will not contact real password, it will contain
+    '1' when the backend is supposed to reset it.
+
+
+Backend cronjob
+---------------
+
+The mailing list cronjob runs on `mgt0` (in the old setup
+before the 2017 migration to newer VM, the script was on `internal` -
+outdated wiki pages might still refer to it there).
+
+In `mgt0:/etc/cron.d/savane`:
+
+    PATH=...:/opt/savannah/bin:...
+    # New list creation
+    */5 * * * *     root    sv_mailman --cron
+
+The script is `mgt0:/opt/savannah/bin/sv_mailman`.
+
+The above script is the installed version, originating from
+`mgt0:/opt/savannah/savane/backend/mail/sv_mailman.in` (and also in
+git
+<https://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in>).
+If you want to modify the script, NEVER update the installed version
+in `/opt/savannah/bin`. ALAWYS update the source version in
+`/opt/savannah/savane/backend/mail/sv_mailman.in` then run `make
+instal` (and of course, check-in the updated version too).
+
+The script reads the `mail_group_list` MySQL table,
+and performs the required actions based on the `status` and `password`
+fields.
+
+The script runs the following local scripts (on `mgt0`):
+
+- `/usr/sbin/newlist` - creating new mailing list
+- `/usr/sbin/config_list` - configuring a list (currently a no-op)
+- `/usr/sbin/rmlist` - removing a list
+- `/usr/lib/mailman/bin/change_pw` - resetting a mailing list password.
+
+Sadly, the paths are hard-coded in `sv_mailman` (there's even a comment
+about how bad they are).
+
+On `mgt0`, symlinks were created to the actual scripts:
+
+    address@hidden:~$ ls -lhog /usr/sbin/{newlist,config_list,rmlist} 
/usr/lib/mailman/bin/change_pw 
+    lrwxrwxrwx 1 28 Apr 17 22:44 /usr/lib/mailman/bin/change_pw -> 
/opt/savannah/sbin/change_pw
+    lrwxrwxrwx 1 30 Apr 17 22:43 /usr/sbin/config_list -> 
/opt/savannah/sbin/config_list
+    lrwxrwxrwx 1 26 Apr 17 22:43 /usr/sbin/newlist -> 
/opt/savannah/sbin/newlist
+    lrwxrwxrwx 1 25 Apr 17 22:43 /usr/sbin/rmlist -> /opt/savannah/sbin/rmlist
+
+FIXME: remove the hard-coded PATHs from `sv_mailman`, and install an
+updated version.
+
+NOTE: These scripts are proxies to `lists.gnu.org` - they do not
+perform any local action on `mgt0`. Their names are likely left-over
+from the time the entire savannah system (including Mailman)
+was running on the same local machine.
+
+
+SSH Proxing to lists.gnu.org
+----------------------------
+
+The `newlist`/`rmlist`/`change_pw` scripts on `mgt0`
+proxy the needed commands to `lists.gnu.org`,
+where the Mailman package is installed.
+
+1. On `mgt0`, the script `sv_mailman` runs as root from cron-jobs.
+2. The `sv_mailman` runs one of the action scripts (e.g. `newlist`).
+   the script also generates random passwords if needed.
+3. The `newlist` script (on `mgt0`) connets with SSH to address@hidden
+   (this happens as user address@hidden).
+4. The corresponding pubkey in `lists.gnu.org:/home/list/.ssh/authorized_keys`
+   is defined as follows:
+
+        command="./savannah_wrapper.pl" ssh-rsa AAAAB3NzaC1yc2...
+
+5. When the `newlist` script from `mgt0` connects with SSH to `lists.gnu.org`
+   instead of getting a shell, the 
`lists.gnu.org:/home/list/savannah_wrapper.pl`
+   script is executed.
+6. This scripts parses limited input from STDIN, and if it contains
+   recognized commands and parameters, it runs the corresponding program
+   locally on `lists.gnu.org`.
+7. `sv_mainman` sends email notification to the administrators of the mailing
+   lists, with the newly generated random passwords.
+   (FIXME: emails are sent even if commands fail - there's no error
+   checking).
+
+Example: running the following on `mgt0` as user `root` will result in
+changing the mailing-list password on `lists.gnu.org`:
+
+    # ssh -T address@hidden <<EOF
+    COMMAND=change_pw
+    LIST_NAME=bug-datamash
+    PASSWORD=123456
+    EOF
+
+The corresponding code in `lists.gnu.org:/home/list/savannah_wrapper.pl` (
+lots of code omitted for brevity):
+
+    ...
+    my %vars;
+    while (<>) {
+        chomp;
+        my ($variable, $value) = split('=');
+        $vars{lc($variable)} = $value;
+    }
+    ...
+    if ($vars{'command'} eq 'change_pw') {
+       (system('change_pw',
+               '-l',  $vars{'list_name'},
+               '-p', $vars{'password'},
+               '--quiet')
+        == 0) or die "change_pw: $!";
+    }
+
+
+
+TODO: document the "virtual host" hack (e.g. allowing gnu/nongnu mailing 
lists).
+
+
+Logs
+----
+
+On `mgt0`, the log file is `mgt0:/var/log/sv_database2system.log`:
+
+    [sv_mailman] Sun Apr  9 01:50:01 2017 - starting
+    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
<Fossilet> config_list.
+    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
<Fossilet> reconfigured.
+    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
password was reset.
+    [sv_mailman] Sun Apr  9 01:50:02 2017 - Mail sent to address@hidden, 
address@hidden, address@hidden
+    [sv_mailman] Sun Apr  9 01:50:02 2017 - work finished
+
+On `lists.gnu.org`, the log file is 
`lists.gnu.org:/home/list/savannah_wrapper.log`.
+It contains all the passed parameters (and thus passwords in clear-text):
+
+    Tue Apr 18 02:52:06 2017
+    password = 12345
+    list_name = pretest-users
+    command = change_pw
+
+
+Manual walk-through example - changing password
+-----------------------------------------------
+
+When a project-admin on savannah requests password-reset
+on their mailing list, the `password` field is set to `1`.
+Equivalent command:
+
+    $ mysql savane
+    mysql> UPDATE mail_group_list SET password='1' where 
list_name='pretest-users' LIMIT 1;
+
+On `mgt0` the `sv_mailman` is run periodically from cron.
+Equivalent command:
+
+    $ ssh address@hidden
+    # cd /opt/savannah/bin
+    # ./sv_mailman --cron
+    New pretest-users password: foobar12345
+
+The log file `mgt0:/var/log/sv_database2system.log` will show:
+
+    [sv_mailman] Mon 17 Apr 2017 10:52:05 PM EDT - starting
+    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - List pretest-users password 
was reset.
+    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - Mail sent to address@hidden 
for list pretest-users.
+    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - work finished
+
+The `sv_mailman` generated a new random password and executed `change_pw`.
+Equivalent command (still on `mgt0`):
+
+    $ ssh address@hidden
+    # cd /opt/savannah/sbin
+    # ./change_pw -n pretest-users -p 123456
+    New pretest-users password: 123456
+
+The `change_pw` script connects to `lists.gnu.org`
+and transmits the commands as STDIN.
+Equivalent command (still on `mgt0`):
+
+    # ssh -T address@hidden <<EOF
+    COMMAND=change_pw
+    LIST_NAME=pretest-users
+    PASSWORD=123456
+    EOF
+
+
+The script `savannah_wrapper.pl` on `lists.gnu.org`
+reads the commands from STDIN and execute the needed
+programs.
+Equivalent command (on `lists.gnu.org`):
+
+    $ ssh address@hidden
+    lists:~$ which change_pw
+    /home/list/mailman/bin/change_pw
+    lists:~$ change_pw -l pretest-users -p 123456 --quiet
+    New pretest-users password: 123456
+
+
+
+
+Mailing list creation - FOR REFERENCE ONLY
+------------------------------------------
+
+**The Following section describes the original setup. It is kept for
+historical purposes**
+
+I (Sylvain) basically installed sv_mailman on internal.sv.gnu.org,
+modified it to support mailman virtual hosts (and use the proper admin
+mail instead of address@hidden).  Then I wrote a fake
+`/usr/sbin/newlist` that connects to address@hidden via SSH and call
+an helper script there (`savannah_wrapper.pl`).  This happens via a
+Savannah public key in address@hidden's `authorized_keys` with a
+`command=` restriction.
+
+There is also an empty `/usr/sbin/config_list` on internal that does
+nothing; we don't want what Savane currently does there.
+
+The cron job runs on internal every five minutes, file
+`/etc/cron.d/sv`. The script being run is named sv_mailman.
+
+On lists.gnu.org, there is a script `mycreate_list.pl` which can be used
+to create lists by hand.  It duplicates what `savannah_wrapper.pl` does
+for Savannah actions.  (The scripts must be kept in sync by hand.  They
+should be merged so there is only one.)