[Savannah-cvs] [SCM] Savane-cleanup framework branch, master, updated. dd789cf82fdc58a4cda42d66b6d38fd2121b11b7

[Sylvain Beucler]

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "Savane-cleanup framework".

The branch, master has been updated
       via  dd789cf82fdc58a4cda42d66b6d38fd2121b11b7 (commit)
      from  ca244d41fab94ff2cfe648073010693d1a1edb7d (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
http://git.savannah.gnu.org/cgit/savane-cleanup/framework.git/commit/?id=dd789cf82fdc58a4cda42d66b6d38fd2121b11b7

commit dd789cf82fdc58a4cda42d66b6d38fd2121b11b7
Author: Sylvain Beucler <address@hidden>
Date:   Sun Aug 9 10:59:30 2009 +0200

    Update doc about LDAP and libnss-*

diff --git a/doc/LDAP b/doc/LDAP
index a9bf0c9..e94d3dc 100644
--- a/doc/LDAP
+++ b/doc/LDAP
@@ -1,24 +1,89 @@
-Django
+Goal
+====
+
+LDAP is supported by several 3rd-party applications to connect to an
+existing users/groups base.  Using LDAP natively in our project would
+allow to maintain a users/groups base that other projects could use,
+with real-time updates.
+
+Ideally Savane could connect to an existing LDAP, either pre-existing
+or dedicated to Savane, have a separate app for write accesses to
+LDAP,
+
+
+Issues
 ======
 
-Backend:
-http://code.djangoproject.com/ticket/11526
-Synchro:
-http://www.djangosnippets.org/snippets/893/
+There are a few issues with using LDAP+Django+Unix:
+
+- (Open)LDAP is much slower than MySQL (20s to list 60000 users, < 1s
+  in MySQL without cache).  Increasing slapd's cache didn't help.  We
+  didn't find any way to improve this, and even then, this would mean
+  it's hard to install properly and poorly documented.  389DS, another
+  free LDAP server implementation, doesn't advertise improved
+  performances, and praises OpenLDAP's, so there's little hope there.
+  
http://directory.fedoraproject.org/wiki/FAQ#How_is_Fedora_Directory_Server_different_from_OpenLDAP.3F
+
+- There's no ORM for LDAP, so much User-related Django code would need
+  to be replaced
+
+- There's no official support for LDAP in Django, and what's planned
+  is a low-quality, replication based backend (instead of direct,
+  synchronized use without caching), that needs to be complemented
+  with a cron'd refresh of user profile data (email, real name, etc.)
+
+  Backend: http://code.djangoproject.com/ticket/11526
+  Synchro: http://www.djangosnippets.org/snippets/893/
+
+- Django's "sha1$" passwords are ridiculously incompatible with SSHA
+  passwords used by LDAP (among others).
+
+- LDAP queries are limited.  For example you cannot use the '<'
+  operator on shadowExpire or uidNumber, because you need to alter the
+  schema for this, and this is considered bad practice since you're
+  diverging from the RFC.  This means it's difficult to implement
+  sanity checks such as uidNumber >= 1000 when importing system users.
+
+Some solutions:
+
+- Use slapd-sql so LDAP fetchs the data transparently in SQL. But it's
+  experimental, we did test.
+  http://www.openldap.org/software/man.cgi?query=slapd-sql
+
+- Export the database to LDAP (instead of the other way around).  This
+  involves a replication delay.  Possible Savane could update LDAP
+  when a user ou group information is changed.  This means however
+  than LDAP isn't the canonical users/groups base anymore - just a
+  convenience copy.  Also implement a custom Django auth backend with
+  support for CRYPT or SSHA passwords.
+
+- Modify 3rd-party apps so they use an external database for
+  authentication, instead of using an external LDAP directory.
 
-It looks like a truly synchronized LDAP Django auth backend is
-difficult. In a first step we'll just export the Django userbase to
-LDAP (instead of the other way around).
+- Is LDAP is used for the system (Unix) through libnss-ldap(d), the
+  performances issues are not fixed.  Either you need to rely on nscd
+  (but this means you'll get a cache delay before changes are taken
+  into account, which defeats the point of using libnss-*), either you
+  rely on libnss-mysql-bg, which is more efficient (cf. NSS-MYSQL).
 
-In addition, Django's "sha1$" passwords are ridiculously incompatible
-with SSHA passwords used by LDAP (among others).  A solution is to
-implement a custom Django auth backend with support for SSHA
-passwords.
+
+Plan
+====
+
+Currently we plan to:
+
+- Use libnss-mysql-bg (possible switch to libnss-pgsql later) for the
+  system.
+
+- Export the DB to LDAP if we need a 3rd-party app with LDAP support
+  in the future.
 
 
 OpenLDAP
 ========
 
+# Installation notes
+
 # - domain: savannah.gnu.org
 # - organisation: (whatever)
 # - Allow LDAPv2 protocol: no
@@ -64,6 +129,7 @@ EOF
 
 (in all case add the indexes)
 
+
 Unix auth
 =========
 
@@ -100,25 +166,3 @@ apt-get --assume-yes install nscd
 # number of result entries in slapd (sizelimit), or filter out some
 # users, e.g. with nss_base_passwd
 # ou=users,dc=savannah,dc=gnu,dc=org?sub?!(shadowExpire=10)
-
-
-Django and LDAP
-===============
-
-After some discussion at #savannah some problems came up with  on
-using LDAP with Django, here is part of the things that needs to be
-fixed:
-
-    * Too slow to get the user information from the LDAP, the solution
-    proposed by Beuc was use nscd.
-    * Django doesn't support LDAP authentication a patch needs to be
-    apply for this purpose.
-    * There's no transparent way to read/write against the LDAP
-    database with Django.
-
-Some solutions were proposed:
-
-     * A cron job to sync the Django database with the LDAP database.
-     * Trigger some update against the LDAP when is necessary.
-     * Use slapd-sql (http://linux.die.net/man/5/slapd-sql).
-
diff --git a/doc/NSS-MYSQL b/doc/NSS-MYSQL
new file mode 100644
index 0000000..d2ed140
--- /dev/null
+++ b/doc/NSS-MYSQL
@@ -0,0 +1,35 @@
+Status
+======
+
+libnss-mysql-bg was installed at Savannah and configured against the
+old Savane MySQL database.  It works :)  This means we can avoid
+reimplementing sv_users, as well as a part of sv_groups.
+
+Next step is converting the libnss-mysql.cfg file to the new Savane DB
+schema.
+
+libnss-mysql is an alternative, but it's less flexible, since you can
+only configure fields instead of complete SQL queries.
+
+libnss-mysql-bg has 2 annoying bugs:
+
+- When listing users or groups, it may skip large entries. Report:
+  
http://sourceforge.net/tracker/?func=detail&aid=2833844&group_id=56073&atid=479258
+
+- The list of group members is implemented inefficiently (1 MySQL
+  query per group).  This is mainly a problem when listing groups, but
+  so a real problem when authenticating.  Solution suggested at:
+  
http://lists.gnu.org/archive/html/savannah-hackers-public/2009-08/msg00004.html
+
+For sanity we should fix those bugs.  However, if libnss-pgsql is
+better, we could just move to PostgreSQL.
+
+
+Segfault FAQ
+============
+
+- Check /var/log/syslog - maybe you just got an error in your SQL
+  query.
+
+- Check that the svusers group is present (libnss-mysql-bg dislikes
+  NULLs).
diff --git a/sbin/sv-fetch-passwd-ids b/sbin/sv-fetch-passwd-ids
index d4fbd1d..787d81e 100644
--- a/sbin/sv-fetch-passwd-ids
+++ b/sbin/sv-fetch-passwd-ids
@@ -17,32 +17,34 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+# TODO: this is a migration script and it should be placed somewhere
+# else than sbin
+
 # This is meant to be called before disabling the old-school
-# replication crons and switching to LDAP authentication.
+# replication crons and switching to libnss-xxx authentication.
 
 (
-echo "CREATE TEMPORARY TABLE temp_table (username varchar(30), uidNumber int, 
gidNumber int);"
+echo "CREATE TEMPORARY TABLE temp_table (username varchar(30), uidNumber int);"
 echo "INSERT INTO temp_table VALUES "
 # ('admin', 65535), ('loic', 65536);
-cat /etc/passwd | while IFS=: read username pass uid gid rest; do
+cat passwd | while IFS=: read username pass uid rest; do
   if [ $uid -gt 1000 ]; then
-    echo -n "('$username', $uid, $gid),";
+    echo -n "('$username', $uid),";
   fi;
 done | sed 's/,$//'
 echo ";"
 echo "UPDATE auth_user, svmain_extendeduser, temp_table
-  SET svmain_extendeduser.uidNumber = temp_table.uidNumber,
-      svmain_extendeduser.gidNumber = temp_table.gidNumber
+  SET svmain_extendeduser.uidNumber = temp_table.uidNumber
   WHERE auth_user.id = svmain_extendeduser.user_ptr_id
     AND auth_user.username = temp_table.username;"
-) > t.sql
-echo "DROP TABLE temp_table;" >> t.sql
+)
+echo "DROP TABLE temp_table;"
 
 (
 echo "CREATE TEMPORARY TABLE temp_table (name varchar(30), gidNumber int);"
 echo "INSERT INTO temp_table VALUES "
 # ('mifluz', 1004), ('figure', 1006);
-cat /etc/group | while IFS=: read name pass gid rest; do
+cat group | while IFS=: read name pass gid rest; do
   if [ $gid -gt 1000 ]; then
     echo -n "('$name', $gid),";
   fi;
@@ -52,5 +54,5 @@ echo "UPDATE auth_group, svmain_extendedgroup, temp_table
   SET svmain_extendedgroup.gidNumber = temp_table.gidNumber
   WHERE auth_group.id = svmain_extendedgroup.group_ptr_id
     AND auth_group.name = temp_table.name;"
-) >> t.sql
-echo "DROP TABLE temp_table;" >> t.sql
+)
+echo "DROP TABLE temp_table;"

-----------------------------------------------------------------------

Summary of changes:
 doc/LDAP                 |  112 ++++++++++++++++++++++++++++++++--------------
 doc/NSS-MYSQL            |   35 ++++++++++++++
 sbin/sv-fetch-passwd-ids |   24 +++++-----
 3 files changed, 126 insertions(+), 45 deletions(-)
 create mode 100644 doc/NSS-MYSQL


hooks/post-receive
-- 
Savane-cleanup framework