Revision as of 10:24, 7 October 2009

Intro

OSGeo maintains a set of subversion services, currently hosted on the primary Peer1 server at the virtual host http://svn.osgeo.org/. Subversion access is authenticated based on OSGeo Userids and each project has a committer list also maintained in LDAP. Excellent general information on subversion is available from the Subversion Book.

Current Repositories

http://svn.osgeo.org/fdo - FDO (also a bunch of old repositories exist for FDO)
http://svn.osgeo.org/gdal - GDAL
http://svn.osgeo.org/grass - GRASS GIS
http://svn.osgeo.org/mapbender - Mapbender
http://svn.osgeo.org/mapguide - MapGuide
http://svn.osgeo.org/moss4g - MOSS4g
http://svn.osgeo.org/osgeo - OSGeo
http://svn.osgeo.org/ossim - OSSIM
http://svn.osgeo.org/geos - GEOS
http://svn.osgeo.org/mapserver - MapServer
http://svn.osgeo.org/qgis - QGIS
http://svn.osgeo.org/fusion - Fusion
http://svn.osgeo.org/osgeo4w - OSGeo4W
http://svn.osgeo.org/metacrs - MetaCRS
https://svn.osgeo.org/geoinformatica/ - Geoinformatica

Mirror / Backup

http://svnmirror.osgeo.org is a mirror of the repositories that is on a 15 minute svnsync. The physical machine is located at TelaScience. This is the only official backup of the svn repositories, then are not rsync'ed to osgeo2 or anything like that.

Administration

Howard Butler (hobu), and Shawn Barnes (sbarnes) are the primary contacts for Subversion administration issues. Requests can be submitted via the SAC Trac.

Add/Remove Committer

Any member of a subversion committer group and add and remove members to the commiter group at an url of the following form, with the project name substituted for gdal. OSGeo Userids has details on finding or creating OSGeo userids.

 https://www.osgeo.org/cgi-bin/auth/ldap_group.py?group=gdal

Creating a repository

Create the repository on the main server:

  $ sudo mkdir /var/www/svn/repos/<repo_name>
  $ sudo svnadmin create /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Enable it's use from apache:

  $ cd /etc/httpd/conf.d/subversion
  $ sudo cp osgeo4w.conf <repo_name>.conf
  $ sudo vi <repo_name>.conf  
      (change all "osgeo4w' to "<repo_name>")
  $ sudo svn add <repo_name>.conf
  $ sudo svn commit -m "new repository added by frankw" <repo_name>.conf

Create the authorization group for the svn repository at:

  https://www.osgeo.org/ldapadmin/

Under "ou=svn" add a new entry of type Custom->groupOfNames, with RDN that is your "cn=<repo_name>" and add one initial admin's osgeo userid there in this form:

 uid=warmerdam,ou=people,dc=osgeo,dc=org

Then restart apache gracefully:

 $ sudo /usr/sbin/apachectl graceful

Notify hobu/mloskot about adding a mirror of the newly created repository to http://svnmirror.osgeo.org

Loading a repository from a dumpfile

By root on main server:

  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Dumping a repository

By root on main server:

  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile

Recovering / unlocking repository

subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
Need policy on who to contact and who can run 'svnadmin recover' as

priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository:

  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered

  $ sudo /sbin/service httpd restart
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Authz Rules

By default we use one LDAP "group" for an entire subversion repository. Everyone in that group has commit priveledges on the whole repository. However, some project desire to have subgroups. For instance a core commiter group with update on the trunk source tree, and a broader contributor group that has commit access in a subarea, such as /sandbox. We accomplish this using an "authz" file to hold the rules for the respository, and multiple LDAP groups.

A project wishing this will need to prepare an authz template file looking something like:

[groups]
committers = @@gdal@@
sandbox = @@gdal_sandbox@@

[/]
* = r
@committers = rw

[/sandbox]
* = r
@committers = rw
@sandbox = rw

This file should be submitted to SAC via a Trac ticket for placement on the server, and for creation of any corresponding LDAP groups. Thereafter any successful update via https://www.osgeo.org/cgi-bin/auth/ldap_group.py?group=project will result in the authz being generated from the corresponding authz.tmpl.

Administration Notes

The authz.tmpl files need to be named exactly authz.tmpl and place in the corresponding project repository directory, /var/www/svn/repos/gdal for instance. Make sure the file is owned by apache.apache.

The corresponding /etc/httpd/conf.d/subversion/project.conf file should look something like the following. Note the removal of <LimitExcept> stuff, and reference to any specific ldap group as well as the AuthzSVNAccessFile.

    <Location /gdal>
      DAV svn
      SVNPath /var/www/svn/repos/gdal
      AuthzSVNAccessFile /var/www/svn/repos/gdal/authz
      Include conf.d/ldap_auth_url.inc
      Satisfy Any
      Require valid-user
    </Location>

The LDAP groups should be created as groupOfNames nodes below the ou=svn element using PHPLDAPAdmin. Make sure the membership is pre-initialized with an administrator for the project so they can do future updates themselves. Try to keep the names of the groups clearly related to the base project. eg. gdal, gdal_sandbox, gdal_documentors.

Notes

The script /var/www/svn/site-local/ldap_update_authz.py script is run by an os.system() call out of ldap_group.py on any successful group update. It will scan for /var/www/svn/repos/*/authz.tmpl.

There are lots of things that can go wrong in the script which has poor error checking, and it might only manifest as some or all of the authz file no longer getting regenerated. So when an authz.tmpl file is put in place, or updated do a test update and verify that all authz files are being updated (check timestamps).

Anyone (on osgeo1) can run ldap_update_authz.py, but if authz files are created by someone that apache.apache can't delete, problems may ensue. If this occurs, the following as root may help.

 cd /var/www/svn
 chown apache.apache */*/authz*

SVN Mirror

The OSGeo SVN mirror is maintained at http://svnmirror.osgeo.org This machine is located at TelaScience, but it is not a blade and not part of the normal cadre of machines that OSGeo typically uses of TelaScience's. The mirror is Subversion 1.5.6 svnsync-based. When new repositories are created, a new mirror repository must also be created if it is wished that it is backed up. Nothing prevents individual users or projects from maintaining their own backups with svnsync.

The mirror itself is located in /state/partition1/osgeo_svn_mirror There is a in /etc/cron.hourly/svnmirror that runs the syncs.sh script in /state/partition1/osgeo_svn_mirror to do the repository synchronization.

To create a new mirror:

1) Make a new .conf file in /etc/httpd/conf.d/subversion/ for the repository (use an existing one as a template)

2) Initialize the repo

  svnadmin init /state/partition1/osgeo_svn_mirror/repos/newrepo

3) restart apache

  sudo /etc/init.d/httpd restart

4) Copy the pre-revprop-change hook script from an existing repo into the new repo

  sudo cp /state/partition1/osgeo_svn_mirror/repos/gdal/hooks/pre-revprop-change /state/partition1/osgeo_svn_mirror/repos/newrepo/hooks/pre-revprop-change

5) init the svnsync (password is empty \'\' without the backslashes)

  svnsync init --username svnsync --password  http://svnmirror.osgeo.org/newrepo http://svn.osgeo.org/newrepo

6) Add it to the repositories list in /state/partition1/osgeo_svn_mirror/syncs.sh

7) Run the syncs.sh script

  sudo ./syncs.sh

Difference between revisions of "Subversion"