Migration Documentation

From OSGeo
Jump to navigation Jump to search

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

  • OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
  • For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

  • apache - httpd-2.0.52-28.ent.i386
  • postfix - postfix-2.2.10-1.RHEL4.2.i386
  • php - php-4.3.9-3.22PIDH.i386
  • python - python-2.3.4-14.3.i386
  • mailman - mailman-

Red Hat EL 4 rpms installed manually (rpm -i)

  • MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
  • clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • sqlite-2.8.16-1.2.el4.rf.i386.rpm
  • python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • python-sqlite-1.0.1-12.el4.rf.i386.rpm
  • subversion-1.4.3-0.1.el4.rf.i386.rpm
  • mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

  • MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

  • drupal-4.7.4.tar.gz
  • phpldapadmin-
  • trac-0.10.3.tar.gz

Paths to services directories

Apache root directory


Subversion parent directory


Trac parent directory


IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:


  -A OUTPUT -p tcp --dport 22 -j ACCEPT


note: this file was placed here by PEER1

  -A INPUT -s -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d -p tcp -j ACCEPT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.


As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/


Postfix documentation

The postfix daemon can be be stared|stopped|restarted|reloaded with,

$ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart

The main postfix config file is,


The aliases file is,


If aliases file is edited remember to run newaliases

 $ sudo /usr/bin/newaliases
 $ sudo /sbin/service postfix reload


OpenLDAP Admin Guide

The ldap daemon can manipulated with,

$ sudo /sbin/service ldap start|stop|restart|

The main ldap config file is,


Currently ldap structure is pretty basic. The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.

Loading an ldif file into an ldap directory

  $ ldapadd -a -W -x -D "cn=Manager,dc=osgeo,dc=org" -f fileName.ldif

Accessing the ldap directory through phpldapadmin

access an ldapadmin interface at https://www.osgeo.org/ldapadmin

Access to this interface is restricted to the admin group

users can be added to the ou=people,dc=osgeo,dc=org once users are added they can then be added to cn=project,ou=svn,dc=osgeo,dc=org

login as cn=Manager,dc=osgeo,dc=org

  • expand dc=osgeo,dc=org tree
  • expand the ou=people tree
  • click on 'create new entry here'
  • choose 'custom' and then 'proceed'
    • RDN is in the form of uid=username
    • ObjectClass is InetOrgPerson
    • click proceed
      • cn is common name in form of firstname lastname
      • sn is surname - lastname
      • no optionals are required
    • click 'create object'
      • on this page click 'add new attribute'
        • choose mail
          • enter email address
          • click add
        • on this page click 'add new attribute'
          • choose userPassord
          • enter password as an md5
          • click add

this adds the new user to ou=people

to add users to the cn=project,ou=svn group

  • expand the ou=svn tree
  • click on cn=project
    • under member click 'add value'
    • add user to this in same form as you and howard are listed
    • or can click on the 'folder with magnifying glasss' to select a user

that's it.

ldap structure

  • dc=osgeo,dc=org
    • cn=Manager
    • ou=people
      • Separate entity for each user
  • uid=login,ou=people,dc=osgeo,dc=org
    cn=firstName lastName
    • ou=projects,dc=osgeo,dc=org
    description=separate entity for each osgeo project with list of members
      • Separate entity for each project group
    description=osgeo sysadmin group
    • ou=svn
    description=separate entity for each repository with list of members with commit rights
      • separate entity for each svn group
    member= dn of member
  • cn=gdal,ou=svn,dc=osgeo,dc=org
    member= dn of member
  • cn=mapguide,ou=svn,dc=osgeo,dc=org
    member= dn of member
  • cn=mapbender,ou=svn,dc=osgeo,dc=org
    member= dn of member

Example ldif file

   version: 1

   dn: dc=osgeo,dc=org
   objectClass: dcObject
   objectClass: organization
   description: OSGeo ldap dit
   o: OSGeo
   dc: osgeo

   dn: cn=Manager,dc=osgeo,dc=org
   objectClass: organizationalRole
   cn: Manager

   dn: ou=people,dc=osgeo,dc=org
   ou: people
   description: all users of osgeo
   objectClass: organizationalUnit

   dn: uid=jsmith,ou=people,dc=osgeo,dc=org
   objectClass: inetOrgPerson
   uid: jsmith
   cn: Jon Smith
   sn: Smith 
   givenName: Jon
   mail: jsmith@somewhere.com
   userPassword: {md5}5Or4zfzGqo3jh/6iIUgKcA==

   dn: uid=jbrown,ou=people,dc=osgeo,dc=org
   objectClass: inetOrgPerson
   uid: jbrown
   cn: Jane Brown
   sn: Brown
   givenName: Jane 
   mail: jbrown@someotherplace.com
   userPassword: {md5}1iWhTyvkK2m4Uuar+Dp/IA==

   dn: ou=projects,dc=osgeo,dc=org
   ou: projects
   description: separate entity for each osgeo project with list of members
   objectClass: organizationalUnit

   dn: cn=admin,ou=projects,dc=osgeo,dc=org
   cn: admin
   description: osgeo sysadmin group
   objectClass: groupOfNames
   member: uid=jbrown,ou=people,dc=osgeo,dc=org
   member: uid=jsmith,ou=people,dc=osgeo,dc=org

   dn: ou=svn,dc=osgeo,dc=org
   ou: svn
   description: separate entity for for each repository.list of members with commit rights
   objectClass: organizationalUnit

   dn: cn=fdo,ou=svn,dc=osgeo,dc=org
   objectClass: groupOfNames
   cn: fdo
   member: uid=jsmith,ou=people,dc=osgeo,dc=org
   member: uid=jbrown,ou=people,dc=osgeo,dc=org

   dn: cn=gdal,ou=svn,dc=osgeo,dc=org
   cn: gdal
   objectClass: groupOfNames
   objectClass: top
   member: uid=jbrown,ou=people,dc=osgeo,dc=org
   member: uid=jsmith,ou=people,dc=osgeo,dc=org

   dn: cn=mapbender,ou=svn,dc=osgeo,dc=org
   objectClass: groupOfNames
   cn: mapbender
   member: uid=jsmith,ou=people,dc=osgeo,dc=org

   dn: cn=mapguide,ou=svn,dc=osgeo,dc=org
   objectClass: groupOfNames
   cn: mapguide
   member: uid=jbrown,ou=people,dc=osgeo,dc=org


Subversion Book

Creating a repository

  $ sudo mkdir /var/www/svn/repos/<repo_name>
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Loading a repository from a dumpfile

  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name> 

Dumping a repository

  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile

Recovering / unlocking repository

  • subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
  • Need policy on who to contact and who can run 'svnadmin recover' as

priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository:

  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered

  $ sudo /sbin/service httpd restart
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>


Trac Instances

Mailman Maintenance

create a new mailing list

  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd

renaming a list

  • create new list
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
  • move original lists archive to newlists archive location
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
  • create archive
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
  • export subscribers from old list regular and digest members
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
  • import subscribers into new list
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
  • remove old list
  $ sudo /usr/lib/mailman/bin/rmlist listname
  • update aliases and check that proper permissions are set
  $ sudo /usr/lib/mailman/bin/genaliases
  $ sudo /usr/lib/mailman/bin/check_perms -f
  • edit postfix aliases - /etc/aliases
  oldlist:               newlist@lists.osgeo.org
  oldlist-request:       newlist@lists.osgeo.org
  oldlist-admin:         newlist@lists.osgeo.org
  oldlist-owner:         newlist@lists.osgeo.org
  • update postfix with new aliases
  $ sudo /usr/bin/newaliases
  $ sudo /sbin/service postfix reload


Daily tape backup

  • PEER1 is doing daily tape backup of entire system

Other backups


MySQL is being backed up using MySQL-zrm

current cron jobs for backups


  #### Backup cron jobs 
  # min hour day month dayofweek user command
  # Daily rsync to /home/back
  # /etc
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
  # /var/www/html
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
  # Every 3 hours 
  # /var/lib/mailman
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
  # /etc/mysql-zrm
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
  # /var/lib/mysql-zrm
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
  # call backup_trac_svn.sh to backup 
  # subversion /var/www/svn/repos
  # trac /var/www/trac
  0 1 * * * root /root/scripts/backup_svn_full.sh
  45 */3 * * * root /root/scripts/backup_svn_incremental.sh
  45 */3 * * * root /root/scripts/trac_backup.pl
  # once a day rsync /home/back to osgeo2.osgeo.net
  0 22 * * * root /root/scripts/rsync_back.pl

backup script are in /root/scripts

backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental

backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full. 14 copies of the repository are kept.

When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects. You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. rsync_back rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)


dns is administered via PairNIC

The zone file for the osgeo.org dns was mirrored from CN's zonefile. There is some work here to get rid of * (wildcard) entry in dns.

current dns records

@ 	             IN A  
www                 IN A  		
mapguide 	     IN A 		 
find.geodata 	     IN A 		
mapbender2 	     IN A	
dev.geodata 	     IN A
wiki 	             IN A 	
svn 	             IN A 	
geodata 	     IN A 	
*.wiki 	     IN A 	
access.geodata      IN A 	
buildbot 	     IN A 	
fdo2 	             IN A 		
www.communitymapbuilder IN A 	
* 	             IN A 	
community 	     IN A 	
download 	     IN A 	
testbed 	     IN A 	
lists 	             IN A 	
mapguide2 	     IN A 	
*.community 	     IN A 	
mapbender 	     IN A 	
mail 	             IN A 	
public.geodata      IN A 	
communitymapbuilder IN A 	
fdo                 IN A
MX records
gdal.osgeo.org     IN CNAME   www.gdal.org

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.


$ svndumpfilter include --help

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

This process followed for all repositories

merge fdogdal trunk

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines,

  Node-path: trunk/
  Node-action: add
  Node-path: trunk/www/
  Node-action: add
  Node-path: trunk/Providers/
  Node-action: add

Rename index.html to fdogdal-index.html

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp
 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp