Migration Documentation

= Description of current layout and installed software =

Notes from Shawn on our PEER1 system:


 * OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
 * For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

 * apache - httpd-2.0.52-28.ent.i386
 * postfix - postfix-2.2.10-1.RHEL4.2.i386
 * php - php-4.3.9-3.22PIDH.i386
 * python - python-2.3.4-14.3.i386
 * mailman - mailman-2.1.5.1-34.rhel4.5.i386

Red Hat EL 4 rpms installed manually (rpm -i)

 * MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
 * MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
 * MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
 * MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
 * clearsilver-0.10.1-1.2.el4.rf.i386.rpm
 * sqlite-2.8.16-1.2.el4.rf.i386.rpm
 * python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
 * python-sqlite-1.0.1-12.el4.rf.i386.rpm
 * subversion-1.4.3-0.1.el4.rf.i386.rpm
 * mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

 * MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

 * drupal-4.7.4.tar.gz
 * phpldapadmin-0.9.8.3.tar.gz
 * trac-0.10.3.tar.gz

= Paths to services directories =

Apache root directory /var/www/html/

Subversion parent directory /var/www/svn/repos/

Trac parent directory /var/www/trac/

= IP tables custom =

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH *filter -A OUTPUT -p tcp --dport 22 -j ACCEPT COMMIT

/etc/sysconfig/iptables-custom/VERITAS

note: this file was placed here by PEER1 *filter -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT COMMIT

= System login and maintenance procedures =

'''Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.'''

= Apache = As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/


 * ssl.conf
 * subversion.conf
 * trac.conf
 * virtual_host.conf
 * rewrite.conf
 * mailman.conf
 * trac.gdal.conf
 * php.conf
 * python.conf
 * perl.conf
 * phpldapadmin.conf - http auth and ssl directives for access to https://www.osgeo.org/ldapadmin
 * working.conf - http auth and ssl directives for access to https://www.osgeo.org/_ldap/ldap.php
 * webalizer.conf - http auth and ssl directives for access to https://www.osgeo.org/usage.php

= Postfix = Postfix documentation

The postfix daemon can be be stared|stopped|restarted|reloaded with, $ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart

The main postfix config file is, /etc/postfix/main.cfg

The aliases file is, /etc/aliases

If aliases file is edited remember to run newaliases $ sudo /usr/bin/newaliases $ sudo /sbin/service postfix reload

= OpenLDAP = OpenLDAP Admin Guide

The ldap daemon can manipulated with, $ sudo /sbin/service ldap start|stop|restart|

The main ldap config file is, /etc/openldap/slapd.conf

Currently ldap structure is pretty basic. The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.

 Loading an ldif file into an ldap directory  $ ldapadd -a -W -x -D "cn=Manager,dc=osgeo,dc=org" -f fileName.ldif

 Accessing the ldap directory through phpldapadmin 

access an ldapadmin interface at https://www.osgeo.org/ldapadmin

Access to this interface is restricted to the admin group

users can be added to the ou=people,dc=osgeo,dc=org once users are added they can then be added to cn=project,ou=svn,dc=osgeo,dc=org

login as cn=Manager,dc=osgeo,dc=org


 * expand dc=osgeo,dc=org tree
 * expand the ou=people tree
 * click on 'create new entry here'
 * choose 'custom' and then 'proceed'
 * RDN is in the form of uid=username
 * ObjectClass is InetOrgPerson
 * click proceed
 * cn is common name in form of firstname lastname
 * sn is surname - lastname
 * no optionals are required
 * click 'create object'
 * on this page click 'add new attribute'
 * choose mail
 * enter email address
 * click add
 * on this page click 'add new attribute'
 * choose userPassord
 * enter password as an md5
 * click add

this adds the new user to ou=people

to add users to the cn=project,ou=svn group
 * expand the ou=svn tree
 * click on cn=project
 * under member click 'add value'
 * add user to this in same form as you and howard are listed
 * or can click on the 'folder with magnifying glasss' to select a user

that's it.

ldap structure

 * dc=osgeo,dc=org


 * cn=Manager


 * ou=people


 * Separate entity for each user


 * uid=login,ou=people,dc=osgeo,dc=org
 * objectClass=inetOrgPerson
 * cn=firstName lastName
 * sn=lastName
 * uid=login
 * mail=email@address
 * userPassword={md5}YPTyViiMKhiuWKEmFUOKLA==


 * ou=projects,dc=osgeo,dc=org
 * objectClass=organizationalUnit
 * ou=project
 * description=separate entity for each osgeo project with list of members


 * Separate entity for each project group
 * cn=admin,ou=projects,dc=osgeo,dc=org
 * objectClass=groupOfNames
 * cn=admin
 * description=osgeo sysadmin group


 * ou=svn
 * objectClass=organizationalUnit
 * ou=svn
 * description=separate entity for each repository with list of members with commit rights


 * separate entity for each svn group
 * cn=fdo,ou=svn,dc=osgeo,dc=org
 * objectClass=groupOfNames
 * cn=fdo
 * member= dn of member


 * cn=gdal,ou=svn,dc=osgeo,dc=org
 * objectClass=groupOfNames
 * cn=gdal
 * member= dn of member


 * cn=mapguide,ou=svn,dc=osgeo,dc=org
 * objectClass=groupOfNames
 * cn=mapguide
 * member= dn of member


 * cn=mapbender,ou=svn,dc=osgeo,dc=org
 * objectClass=groupOfNames
 * cn=mapbender
 * member= dn of member

Example ldif file

= Subversion = Subversion Book

Creating a repository
$ sudo mkdir /var/www/svn/repos/ $ sudo svnadmin /var/www/svn/repos/ $ sudo chown -R apache:apache /var/www/svn/repos/

Loading a repository from a dumpfile
$ sudo svnadmin load /var/www/svn/repos/ < /path/to/dumpfile $ sudo chown -R apache:apache /var/www/svn/repos/

Dumping a repository
$ sudo svnadmin dump /var/www/svn/repos/ > dumpfile

Recovering / unlocking repository
priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)
 * subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
 * Need policy on who to contact and who can run 'svnadmin recover' as

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository:

$ sudo svnadmin --wait recover /var/www/svn/repos/ $ sudo chown -R apache:apache /var/www/svn/repos/

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered

$ sudo /sbin/service httpd restart $ sudo svnadmin --wait recover /var/www/svn/repos/ $ sudo chown -R apache:apache /var/www/svn/repos/

= Trac =

Trac Instances

= Mailman Maintenance =

create a new mailing list
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd

renaming a list
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \ /var/lib/mailman/newlist/newlist.mbox/ $ sudo /usr/lib/mailman/bin/arch --wipe newlist $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname $ sudo /usr/lib/mailman/bin/rmlist listname $ sudo /usr/lib/mailman/bin/genaliases $ sudo /usr/lib/mailman/bin/check_perms -f oldlist:              newlist@lists.osgeo.org oldlist-request:      newlist@lists.osgeo.org oldlist-admin:        newlist@lists.osgeo.org oldlist-owner:        newlist@lists.osgeo.org $ sudo /usr/bin/newaliases $ sudo /sbin/service postfix reload
 * create new list
 * move original lists archive to newlists archive location
 * create archive
 * export subscribers from old list regular and digest members
 * import subscribers into new list
 * remove old list
 * update aliases and check that proper permissions are set
 * edit postfix aliases - /etc/aliases
 * update postfix with new aliases

= Backups =

Daily tape backup

 * PEER1 is doing daily tape backup of entire system

Mysql
MySQL is being backed up using MySQL-zrm

current cron jobs for backups
/etc/cron.d/backup.cron

#### Backup cron jobs # min hour day month dayofweek user command #  # Daily rsync to /home/back # /etc 05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/ # /var/www/html 08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/ # Every 3 hours # /var/lib/mailman 20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/ # /etc/mysql-zrm 33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/ # /var/lib/mysql-zrm 34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/ # call backup_trac_svn.sh to backup # subversion /var/www/svn/repos # trac /var/www/trac

0 1 * * * root /root/scripts/backup_svn_full.sh  45 */3 * * * root /root/scripts/backup_svn_incremental.sh   45 */3 * * * root /root/scripts/trac_backup.pl

# once a day rsync /home/back to osgeo2.osgeo.net 0 22 * * * root /root/scripts/rsync_back.pl

backup script are in /root/scripts

backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental

backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full. 14 copies of the repository are kept.

When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects. You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. rsync_back rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)

= DNS = dns is administered via PairNIC

The zone file for the osgeo.org dns was mirrored from CN's zonefile. There is some work here to get rid of * (wildcard) entry in dns.

current dns records @ 	            IN A      66.223.95.242 www                IN A      66.223.95.242 mapguide 	    IN A      66.223.95.245 find.geodata 	    IN A      198.202.74.216 mapbender2 	    IN A      66.223.95.245 dev.geodata 	    IN A      198.202.74.216 wiki 	            IN A      85.214.35.116 svn 	            IN A      66.223.95.242 geodata 	    IN A      66.223.95.242 *.wiki 	    IN A      85.214.35.116 access.geodata     IN A      198.202.74.216 buildbot 	    IN A      198.202.74.219 fdo2 	            IN A      66.223.95.245 www.communitymapbuilder IN A 63.246.7.187 * 	            IN A      66.223.95.242 community 	    IN A      66.223.95.242 download 	    IN A      198.202.74.219 testbed 	    IN A      140.211.166.123 lists 	            IN A      66.223.95.242 mapguide2 	    IN A      66.223.95.245 *.community 	    IN A      140.211.166.123 mapbender 	    IN A      66.223.95.245 mail 	            IN A      66.223.95.242 public.geodata     IN A      198.202.74.216 communitymapbuilder IN A     63.246.7.187 fdo                IN A      66.223.95.245

MX records lists.osgeo.org mail.osgeo.org

CNAME gdal.osgeo.org    IN CNAME   www.gdal.org

= FDO Repository Merge =

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.

SVNDUMPFILTER

$ svndumpfilter include --help Filter out nodes without given prefixes from dumpstream Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs		Remove revisions emptied by filtering --renumber-revs			Renumbe revisions left after filtering --preserve-revprops		Don't filter revision properties --quiet					Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs		Remove revisions emptied by filtering --renumber-revs			Renumbe revisions left after filtering --preserve-revprops		Don't filter revision properties --quiet					Do not display filtering statistics

This process followed for all repositories

merge fdogdal trunk

$ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines, Node-path: trunk/ ....  Node-action: add ....  PROPS-END

Node-path: trunk/www/ ....  Node-action: add ....  PROPS-END

Node-path: trunk/Providers/ ....  Node-action: add ....  PROPS-END

Rename index.html to fdogdal-index.html $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp

$ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp