Migration Documentation

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

• OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
• For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

• apache - httpd-2.0.52-28.ent.i386
• postfix - postfix-2.2.10-1.RHEL4.2.i386
• php - php-4.3.9-3.22PIDH.i386
• python - python-2.3.4-14.3.i386
• mailman - mailman-2.1.5.1-34.rhel4.5.i386

Red Hat EL 4 rpms installed manually (rpm -i)

• MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
• MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
• MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
• MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
• clearsilver-0.10.1-1.2.el4.rf.i386.rpm
• sqlite-2.8.16-1.2.el4.rf.i386.rpm
• python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
• python-sqlite-1.0.1-12.el4.rf.i386.rpm
• subversion-1.4.3-0.1.el4.rf.i386.rpm
• mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

• MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

• drupal-4.7.4.tar.gz
• phpldapadmin-0.9.8.3.tar.gz
• trac-0.10.3.tar.gz

Paths to services directories

Apache root directory

   /var/www/html/

Subversion parent directory

   /var/www/svn/repos/

Trac parent directory

  /var/www/trac/

IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH

  *filter
  -A OUTPUT -p tcp --dport 22 -j ACCEPT
  COMMIT

/etc/sysconfig/iptables-custom/VERITAS

note: this file was placed here by PEER1

  *filter
  -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
  COMMIT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.

Apache

As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/

• ssl.conf
• subversion.conf
• trac.conf
• virtual_host.conf
• rewrite.conf
• mailman.conf
• trac.gdal.conf
• php.conf
• python.conf
• perl.conf
• phpldapadmin.conf - http auth and ssl directives for access to https://www.osgeo.org/ldapadmin
• working.conf - http auth and ssl directives for access to https://www.osgeo.org/_ldap/ldap.php
• webalizer.conf - http auth and ssl directives for access to https://www.osgeo.org/usage.php

OpenLDAP

OpenLDAP Admin Guide

Currently ldap structure is pretty basic. The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.

ldap structure

• dc=osgeo, dc=org
  • cn=Manager
  • ou=people
    • Separate entity for each user

  uid=login, ou=people, dc=osgeo, dc=org

  objectClass=inetOrgPerson

  cn=firstName lastName

  sn=lastName

  uid=login

  mail=email@address

  userPassword={md5}YPTyViiMKhiuWKEmFUOKLA==

  • ou=projects
    • Separate entity for each project group

  cn=admin,ou=projects,dc=osgeo,dc=org

  objectClass=groupOfNames

  cn=admin

  description=

  • ou=svn
    • separate entity for each svn group

Example ldif file

  version: 1

  dn: dc=osgeo,dc=org
  objectClass: dcObject
  objectClass: organization
  description: OSGeo ldap dit
  o: OSGeo
  dc: osgeo

  dn: cn=Manager,dc=osgeo,dc=org
  objectClass: organizationalRole
  cn: Manager

  dn: ou=people,dc=osgeo,dc=org
  ou: people
  description: all users of osgeo
  objectClass: organizationalUnit

  dn: uid=jsmith,ou=people,dc=osgeo,dc=org
  objectClass: inetOrgPerson
  uid: jsmith
  cn: Jon Smith
  sn: Smith 
  givenName: Jon
  mail: jsmith@somewhere.com
  userPassword: {md5}5Or4zfzGqo3jh/6iIUgKcA==

  dn: uid=jbrown,ou=people,dc=osgeo,dc=org
  objectClass: inetOrgPerson
  uid: jbrown
  cn: Jane Brown
  sn: Brown
  givenName: Jane 
  mail: jbrown@someotherplace.com
  userPassword: {md5}1iWhTyvkK2m4Uuar+Dp/IA==

  dn: ou=projects,dc=osgeo,dc=org
  ou: projects
  description: separate entity for each osgeo project with list of members
  objectClass: organizationalUnit

  dn: cn=admin,ou=projects,dc=osgeo,dc=org
  cn: admin
  description: osgeo sysadmin group
  objectClass: groupOfNames
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
  member: uid=jsmith,ou=people,dc=osgeo,dc=org

  dn: ou=svn,dc=osgeo,dc=org
  ou: svn
  description: separate entity for for each repository.list of members with commit rights
  objectClass: organizationalUnit

  dn: cn=fdo,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: fdo
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
  member: uid=jbrown,ou=people,dc=osgeo,dc=org

  dn: cn=gdal,ou=svn,dc=osgeo,dc=org
  cn: gdal
  objectClass: groupOfNames
  objectClass: top
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
  member: uid=jsmith,ou=people,dc=osgeo,dc=org

  dn: cn=mapbender,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: mapbender
  member: uid=jsmith,ou=people,dc=osgeo,dc=org

  dn: cn=mapguide,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: mapguide
  member: uid=jbrown,ou=people,dc=osgeo,dc=org

Subversion

Subversion Book

Creating a repository

  $ sudo mkdir /var/www/svn/repos/<repo_name>
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Loading a repository from a dumpfile

  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name> 

Dumping a repository

  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile

Recovering / unlocking repository

• subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
• Need policy on who to contact and who can run 'svnadmin recover' as

priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository:

  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered

  $ sudo /sbin/service httpd restart
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

TRAC

Trac Guide

Creating a trac instance

  $ sudo trac-admin /var/www/trac/<proj_name> initenv
  $ sudo chown -R apache:apache /var/www/trac/<proj_name>

Hotcopy a trac instance

  $ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to

Mailman Maintenance

create a new mailing list

  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd

renaming a list

• create new list

  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd

• move original lists archive to newlists archive location

  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
    /var/lib/mailman/newlist/newlist.mbox/

• create archive

  $ sudo /usr/lib/mailman/bin/arch --wipe newlist

• export subscribers from old list regular and digest members

  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt

• import subscribers into new list

  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname

• remove old list

  $ sudo /usr/lib/mailman/bin/rmlist listname

• update aliases and check that proper permissions are set

  $ sudo /usr/lib/mailman/bin/genaliases
  $ sudo /usr/lib/mailman/bin/check_perms -f

Backups

Daily tape backup

• PEER1 is doing daily tape backup of entire system

Other backups

Mysql

MySQL is being backed up using MySQL-zrm

current cron jobs for backups

/etc/cron.d/backup.cron

  #### Backup cron jobs 
  # min hour day month dayofweek user command
  #
  # Daily rsync to /home/back
  # /etc
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
  # /var/www/html
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
  # Every 3 hours 
  # /var/lib/mailman
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
  # /etc/mysql-zrm
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
  # /var/lib/mysql-zrm
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
  # call backup_trac_svn.sh to backup 
  # subversion /var/www/svn/repos
  # trac /var/www/trac
  45 */3 * * * root /root/scripts/backup_trac_svn.sh
  # once a day rsync /home/back to osgeo2.osgeo.net
  0 22 * * * root /root/scripts/rsync_back.pl

backup script are in /root/scripts

backup_trac_svn.sh calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days. rsync_back rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)

DNS

dns is administered via PairNIC

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.

SVNDUMPFILTER

$ svndumpfilter include --help

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

This process followed for all repositories

merge fdogdal trunk

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines,

  Node-path: trunk/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/www/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/Providers/
  ....
  Node-action: add
  ....
  PROPS-END

Rename index.html to fdogdal-index.html

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp

 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp