Difference between revisions of "Migration Documentation"

From OSGeo
Jump to navigation Jump to search
m (→‎LDAP: move out LDAP content to another page.)
(8 intermediate revisions by 2 users not shown)
Line 106: Line 106:
   $ sudo /sbin/service postfix reload
   $ sudo /sbin/service postfix reload
= OpenLDAP =
= LDAP =
[http://www.openldap.org/doc/admin23/ OpenLDAP Admin Guide]
The ldap daemon can manipulated with,
See [[SAC:LDAP]]
$ sudo /sbin/service ldap start|stop|restart|
The main ldap config file is,
Currently ldap structure is pretty basic.  The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.
''' Loading an ldif file into an ldap directory '''
  $ ldapadd -a -W -x -D "cn=Manager,dc=osgeo,dc=org" -f fileName.ldif
''' Accessing the ldap directory through phpldapadmin '''
access an ldapadmin interface at https://www.osgeo.org/ldapadmin
Access to this interface is restricted to the admin group
users can be added to the ou=people,dc=osgeo,dc=org
once users are added they can then be added to
login as cn=Manager,dc=osgeo,dc=org
*expand dc=osgeo,dc=org tree
*expand the ou=people tree
*click on 'create new entry here'
*choose 'custom' and then 'proceed'
**RDN is in the form of uid=username
**ObjectClass is InetOrgPerson
**click proceed
***cn is common name in form of firstname lastname
***sn is surname - lastname
***no optionals are required
**click 'create object'
***on this page click 'add new attribute'
****choose mail
*****enter email address
*****click add
****on this page click 'add new attribute'
*****choose userPassord
*****enter password as an md5
*****click add
this adds the new user to ou=people
to add users to the cn=project,ou=svn group
*expand the ou=svn tree
*click on cn=project
**under member click 'add value'
**add user to this in same form as you and howard are listed
**or can click on the 'folder with magnifying glasss' to select a user
that's it.
=== ldap structure ===
***Separate entity for each user
*:cn=firstName lastName
*:description=separate entity for each osgeo project with list of members
***Separate entity for each project group
*:description=osgeo sysadmin group
*:description=separate entity for each repository with list of members with commit rights
***separate entity for each svn group
*:member= dn of member
*:member= dn of member
*:member= dn of member
*:member= dn of member
'''Example ldif file'''
  version: 1
  dn: dc=osgeo,dc=org
  objectClass: dcObject
  objectClass: organization
  description: OSGeo ldap dit
  o: OSGeo
  dc: osgeo
  dn: cn=Manager,dc=osgeo,dc=org
  objectClass: organizationalRole
  cn: Manager
  dn: ou=people,dc=osgeo,dc=org
  ou: people
  description: all users of osgeo
  objectClass: organizationalUnit
  dn: uid=jsmith,ou=people,dc=osgeo,dc=org
  objectClass: inetOrgPerson
  uid: jsmith
  cn: Jon Smith
  sn: Smith
  givenName: Jon
  mail: jsmith@somewhere.com
  userPassword: {md5}5Or4zfzGqo3jh/6iIUgKcA==
  dn: uid=jbrown,ou=people,dc=osgeo,dc=org
  objectClass: inetOrgPerson
  uid: jbrown
  cn: Jane Brown
  sn: Brown
  givenName: Jane
  mail: jbrown@someotherplace.com
  userPassword: {md5}1iWhTyvkK2m4Uuar+Dp/IA==
  dn: ou=projects,dc=osgeo,dc=org
  ou: projects
  description: separate entity for each osgeo project with list of members
  objectClass: organizationalUnit
  dn: cn=admin,ou=projects,dc=osgeo,dc=org
  cn: admin
  description: osgeo sysadmin group
  objectClass: groupOfNames
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
  dn: ou=svn,dc=osgeo,dc=org
  ou: svn
  description: separate entity for for each repository.list of members with commit rights
  objectClass: organizationalUnit
  dn: cn=fdo,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: fdo
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
  dn: cn=gdal,ou=svn,dc=osgeo,dc=org
  cn: gdal
  objectClass: groupOfNames
  objectClass: top
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
  dn: cn=mapbender,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: mapbender
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
  dn: cn=mapguide,ou=svn,dc=osgeo,dc=org
  objectClass: groupOfNames
  cn: mapguide
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
= Subversion =
= Subversion =
[http://svnbook.red-bean.com/nightly/en/index.html Subversion Book]
=== Creating a repository ===
  $ sudo mkdir /var/www/svn/repos/<repo_name>
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
=== Loading a repository from a dumpfile ===
See [[Subversion]] for details on subversion configuration.
  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
=== Dumping a repository ===
  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile
=== Recovering / unlocking repository ===
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
* Need policy on who to contact and who can run 'svnadmin recover' as
priviledged access is needed (may need to stop/start apache to drop
requests to repository before recover)
Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.
If the repository locks the following command should be used to recover the repository:
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered
  $ sudo /sbin/service httpd restart
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
= Trac =  
= Trac =  
Line 335: Line 119:
= Mailman Maintenance =
= Mailman Maintenance =
=== create a new mailing list ===
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
See [[SAC:Mailing Lists]].
=== renaming a list ===
* create new list
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
* move original lists archive to newlists archive location
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
* create archive
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
* export subscribers from old list regular and digest members
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
* import subscribers into new list
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
* remove old list
  $ sudo /usr/lib/mailman/bin/rmlist listname
* update aliases and check that proper permissions are set
  $ sudo /usr/lib/mailman/bin/genaliases
  $ sudo /usr/lib/mailman/bin/check_perms -f
* edit postfix aliases - /etc/aliases
  oldlist:              newlist@lists.osgeo.org
  oldlist-request:      newlist@lists.osgeo.org
  oldlist-admin:         newlist@lists.osgeo.org
  oldlist-owner:        newlist@lists.osgeo.org
* update postfix with new aliases
  $ sudo /usr/bin/newaliases
  $ sudo /sbin/service postfix reload
= Backups =  
= Backups =  
=== Daily tape backup ===
See [[SAC:Backups]].
* PEER1 is doing daily tape backup of entire system
=== Other backups ===
==== Mysql ====
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
==== current cron jobs for backups ====
  #### Backup cron jobs
  # min hour day month dayofweek user command
  # Daily rsync to /home/back
  # /etc
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
  # /var/www/html
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
  # Every 3 hours
  # /var/lib/mailman
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
  # /etc/mysql-zrm
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
  # /var/lib/mysql-zrm
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
  # call backup_trac_svn.sh to backup
  # subversion /var/www/svn/repos
  # trac /var/www/trac
  0 1 * * * root /root/scripts/backup_svn_full.sh
  45 */3 * * * root /root/scripts/backup_svn_incremental.sh
  45 */3 * * * root /root/scripts/trac_backup.pl
  # once a day rsync /home/back to osgeo2.osgeo.net
  0 22 * * * root /root/scripts/rsync_back.pl
backup script are in /root/scripts
backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental
backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full.  14 copies of the repository are kept. 
When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects.  You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. 
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
= DNS =
= DNS =
dns is administered via [https://www.pairnic.com/login.html PairNIC]
The zone file for the osgeo.org dns was mirrored from CN's zonefile.  There is some work here to get rid of * (wildcard) entry in dns.
See [[SAC DNS Registry]].
'''current dns records'''
@             IN A 
www                IN A 
mapguide     IN A
find.geodata     IN A
mapbender2     IN A
dev.geodata     IN A
wiki             IN A
svn             IN A
geodata     IN A
*.wiki     IN A
access.geodata      IN A
buildbot     IN A
fdo2             IN A
www.communitymapbuilder IN A
*             IN A
community     IN A
download     IN A
testbed     IN A
lists             IN A
mapguide2     IN A
*.community     IN A
mapbender     IN A
mail             IN A
public.geodata      IN A
communitymapbuilder IN A
fdo                IN A
MX records
gdal.osgeo.org    IN CNAME  www.gdal.org
= FDO Repository Merge =
= FDO Repository Merge =
Line 512: Line 190:
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp

Latest revision as of 14:18, 2 February 2008

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

  • OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
  • For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

  • apache - httpd-2.0.52-28.ent.i386
  • postfix - postfix-2.2.10-1.RHEL4.2.i386
  • php - php-4.3.9-3.22PIDH.i386
  • python - python-2.3.4-14.3.i386
  • mailman - mailman-

Red Hat EL 4 rpms installed manually (rpm -i)

  • MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
  • clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • sqlite-2.8.16-1.2.el4.rf.i386.rpm
  • python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • python-sqlite-1.0.1-12.el4.rf.i386.rpm
  • subversion-1.4.3-0.1.el4.rf.i386.rpm
  • mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

  • MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

  • drupal-4.7.4.tar.gz
  • phpldapadmin-
  • trac-0.10.3.tar.gz

Paths to services directories

Apache root directory


Subversion parent directory


Trac parent directory


IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:


  -A OUTPUT -p tcp --dport 22 -j ACCEPT


note: this file was placed here by PEER1

  -A INPUT -s -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d -p tcp -j ACCEPT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.


As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/


Postfix documentation

The postfix daemon can be be stared|stopped|restarted|reloaded with,

$ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart

The main postfix config file is,


The aliases file is,


If aliases file is edited remember to run newaliases

 $ sudo /usr/bin/newaliases
 $ sudo /sbin/service postfix reload




See Subversion for details on subversion configuration.


Trac Instances

Mailman Maintenance

See SAC:Mailing Lists.


See SAC:Backups.


See SAC DNS Registry.

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.


$ svndumpfilter include --help

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

This process followed for all repositories

merge fdogdal trunk

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines,

  Node-path: trunk/
  Node-action: add
  Node-path: trunk/www/
  Node-action: add
  Node-path: trunk/Providers/
  Node-action: add

Rename index.html to fdogdal-index.html

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp
 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp