Difference between revisions of "Migration Documentation"

From OSGeo
Jump to navigation Jump to search
m (→‎LDAP: move out LDAP content to another page.)
 
(15 intermediate revisions by 4 users not shown)
Line 89: Line 89:
 
*working.conf - http auth and ssl directives for access to https://www.osgeo.org/_ldap/ldap.php
 
*working.conf - http auth and ssl directives for access to https://www.osgeo.org/_ldap/ldap.php
 
*webalizer.conf - http auth and ssl directives for access to https://www.osgeo.org/usage.php
 
*webalizer.conf - http auth and ssl directives for access to https://www.osgeo.org/usage.php
−
 
−
= OpenLDAP =
 
−
[http://www.openldap.org/doc/admin23/ OpenLDAP Admin Guide]
 
  
−
Currently ldap structure is pretty basic. The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.
+
= Postfix =
 +
[http://www.postfix.org/documentation.html Postfix documentation]
  
−
=== ldap structure ===
+
The postfix daemon can be be stared|stopped|restarted|reloaded with,
−
*dc=osgeo, dc=org
+
$ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart
−
**cn=Manager
 
−
**ou=people
 
−
***Separate entity for each user
 
−
*:uid=login, ou=people, dc=osgeo, dc=org
 
−
*:objectClass=inetOrgPerson
 
−
*:cn=firstName lastName
 
−
*:sn=lastName
 
−
*:uid=login
 
−
*:mail=email@address
 
−
*:userPassword={md5}YPTyViiMKhiuWKEmFUOKLA==
 
−
**ou=projects
 
−
***Separate entity for each project group
 
−
*:cn=admin,ou=projects,dc=osgeo,dc=org
 
−
*:objectClass=groupOfNames
 
−
*:cn=admin
 
−
*:description=
 
−
**ou=svn
 
−
***separate entity for each svn group
 
  
−
'''Example ldif file'''
+
The main postfix config file is,
−
  version: 1
+
/etc/postfix/main.cfg
  
−
  dn: dc=osgeo,dc=org
+
The aliases file is,
−
  objectClass: dcObject
+
/etc/aliases
−
  objectClass: organization
 
−
  description: OSGeo ldap dit
 
−
  o: OSGeo
 
−
  dc: osgeo
 
  
−
  dn: cn=Manager,dc=osgeo,dc=org
+
If aliases file is edited remember to run newaliases
−
  objectClass: organizationalRole
+
  $ sudo /usr/bin/newaliases
−
  cn: Manager
+
  $ sudo /sbin/service postfix reload
  
−
  dn: ou=people,dc=osgeo,dc=org
+
= LDAP =
−
  ou: people
 
−
  description: all users of osgeo
 
−
  objectClass: organizationalUnit
 
−
 
 
−
  dn: uid=jsmith,ou=people,dc=osgeo,dc=org
 
−
  objectClass: inetOrgPerson
 
−
  uid: jsmith
 
−
  cn: Jon Smith
 
−
  sn: Smith
 
−
  givenName: Jon
 
−
  mail: jsmith@somewhere.com
 
−
  userPassword: {md5}5Or4zfzGqo3jh/6iIUgKcA==
 
−
 
 
−
  dn: uid=jbrown,ou=people,dc=osgeo,dc=org
 
−
  objectClass: inetOrgPerson
 
−
  uid: jbrown
 
−
  cn: Jane Brown
 
−
  sn: Brown
 
−
  givenName: Jane
 
−
  mail: jbrown@someotherplace.com
 
−
  userPassword: {md5}1iWhTyvkK2m4Uuar+Dp/IA==
 
−
 
 
−
  dn: ou=projects,dc=osgeo,dc=org
 
−
  ou: projects
 
−
  description: separate entity for each osgeo project with list of members
 
−
  objectClass: organizationalUnit
 
−
 
 
−
  dn: cn=admin,ou=projects,dc=osgeo,dc=org
 
−
  cn: admin
 
−
  description: osgeo sysadmin group
 
−
  objectClass: groupOfNames
 
−
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
−
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
−
 
 
−
  dn: ou=svn,dc=osgeo,dc=org
 
−
  ou: svn
 
−
  description: separate entity for for each repository.list of members with commit rights
 
−
  objectClass: organizationalUnit
 
−
 
 
−
  dn: cn=fdo,ou=svn,dc=osgeo,dc=org
 
−
  objectClass: groupOfNames
 
−
  cn: fdo
 
−
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
−
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
−
 
 
−
  dn: cn=gdal,ou=svn,dc=osgeo,dc=org
 
−
  cn: gdal
 
−
  objectClass: groupOfNames
 
−
  objectClass: top
 
−
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
−
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
−
 
 
−
  dn: cn=mapbender,ou=svn,dc=osgeo,dc=org
 
−
  objectClass: groupOfNames
 
−
  cn: mapbender
 
−
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
−
 
 
−
  dn: cn=mapguide,ou=svn,dc=osgeo,dc=org
 
−
  objectClass: groupOfNames
 
−
  cn: mapguide
 
−
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
  
 +
See [[SAC:LDAP]]
  
 
= Subversion =
 
= Subversion =
−
[http://svnbook.red-bean.com/nightly/en/index.html Subversion Book]
 
−
=== Creating a repository ===
 
−
  $ sudo mkdir /var/www/svn/repos/<repo_name>
 
−
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
 
−
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
  
−
=== Loading a repository from a dumpfile ===
+
See [[Subversion]] for details on subversion configuration.
−
  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
 
−
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
  
−
=== Dumping a repository ===
+
= Trac =  
−
  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile
 
  
−
=== Recovering / unlocking repository ===
+
[[Trac Instances]]
−
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
 
−
* Need policy on who to contact and who can run 'svnadmin recover' as
 
−
priviledged access is needed (may need to stop/start apache to drop
 
−
requests to repository before recover)
 
  
−
Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.
+
= Mailman Maintenance =
  
−
If the repository locks the following command should be used to recover the repository:
+
See [[SAC:Mailing Lists]].
−
 
 
−
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 
−
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
−
 
 
−
In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered
 
−
 
 
−
  $ sudo /sbin/service httpd restart
 
−
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 
−
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
−
 
 
−
= TRAC =
 
−
[http://trac.edgewall.org/wiki/TracGuide Trac Guide]
 
−
=== Creating a trac instance ===
 
−
  $ sudo trac-admin /var/www/trac/<proj_name> initenv
 
−
  $ sudo chown -R apache:apache /var/www/trac/<proj_name>
 
−
 
 
−
=== Hotcopy a trac instance ===
 
−
  $ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to
 
−
 
 
−
= Mailman Maintenance =
 
−
=== create a new mailing list ===
 
−
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
 
−
=== renaming a list ===
 
−
* create new list
 
−
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
 
−
* move original lists archive to newlists archive location
 
−
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
 
−
    /var/lib/mailman/newlist/newlist.mbox/
 
−
* create archive
 
−
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
 
−
* export subscribers from old list regular and digest members
 
−
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
 
−
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
 
−
* import subscribers into new list
 
−
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
 
−
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
 
−
* remove old list
 
−
  $ sudo /usr/lib/mailman/bin/rmlist listname
 
−
* update aliases and check that proper permissions are set
 
−
  $ sudo /usr/lib/mailman/bin/genaliases
 
−
  $ sudo /usr/lib/mailman/bin/check_perms -f
 
  
 
= Backups =  
 
= Backups =  
  
−
=== Daily tape backup ===
+
See [[SAC:Backups]].
−
* PEER1 is doing daily tape backup of entire system
 
  
−
=== Other backups ===
+
= DNS =
−
==== Mysql ====
 
−
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
 
−
==== current cron jobs for backups ====
 
−
/etc/cron.d/backup.cron
 
  
−
  #### Backup cron jobs
+
See [[SAC DNS Registry]].
−
  # min hour day month dayofweek user command
 
−
  #
 
−
  # Daily rsync to /home/back
 
−
  # /etc
 
−
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
 
−
  # /var/www/html
 
−
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
 
−
  # Every 3 hours
 
−
  # /var/lib/mailman
 
−
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
 
−
  # /etc/mysql-zrm
 
−
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
 
−
  # /var/lib/mysql-zrm
 
−
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
 
−
  # call backup_trac_svn.sh to backup
 
−
  # subversion /var/www/svn/repos
 
−
  # trac /var/www/trac
 
−
  45 */3 * * * root /root/scripts/backup_trac_svn.sh
 
−
  # once a day rsync /home/back to osgeo2.osgeo.net
 
−
  0 22 * * * root /root/scripts/rsync_back.pl
 
−
 
 
−
backup script are in /root/scripts
 
−
 
 
−
'''backup_trac_svn.sh''' calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days.
 
−
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
 
−
 
 
−
= DNS =
 
−
dns is administered via [https://www.pairnic.com/login.html PairNIC]
 
  
 
= FDO Repository Merge =
 
= FDO Repository Merge =
Line 360: Line 190:
  
 
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
 
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
 +
 +
[[Category:Infrastructure]]

Latest revision as of 14:18, 2 February 2008

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

  • OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
  • For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

  • apache - httpd-2.0.52-28.ent.i386
  • postfix - postfix-2.2.10-1.RHEL4.2.i386
  • php - php-4.3.9-3.22PIDH.i386
  • python - python-2.3.4-14.3.i386
  • mailman - mailman-2.1.5.1-34.rhel4.5.i386

Red Hat EL 4 rpms installed manually (rpm -i)

  • MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
  • clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • sqlite-2.8.16-1.2.el4.rf.i386.rpm
  • python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • python-sqlite-1.0.1-12.el4.rf.i386.rpm
  • subversion-1.4.3-0.1.el4.rf.i386.rpm
  • mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

  • MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

  • drupal-4.7.4.tar.gz
  • phpldapadmin-0.9.8.3.tar.gz
  • trac-0.10.3.tar.gz

Paths to services directories

Apache root directory 

   /var/www/html/

Subversion parent directory 

   /var/www/svn/repos/

Trac parent directory 

  /var/www/trac/

IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH 

  *filter
  -A OUTPUT -p tcp --dport 22 -j ACCEPT
  COMMIT

/etc/sysconfig/iptables-custom/VERITAS

note: this file was placed here by PEER1 

  *filter
  -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
  COMMIT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.

Apache

As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/

Postfix

Postfix documentation

The postfix daemon can be be stared|stopped|restarted|reloaded with, 

$ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart

The main postfix config file is, 

/etc/postfix/main.cfg

The aliases file is, 

/etc/aliases

If aliases file is edited remember to run newaliases 

 $ sudo /usr/bin/newaliases
 $ sudo /sbin/service postfix reload

LDAP

See SAC:LDAP

Subversion

See Subversion for details on subversion configuration.

Trac

Trac Instances

Mailman Maintenance

See SAC:Mailing Lists.

Backups

See SAC:Backups.

DNS

See SAC DNS Registry.

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.


SVNDUMPFILTER

$ svndumpfilter include --help 

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics


This process followed for all repositories

merge fdogdal trunk 

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines, 

  Node-path: trunk/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/www/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/Providers/
  ....
  Node-action: add
  ....
  PROPS-END

Rename index.html to fdogdal-index.html 

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp

 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
Retrieved from "http://wiki.osgeo.org/w/index.php?title=Migration_Documentation&oldid=22354"