Difference between revisions of "Migration Documentation"

From OSGeo
Jump to navigation Jump to search
m (→‎LDAP: move out LDAP content to another page.)
 
(8 intermediate revisions by 2 users not shown)
Line 106: Line 106:
 
   $ sudo /sbin/service postfix reload
 
   $ sudo /sbin/service postfix reload
  
= OpenLDAP =
+
= LDAP =
[http://www.openldap.org/doc/admin23/ OpenLDAP Admin Guide]
 
  
The ldap daemon can manipulated with,
+
See [[SAC:LDAP]]
$ sudo /sbin/service ldap start|stop|restart|
 
 
 
The main ldap config file is,
 
/etc/openldap/slapd.conf
 
 
 
Currently ldap structure is pretty basic.  The purpose for keeping this structure simple is to allow for a more complex structure to be evolved as ldap becomes increasingly integrated into the full osgeo systems structure.
 
 
 
''' Loading an ldif file into an ldap directory '''
 
  $ ldapadd -a -W -x -D "cn=Manager,dc=osgeo,dc=org" -f fileName.ldif
 
 
 
''' Accessing the ldap directory through phpldapadmin '''
 
 
 
access an ldapadmin interface at https://www.osgeo.org/ldapadmin
 
 
 
Access to this interface is restricted to the admin group
 
 
 
users can be added to the ou=people,dc=osgeo,dc=org
 
once users are added they can then be added to
 
cn=project,ou=svn,dc=osgeo,dc=org
 
 
 
login as cn=Manager,dc=osgeo,dc=org
 
 
 
*expand dc=osgeo,dc=org tree
 
*expand the ou=people tree
 
*click on 'create new entry here'
 
*choose 'custom' and then 'proceed'
 
**RDN is in the form of uid=username
 
**ObjectClass is InetOrgPerson
 
**click proceed
 
***cn is common name in form of firstname lastname
 
***sn is surname - lastname
 
***no optionals are required
 
**click 'create object'
 
***on this page click 'add new attribute'
 
****choose mail
 
*****enter email address
 
*****click add
 
****on this page click 'add new attribute'
 
*****choose userPassord
 
*****enter password as an md5
 
*****click add
 
 
 
this adds the new user to ou=people
 
 
 
to add users to the cn=project,ou=svn group
 
*expand the ou=svn tree
 
*click on cn=project
 
**under member click 'add value'
 
**add user to this in same form as you and howard are listed
 
**or can click on the 'folder with magnifying glasss' to select a user
 
 
 
that's it.
 
 
 
=== ldap structure ===
 
*dc=osgeo,dc=org
 
 
 
**cn=Manager
 
 
 
**ou=people
 
 
 
***Separate entity for each user
 
 
 
*:uid=login,ou=people,dc=osgeo,dc=org
 
*:objectClass=inetOrgPerson
 
*:cn=firstName lastName
 
*:sn=lastName
 
*:uid=login
 
*:mail=email@address
 
*:userPassword={md5}YPTyViiMKhiuWKEmFUOKLA==
 
 
 
**ou=projects,dc=osgeo,dc=org
 
*:objectClass=organizationalUnit
 
*:ou=project
 
*:description=separate entity for each osgeo project with list of members
 
 
 
***Separate entity for each project group
 
*:cn=admin,ou=projects,dc=osgeo,dc=org
 
*:objectClass=groupOfNames
 
*:cn=admin
 
*:description=osgeo sysadmin group
 
 
 
**ou=svn
 
*:objectClass=organizationalUnit
 
*:ou=svn
 
*:description=separate entity for each repository with list of members with commit rights
 
 
 
***separate entity for each svn group
 
*:cn=fdo,ou=svn,dc=osgeo,dc=org
 
*:objectClass=groupOfNames
 
*:cn=fdo
 
*:member= dn of member
 
 
 
*:cn=gdal,ou=svn,dc=osgeo,dc=org
 
*:objectClass=groupOfNames
 
*:cn=gdal
 
*:member= dn of member
 
 
 
*:cn=mapguide,ou=svn,dc=osgeo,dc=org
 
*:objectClass=groupOfNames
 
*:cn=mapguide
 
*:member= dn of member
 
 
 
*:cn=mapbender,ou=svn,dc=osgeo,dc=org
 
*:objectClass=groupOfNames
 
*:cn=mapbender
 
*:member= dn of member
 
 
 
 
 
'''Example ldif file'''
 
<pre><code>
 
  version: 1
 
 
 
  dn: dc=osgeo,dc=org
 
  objectClass: dcObject
 
  objectClass: organization
 
  description: OSGeo ldap dit
 
  o: OSGeo
 
  dc: osgeo
 
 
 
  dn: cn=Manager,dc=osgeo,dc=org
 
  objectClass: organizationalRole
 
  cn: Manager
 
 
 
  dn: ou=people,dc=osgeo,dc=org
 
  ou: people
 
  description: all users of osgeo
 
  objectClass: organizationalUnit
 
 
 
  dn: uid=jsmith,ou=people,dc=osgeo,dc=org
 
  objectClass: inetOrgPerson
 
  uid: jsmith
 
  cn: Jon Smith
 
  sn: Smith
 
  givenName: Jon
 
  mail: jsmith@somewhere.com
 
  userPassword: {md5}5Or4zfzGqo3jh/6iIUgKcA==
 
 
 
  dn: uid=jbrown,ou=people,dc=osgeo,dc=org
 
  objectClass: inetOrgPerson
 
  uid: jbrown
 
  cn: Jane Brown
 
  sn: Brown
 
  givenName: Jane
 
  mail: jbrown@someotherplace.com
 
  userPassword: {md5}1iWhTyvkK2m4Uuar+Dp/IA==
 
 
 
  dn: ou=projects,dc=osgeo,dc=org
 
  ou: projects
 
  description: separate entity for each osgeo project with list of members
 
  objectClass: organizationalUnit
 
 
 
  dn: cn=admin,ou=projects,dc=osgeo,dc=org
 
  cn: admin
 
  description: osgeo sysadmin group
 
  objectClass: groupOfNames
 
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
 
 
  dn: ou=svn,dc=osgeo,dc=org
 
  ou: svn
 
  description: separate entity for for each repository.list of members with commit rights
 
  objectClass: organizationalUnit
 
 
 
  dn: cn=fdo,ou=svn,dc=osgeo,dc=org
 
  objectClass: groupOfNames
 
  cn: fdo
 
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
 
 
  dn: cn=gdal,ou=svn,dc=osgeo,dc=org
 
  cn: gdal
 
  objectClass: groupOfNames
 
  objectClass: top
 
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
 
 
  dn: cn=mapbender,ou=svn,dc=osgeo,dc=org
 
  objectClass: groupOfNames
 
  cn: mapbender
 
  member: uid=jsmith,ou=people,dc=osgeo,dc=org
 
 
 
  dn: cn=mapguide,ou=svn,dc=osgeo,dc=org
 
  objectClass: groupOfNames
 
  cn: mapguide
 
  member: uid=jbrown,ou=people,dc=osgeo,dc=org
 
</code></pre>
 
  
 
= Subversion =
 
= Subversion =
[http://svnbook.red-bean.com/nightly/en/index.html Subversion Book]
 
=== Creating a repository ===
 
  $ sudo mkdir /var/www/svn/repos/<repo_name>
 
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
 
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
  
=== Loading a repository from a dumpfile ===
+
See [[Subversion]] for details on subversion configuration.
  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
 
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
 
 
=== Dumping a repository ===
 
  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile
 
 
 
=== Recovering / unlocking repository ===
 
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
 
* Need policy on who to contact and who can run 'svnadmin recover' as
 
priviledged access is needed (may need to stop/start apache to drop
 
requests to repository before recover)
 
 
 
Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.
 
 
 
If the repository locks the following command should be used to recover the repository:
 
 
 
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
 
 
In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered
 
 
 
  $ sudo /sbin/service httpd restart
 
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 
  
 
= Trac =  
 
= Trac =  
Line 335: Line 119:
  
 
= Mailman Maintenance =
 
= Mailman Maintenance =
=== create a new mailing list ===
+
 
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
+
See [[SAC:Mailing Lists]].
=== renaming a list ===
 
* create new list
 
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
 
* move original lists archive to newlists archive location
 
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
 
    /var/lib/mailman/newlist/newlist.mbox/
 
* create archive
 
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
 
* export subscribers from old list regular and digest members
 
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
 
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
 
* import subscribers into new list
 
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
 
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
 
* remove old list
 
  $ sudo /usr/lib/mailman/bin/rmlist listname
 
* update aliases and check that proper permissions are set
 
  $ sudo /usr/lib/mailman/bin/genaliases
 
  $ sudo /usr/lib/mailman/bin/check_perms -f
 
* edit postfix aliases - /etc/aliases
 
  oldlist:              newlist@lists.osgeo.org
 
  oldlist-request:      newlist@lists.osgeo.org
 
  oldlist-admin:         newlist@lists.osgeo.org
 
  oldlist-owner:        newlist@lists.osgeo.org
 
* update postfix with new aliases
 
  $ sudo /usr/bin/newaliases
 
  $ sudo /sbin/service postfix reload
 
  
 
= Backups =  
 
= Backups =  
  
=== Daily tape backup ===
+
See [[SAC:Backups]].
* PEER1 is doing daily tape backup of entire system
 
 
 
=== Other backups ===
 
==== Mysql ====
 
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
 
==== current cron jobs for backups ====
 
/etc/cron.d/backup.cron
 
 
 
  #### Backup cron jobs
 
  # min hour day month dayofweek user command
 
  #
 
  # Daily rsync to /home/back
 
  # /etc
 
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
 
  # /var/www/html
 
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
 
  # Every 3 hours
 
  # /var/lib/mailman
 
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
 
  # /etc/mysql-zrm
 
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
 
  # /var/lib/mysql-zrm
 
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
 
  # call backup_trac_svn.sh to backup
 
  # subversion /var/www/svn/repos
 
  # trac /var/www/trac
 
 
 
  0 1 * * * root /root/scripts/backup_svn_full.sh
 
  45 */3 * * * root /root/scripts/backup_svn_incremental.sh
 
  45 */3 * * * root /root/scripts/trac_backup.pl
 
 
 
  # once a day rsync /home/back to osgeo2.osgeo.net
 
  0 22 * * * root /root/scripts/rsync_back.pl
 
 
 
backup script are in /root/scripts
 
 
 
backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental
 
 
 
backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full.  14 copies of the repository are kept. 
 
 
 
When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects.  You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. 
 
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
 
  
 
= DNS =
 
= DNS =
dns is administered via [https://www.pairnic.com/login.html PairNIC]
 
 
  
The zone file for the osgeo.org dns was mirrored from CN's zonefile.  There is some work here to get rid of * (wildcard) entry in dns.
+
See [[SAC DNS Registry]].
 
 
'''current dns records'''
 
@             IN A      66.223.95.242 
 
www                IN A      66.223.95.242 
 
mapguide     IN A      66.223.95.245
 
find.geodata     IN A      198.202.74.216
 
mapbender2     IN A      66.223.95.245
 
dev.geodata     IN A      198.202.74.216
 
wiki             IN A      85.214.35.116
 
svn             IN A      66.223.95.242
 
geodata     IN A      66.223.95.242
 
*.wiki     IN A      85.214.35.116
 
access.geodata      IN A      198.202.74.216
 
buildbot     IN A      198.202.74.219
 
fdo2             IN A      66.223.95.245
 
www.communitymapbuilder IN A  63.246.7.187
 
*             IN A      66.223.95.242
 
community     IN A      66.223.95.242
 
download     IN A      198.202.74.219
 
testbed     IN A      140.211.166.123
 
lists             IN A      66.223.95.242
 
mapguide2     IN A      66.223.95.245
 
*.community     IN A      140.211.166.123
 
mapbender     IN A      66.223.95.245
 
mail             IN A      66.223.95.242
 
public.geodata      IN A      198.202.74.216
 
communitymapbuilder IN A      63.246.7.187
 
fdo                IN A      66.223.95.245
 
 
 
MX records
 
lists.osgeo.org
 
mail.osgeo.org
 
 
 
CNAME
 
gdal.osgeo.org    IN CNAME  www.gdal.org
 
  
 
= FDO Repository Merge =
 
= FDO Repository Merge =
Line 512: Line 190:
  
 
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
 
   $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
 +
 +
[[Category:Infrastructure]]

Latest revision as of 14:18, 2 February 2008

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

  • OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
  • For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

  • apache - httpd-2.0.52-28.ent.i386
  • postfix - postfix-2.2.10-1.RHEL4.2.i386
  • php - php-4.3.9-3.22PIDH.i386
  • python - python-2.3.4-14.3.i386
  • mailman - mailman-2.1.5.1-34.rhel4.5.i386

Red Hat EL 4 rpms installed manually (rpm -i)

  • MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
  • clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • sqlite-2.8.16-1.2.el4.rf.i386.rpm
  • python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • python-sqlite-1.0.1-12.el4.rf.i386.rpm
  • subversion-1.4.3-0.1.el4.rf.i386.rpm
  • mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

  • MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

  • drupal-4.7.4.tar.gz
  • phpldapadmin-0.9.8.3.tar.gz
  • trac-0.10.3.tar.gz

Paths to services directories

Apache root directory

   /var/www/html/

Subversion parent directory

   /var/www/svn/repos/

Trac parent directory

  /var/www/trac/

IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH

  *filter
  -A OUTPUT -p tcp --dport 22 -j ACCEPT
  COMMIT

/etc/sysconfig/iptables-custom/VERITAS

note: this file was placed here by PEER1

  *filter
  -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
  COMMIT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.

Apache

As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl

the main Config file for apache is, /etc/httpd/conf/httpd.conf

additional config files are at /etc/httpd/conf.d/

Postfix

Postfix documentation

The postfix daemon can be be stared|stopped|restarted|reloaded with,

$ sudo /sbin/service postfix start|stop|restart|reload|abort|flush|check|status|condrestart

The main postfix config file is,

/etc/postfix/main.cfg

The aliases file is,

/etc/aliases

If aliases file is edited remember to run newaliases

 $ sudo /usr/bin/newaliases
 $ sudo /sbin/service postfix reload

LDAP

See SAC:LDAP

Subversion

See Subversion for details on subversion configuration.

Trac

Trac Instances

Mailman Maintenance

See SAC:Mailing Lists.

Backups

See SAC:Backups.

DNS

See SAC DNS Registry.

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.


SVNDUMPFILTER

$ svndumpfilter include --help

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics


This process followed for all repositories

merge fdogdal trunk

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines,

  Node-path: trunk/
  ....
  Node-action: add
  ....
  PROPS-END
  Node-path: trunk/www/
  ....
  Node-action: add
  ....
  PROPS-END
  Node-path: trunk/Providers/
  ....
  Node-action: add
  ....
  PROPS-END

Rename index.html to fdogdal-index.html

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp
 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp