Migration Documentation
Description of current layout and installed software
Notes from Shawn on our PEER1 system:
- OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
- For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository
Installed software using up2date
- apache - httpd-2.0.52-28.ent.i386
- postfix - postfix-2.2.10-1.RHEL4.2.i386
- php - php-4.3.9-3.22PIDH.i386
- python - python-2.3.4-14.3.i386
- mailman - mailman-2.1.5.1-34.rhel4.5.i386
Red Hat EL 4 rpms installed manually (rpm -i)
- MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
- MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
- MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
- MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
- clearsilver-0.10.1-1.2.el4.rf.i386.rpm
- sqlite-2.8.16-1.2.el4.rf.i386.rpm
- python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
- python-sqlite-1.0.1-12.el4.rf.i386.rpm
- subversion-1.4.3-0.1.el4.rf.i386.rpm
- mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm
Not specific to Red Hat EL 4 rpms installed manually
- MySQL-zrm-1.1.2-1.noarch.rpm
Source Install
- drupal-4.7.4.tar.gz
- phpldapadmin-0.9.8.3.tar.gz
- trac-0.10.3.tar.gz
Paths to services directories
Apache root directory
/var/www/html/
Subversion parent directory
/var/www/svn/repos/
Trac parent directory
/var/www/trac/
IP tables custom
All iptables rules must be written in /etc/sysconfig/iptables-custom
Current custom rules are:
/etc/sysconfig/iptables-custom/SSH
*filter -A OUTPUT -p tcp --dport 22 -j ACCEPT COMMIT
/etc/sysconfig/iptables-custom/VERITAS
note: this file was placed here by PEER1
*filter -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT COMMIT
System login and maintenance procedures
Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.
Apache maintenance
As the system is Red Hat use /sbin/services httpd start|stop|restart|configtest|reload instead of /usr/sbin/apchectl
the main Config file for apache is, /etc/httpd/conf/httpd.conf
additional config files are at /etc/httpd/conf.d/
- ssl.conf
- subversion.conf
- trac.conf
- virtual_host.conf
- rewrite.conf
- mailman.conf
- trac.gdal.conf
- php.conf
- python.conf
- perl.conf
- phpldapadmin.conf - http auth and ssl directives for access to https://www.osgeo.org/ldapadmin
- working.conf - http auth and ssl directives for access to https://www.osgeo.org/_ldap/ldap.php
- webalizer.conf - http auth and ssl directives for access to https://www.osgeo.org/usage.php
Subversion maintanence
Creating a repository
$ sudo mkdir /var/www/svn/repos/<repo_name> $ sudo svnadmin /var/www/svn/repos/<repo_name> $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
Loading a repository from a dumpfile
$ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
Dumping a repository
$ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile
Recovering / unlocking repository
- subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
- Need policy on who to contact and who can run 'svnadmin recover' as
priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)
Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.
If the repository locks the following command should be used to recover the repository:
$ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name> $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered
$ sudo /sbin/service httpd restart $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name> $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
TRAC Maintenance
Creating a trac instance
$ sudo trac-admin /var/www/trac/<proj_name> initenv $ sudo chown -R apache:apache /var/www/trac/<proj_name>
Hotcopy a trac instance
$ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to
Mailman Maintenance
create a new mailing list
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
renaming a list
- create new list
$sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
- move original lists archive to newlists archive location
$ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
/var/lib/mailman/newlist/newlist.mbox/
- create archive
$ sudo /usr/lib/mailman/bin/arch --wipe newlist
- export subscribers from old list regular and digest members
$ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
- import subscribers into new list
$ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
- remove old list
$ sudo /usr/lib/mailman/bin/rmlist listname
- update aliases and check that proper permissions are set
$ sudo /usr/lib/mailman/bin/genaliases $ sudo /usr/lib/mailman/bin/check_perms -f
Backups
Daily tape backup
- PEER1 is doing daily tape backup of entire system
Other backups
Mysql
MySQL is being backed up using MySQL-zrm
current cron jobs for backups
/etc/cron.d/backup.cron
#### Backup cron jobs # min hour day month dayofweek user command # # Daily rsync to /home/back # /etc 05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/ # /var/www/html 08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/ # Every 3 hours # /var/lib/mailman 20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/ # /etc/mysql-zrm 33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/ # /var/lib/mysql-zrm 34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/ # call backup_trac_svn.sh to backup # subversion /var/www/svn/repos # trac /var/www/trac 45 */3 * * * root /root/scripts/backup_trac_svn.sh # once a day rsync /home/back to osgeo2.osgeo.net 0 22 * * * root /root/scripts/rsync_back.pl
backup script are in /root/scripts
backup_trac_svn.sh calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days. rsync_back rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
FDO Repository Merge
To merge repositories you need to parse through the dumpfiles.
In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.
SVNDUMPFILTER
$ svndumpfilter include --help
Filter out nodes without given prefixes from dumpstream
Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics
$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics
This process followed for all repositories
merge fdogdal trunk
$ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp
edit file and remove lines,
Node-path: trunk/ .... Node-action: add .... PROPS-END
Node-path: trunk/www/ .... Node-action: add .... PROPS-END
Node-path: trunk/Providers/ .... Node-action: add .... PROPS-END
Rename index.html to fdogdal-index.html
$ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp
$ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp