Difference between revisions of "User:Sbarnes"

From OSGeo
Jump to navigation Jump to search
 
(13 intermediate revisions by the same user not shown)
Line 35: Line 35:
 
* trac-0.10.3.tar.gz
 
* trac-0.10.3.tar.gz
  
−
= Apache =
+
= Paths to services directories =
 +
 
 +
'''Apache root directory'''
 
     /var/www/html/
 
     /var/www/html/
  
−
= Subversion =
+
'''Subversion parent directory'''
 
     /var/www/svn/repos/
 
     /var/www/svn/repos/
  
−
= Trac =
+
'''Trac parent directory'''
 
   /var/www/trac/
 
   /var/www/trac/
  
 
= IP tables custom =
 
= IP tables custom =
−
rules must be written here or won't exist when system updated
 
−
  /etc/sysconfig/iptables-custom/
 
  
 +
All iptables rules must be written in /etc/sysconfig/iptables-custom
 +
 +
Current custom rules are:
 +
 +
/etc/sysconfig/iptables-custom/SSH
 +
  *filter
 +
  -A OUTPUT -p tcp --dport 22 -j ACCEPT
 +
  COMMIT
 +
 +
/etc/sysconfig/iptables-custom/VERITAS
 +
 +
''note: this file was placed here by PEER1''
 +
  *filter
 +
  -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
 +
  -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
 +
  COMMIT
  
 
= System login and maintenance procedures =
 
= System login and maintenance procedures =
  
−
-- Note: if change root password must let PEER1 know the new password
+
'''Note: if change root password must let PEER1 know the new password for backup/restore and tickets.  i would refer no root ssh login but, PEER1 admins need root access.'''
−
for backup/restore and tickets.  i would prefer no root ssh login but,
 
−
PEER1 needs root access.
 
  
 
= Subversion maintanence =
 
= Subversion maintanence =
 +
[http://svnbook.red-bean.com/nightly/en/index.html Subversion Book]
 +
=== Creating a repository ===
 +
  $ sudo mkdir /var/www/svn/repos/<repo_name>
 +
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
 +
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 +
 +
=== Loading a repository from a dumpfile ===
 +
  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
 +
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 +
 +
=== Dumping a repository ===
 +
  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile
 +
 +
=== Recovering / unlocking repository ===
 
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
 
* subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
 
* Need policy on who to contact and who can run 'svnadmin recover' as
 
* Need policy on who to contact and who can run 'svnadmin recover' as
Line 61: Line 89:
 
requests to repository before recover)
 
requests to repository before recover)
  
−
= Authorization and authentication for OSGeo services =
+
Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.
  
−
= Backups (offsite, live, and otherwise) =
+
If the repository locks the following command should be used to recover the repository:
  
 +
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 +
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 +
 +
In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered
 +
 +
  $ sudo /sbin/service httpd restart
 +
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
 +
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>
 +
 +
= TRAC Maintenance =
 +
[http://trac.edgewall.org/wiki/TracGuide Trac Guide]
 +
=== Creating a trac instance ===
 +
  $ sudo trac-admin /var/www/trac/<proj_name> initenv
 +
  $ sudo chown -R apache:apache /var/www/trac/<proj_name>
 +
 +
=== Hotcopy a trac instance ===
 +
  $ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to
 +
 +
= Mailman Maintenance =
 +
=== create a new mailing list ===
 +
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
 +
=== renaming a list ===
 +
* create new list
 +
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
 +
* move original lists archive to newlists archive location
 +
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
 +
    /var/lib/mailman/newlist/newlist.mbox/
 +
* create archive
 +
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
 +
* export subscribers from old list regular and digest members
 +
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
 +
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
 +
* import subscribers into new list
 +
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
 +
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
 +
* remove old list
 +
  $ sudo /usr/lib/mailman/bin/rmlist listname
 +
* update aliases and check that proper permissions are set
 +
  $ sudo /usr/lib/mailman/bin/genaliases
 +
  $ sudo /usr/lib/mailman/bin/check_perms -f
 +
 +
= Backups =
 +
 +
=== Daily tape backup ===
 
* PEER1 is doing daily tape backup of entire system
 
* PEER1 is doing daily tape backup of entire system
  
−
== Other backups ==
+
=== Other backups ===
−
* MySQL being backed up every 2 hours with MySQL-zrm
+
==== Mysql ====
−
* rsync backups, to /home/back then rsynced daily to other server
+
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
−
(osgeo.net)
+
==== current cron jobs for backups ====
−
      /etc/ (daily)
+
/etc/cron.d/backup.cron
−
      /var/www/html (daily)
+
 
−
  /var/lib/mailman (every 3 hours)
+
  #### Backup cron jobs
−
          /var/lib/msql-zrm (every 3 hours)
+
  # min hour day month dayofweek user command
−
* subversion backups
+
  #
−
      -via svnadmin dump to /home/back/svn_dump/ (every 3 hours)
+
  # Daily rsync to /home/back
−
* trac backups
+
  # /etc
−
      -via trac-admin hotcopy to /home/back/trac_copy/ (every 3 hours)
+
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
 +
  # /var/www/html
 +
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
 +
  # Every 3 hours
 +
  # /var/lib/mailman
 +
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
 +
  # /etc/mysql-zrm
 +
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
 +
  # /var/lib/mysql-zrm
 +
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
 +
  # call backup_trac_svn.sh to backup
 +
  # subversion /var/www/svn/repos
 +
  # trac /var/www/trac
 +
  45 */3 * * * root /root/scripts/backup_trac_svn.sh
 +
  # once a day rsync /home/back to osgeo2.osgeo.net
 +
  0 22 * * * root /root/scripts/rsync_back.pl
 +
 
 +
backup script are in /root/scripts
  
−
= Implementation of our virtual hosts policy and transition issues =
+
'''backup_trac_svn.sh''' calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days.
 +
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
  
 
= FDO Repository Merge =
 
= FDO Repository Merge =

Latest revision as of 12:36, 31 January 2007

Sandbox Page for osgeo.org transition docs

Description of current layout and installed software

Notes from Shawn on our PEER1 system:

  • OS Version: Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
  • For most elements of system i tried to stay with the default redhat locations and red hat el 4 rpms installed from peer 1's up2date repository. Reasoning, the servers are updated automatically against this repository and supported by PEER1 - reduce sysadmin load on keeping packages updated by using packages not available through PEER1 repository

Installed software using up2date

  • apache - httpd-2.0.52-28.ent.i386
  • postfix - postfix-2.2.10-1.RHEL4.2.i386
  • php - php-4.3.9-3.22PIDH.i386
  • python - python-2.3.4-14.3.i386
  • mailman - mailman-2.1.5.1-34.rhel4.5.i386

Red Hat EL 4 rpms installed manually (rpm -i)

  • MySQL-client-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-server-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-devel-standard-5.0.27-0.rhel4.i386.rpm
  • MySQL-shared-compat-5.0.27-0.rhel4.i386.rpm
  • clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • sqlite-2.8.16-1.2.el4.rf.i386.rpm
  • python-clearsilver-0.10.1-1.2.el4.rf.i386.rpm
  • python-sqlite-1.0.1-12.el4.rf.i386.rpm
  • subversion-1.4.3-0.1.el4.rf.i386.rpm
  • mod_dav_svn-1.4.3-0.1.el4.rf.i386.rpm

Not specific to Red Hat EL 4 rpms installed manually

  • MySQL-zrm-1.1.2-1.noarch.rpm

Source Install

  • drupal-4.7.4.tar.gz
  • phpldapadmin-0.9.8.3.tar.gz
  • trac-0.10.3.tar.gz

Paths to services directories

Apache root directory 

   /var/www/html/

Subversion parent directory 

   /var/www/svn/repos/

Trac parent directory 

  /var/www/trac/

IP tables custom

All iptables rules must be written in /etc/sysconfig/iptables-custom

Current custom rules are:

/etc/sysconfig/iptables-custom/SSH 

  *filter
  -A OUTPUT -p tcp --dport 22 -j ACCEPT
  COMMIT

/etc/sysconfig/iptables-custom/VERITAS

note: this file was placed here by PEER1 

  *filter
  -A INPUT -s 10.0.48.0/24 -d 0/0 -p tcp -j ACCEPT
  -A OUTPUT -s 0/0 -d 10.0.48.0/24 -p tcp -j ACCEPT
  COMMIT

System login and maintenance procedures

Note: if change root password must let PEER1 know the new password for backup/restore and tickets. i would refer no root ssh login but, PEER1 admins need root access.

Subversion maintanence

Subversion Book

Creating a repository

  $ sudo mkdir /var/www/svn/repos/<repo_name>
  $ sudo svnadmin /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Loading a repository from a dumpfile

  $ sudo svnadmin load /var/www/svn/repos/<repo_name> < /path/to/dumpfile
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

Dumping a repository

  $ sudo svnadmin dump /var/www/svn/repos/<repo_name> > dumpfile

Recovering / unlocking repository

  • subversion may lock if user ctl-c during checkout or checkout is interupted with an apache restart
  • Need policy on who to contact and who can run 'svnadmin recover' as

priviledged access is needed (may need to stop/start apache to drop requests to repository before recover)

Subversion was upgraded to version 1.4.3 (30 Jan. 2007) to reduce the repository locking problem if ctl-c used to end a checkout.

If the repository locks the following command should be used to recover the repository: 

  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

In most cases this will work in the odd case that you are waiting a very long time for the command to run then apache may have to be restarted to drop anything accessing the repository and preventing the repository from being recovered 

  $ sudo /sbin/service httpd restart
  $ sudo svnadmin --wait recover /var/www/svn/repos/<repo_name>
  $ sudo chown -R apache:apache /var/www/svn/repos/<repo_name>

TRAC Maintenance

Trac Guide

Creating a trac instance

  $ sudo trac-admin /var/www/trac/<proj_name> initenv
  $ sudo chown -R apache:apache /var/www/trac/<proj_name>

Hotcopy a trac instance

  $ sudo trac-admin /var/www/html/trac/<proj_name> hotcopy /path/to/copy/trac/to

Mailman Maintenance

create a new mailing list

  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd

renaming a list

  • create new list
  $sudo /usr/lib/mailman/bin/newlist listname admins@email lists_passwd
  • move original lists archive to newlists archive location
  $ sudo cp /var/lib/mailman/oldlist/oldlist.mbox/oldlist.mbox \
    /var/lib/mailman/newlist/newlist.mbox/
  • create archive
  $ sudo /usr/lib/mailman/bin/arch --wipe newlist
  • export subscribers from old list regular and digest members
  $ sudo /usr/lib/mailman/bin/list_members -r listname > listname-regular.txt
  $ sudo /usr/lib/mailman/bin/list_members -d listname > listname-digest.txt
  • import subscribers into new list
  $ sudo /usr/lib/mailman/bin/add_members --regular-members=listname-regular.txt --welcome-msg=y listname
  $ sudo /usr/lib/mailman/bin/add_members --digest-members=listname-digest.txt --welcome-msg=y listname
  • remove old list
  $ sudo /usr/lib/mailman/bin/rmlist listname
  • update aliases and check that proper permissions are set
  $ sudo /usr/lib/mailman/bin/genaliases
  $ sudo /usr/lib/mailman/bin/check_perms -f

Backups

Daily tape backup

  • PEER1 is doing daily tape backup of entire system

Other backups

Mysql

MySQL is being backed up using MySQL-zrm

current cron jobs for backups

/etc/cron.d/backup.cron 

  #### Backup cron jobs 
  # min hour day month dayofweek user command
  #
  # Daily rsync to /home/back
  # /etc
  05 10 * * * root /usr/bin/rsync -a --delete /etc/ /home/back/etc/
  # /var/www/html
  08 10 * * * root /usr/bin/rsync -a --delete /var/www/html/ /home/back/html/
  # Every 3 hours 
  # /var/lib/mailman
  20 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mailman/ /home/back/mailman/
  # /etc/mysql-zrm
  33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
  # /var/lib/mysql-zrm
  34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
  # call backup_trac_svn.sh to backup 
  # subversion /var/www/svn/repos
  # trac /var/www/trac
  45 */3 * * * root /root/scripts/backup_trac_svn.sh
  # once a day rsync /home/back to osgeo2.osgeo.net
  0 22 * * * root /root/scripts/rsync_back.pl

backup script are in /root/scripts

backup_trac_svn.sh calls separate scripts to backup trac and svn repositories and place tgz files of the backups in /home/back/svn_backup and /home/back/trac_backup which are held for 14 days. rsync_back rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)

FDO Repository Merge

To merge repositories you need to parse through the dumpfiles.

In the fdo merge all subprojects had the same directory strucuture, /trunk/Providers/ All dumpfiles had to be 'filtered' to just pull the /trunk/Providers/* and drop branches and tags.


SVNDUMPFILTER

$ svndumpfilter include --help 

Filter out nodes without given prefixes from dumpstream

Usage: svndumpfilter include PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics

$ svndumpfilter exclude --help Filter out nodes with given prefixes from dumpstream Usage: svndumpfilter exclude PATH_PREFIX Options: --drop-empty-revs Remove revisions emptied by filtering --renumber-revs Renumbe revisions left after filtering --preserve-revprops Don't filter revision properties --quiet Do not display filtering statistics


This process followed for all repositories

merge fdogdal trunk 

 $ cat fdogdal.dmp | svndumpfilter include trunk/www/ > fdogdal-merge-trunk.www.dmp
 $ cat fdogdal.dmp | svndumpfilter include trunk/Providers/ > fdogdal-merge-trunk.Providers.GDAL.dmp

edit file and remove lines, 

  Node-path: trunk/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/www/
  ....
  Node-action: add
  ....
  PROPS-END

  Node-path: trunk/Providers/
  ....
  Node-action: add
  ....
  PROPS-END

Rename index.html to fdogdal-index.html 

 $ perl -pi.bak -e 's/^Node-path:\ trunk\/www\/index.html/Node-path:\ trunk\/www\/fdogdal-index.html/g' fdo-merge.trunk.dmp

 $ svnadmin load /var/www/svn/repos/fdocore < fdogdal-merge.trunk.dmp
Retrieved from "http://wiki.osgeo.org/w/index.php?title=User:Sbarnes&oldid=11182"