Revision as of 02:19, 17 January 2017

Backups

backup.osgeo.org

This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".

Admins: Frank Warmerdam (rsync), Martin Spott (backula)
access is via SAC shell group.
download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
/backup/bacula/ contains the Bacula storage device
/backup/rsync/ is for rsync backups
/backup/longterm/ is for long term backups of static material (ie conference web sites)
/backup/pgdump/ contains PostgreSQL dumps of the Bacula director database

OS installation

Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
Create a second partition on every disk - for use by MD - allocating all the remaining disk space
Create an MD RAID5 over all the 'second' partitions for use by LVM
Create a LVM volume group on top of the MD physical volume
Create volumes "boot" (256 MByte, Ext2, /boot/), "root" (20 GByte, XFS, /root/), "swap" (8 GByte), "var" (20 GByte, XFS, /var/)
Install the base operating system, don't forget to set up SSH ;-)
Add volumes "backup" (2 TByte, XFS, /backup/), "mirror" (1 TByte, XFS, /mirror/)

Bacula

Configuration as of June, 2011

A Full copy is made every Sat. with incremental backups nightly.

This consists of the following directories:

   File = /boot
   File = /etc
   File = /var/www
   File = /osgeo
   File = /var/lib/mysql
   File = /var/lib/postgresql
   File = /var/lib/dpkg
   File = /var/lib/aptitude/pkgstates

From each of the following machines:

wiki,secure,web,webextra,backup,qgis,projects,adhoc

Trac currently gets trac_backup.zip which is generated during the daily backup cycle

Administration

General configuration is managed in /etc/bacula/bacula-dir.conf.

To review the backups and perform a restore you can use the bconsole application:

 sudo bconsole

Specific configuration of the file/tape storage is in the Storage director /etc/bacula/bacula-sd.conf

Bacula uses the hard drive as a series of Tape drives.
Each Volume is a predetermined size (~500MB currently), when a volume fills bacula makes the next one.
Volumes older than a certain date or when volume increments have reach a configured maximum can be recycled into the available disk pool.

Recovery

If you use bconsole to recover data bacula does the work of finding the files you want and pulling them out of the various volumes.
Directory for recovered files is set in the bacula-dir.conf under:

Job {
 Name = "RestoreFiles"
 Type = Restore
 Client = backup.osgeo.osuosl.org-fd
 FileSet = "Full Set"
 Storage = File
 Pool = Default
 Messages = Standard
 Where = /tmp/bacula-restores
}

Directions for how to restore

sudo bconsole
*restore

Then follow the prompts, using the above linked guide to help find the files you want.

osgeo1 (primary peer1 host)

current cron jobs for backups

/etc/cron.d/backup.cron

#### Backup cron jobs 
# min hour day month dayofweek user command
#
# 2am: Daily rsync to /home/back
05 2 * * * root /root/scripts/daily.sh
# Every 3 hours 
# /etc/mysql-zrm
#33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
# /var/lib/mysql-zrm
#34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
# call backup_trac_svn.sh to backup 
# subversion /var/www/svn/repos
# trac /var/www/trac
45 */3 * * * root /root/scripts/backup_svn_incremental.sh
#45 */3 * * * root /root/scripts/trac_backup.pl
#45 */3 * * * root /root/scripts/backup_trac_svn.sh
# Thisdoes some hourly drupal/mysql optimization
39 * * * * root /root/scripts/drupal_cron.sh

backup script are in /root/scripts

osgeo3 & osgeo4

See: Infrastructure Transition Plan 2010

download.osgeo.org

As noted in Download Server the contents of download.osgeo.org are rsync'ed to osgeo2.

ProjectsVM

Bacula - we need details.

??? How does it work?

svn.osgeo.org

As noted in Subversion the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.

Difference between revisions of "SAC:Backups"