Difference between revisions of "SAC:Backups"

From OSGeo
Jump to: navigation, search
(backup.osgeo.org)
(Bacula: turned to a link)
Line 24: Line 24:
  
 
=== Bacula ===
 
=== Bacula ===
Configuration as of June, 2011
 
  
A Full copy is made every Sat. with incremental backups nightly.
+
See [[SAC:Bacula]]
 
+
This consists of the following directories:
+
    File = /boot
+
    File = /etc
+
    File = /var/www
+
    File = /osgeo
+
    File = /var/lib/mysql
+
    File = /var/lib/postgresql
+
    File = /var/lib/dpkg
+
    File = /var/lib/aptitude/pkgstates
+
From each of the following machines:
+
* wiki,secure,web,webextra,backup,qgis,projects,adhoc
+
 
+
Trac currently gets trac_backup.zip which is generated during the daily backup cycle
+
 
+
==== Administration ====
+
General configuration is managed in /etc/bacula/bacula-dir.conf.
+
 
+
To review the backups and perform a restore you can use the [http://www.bacula.org/5.2.x-manuals/en/console/console/index.html bconsole application]:
+
  sudo bconsole
+
 
+
Specific configuration of the file/tape storage is in the Storage director /etc/bacula/bacula-sd.conf
+
* Bacula uses the hard drive as a series of Tape drives.
+
* Each Volume is a predetermined size (~500MB currently), when a volume fills bacula makes the next one.
+
* Volumes older than a certain date or when volume increments have reach a configured maximum can be recycled into the available disk pool.
+
 
+
==== Recovery ====
+
* If you use bconsole to recover data bacula does the work of finding the files you want and pulling them out of the various volumes.
+
* Directory for recovered files is set in the bacula-dir.conf under:
+
Job {
+
  Name = "RestoreFiles"
+
  Type = Restore
+
  Client = backup.osgeo.osuosl.org-fd
+
  FileSet = "Full Set"
+
  Storage = File
+
  Pool = Default
+
  Messages = Standard
+
  Where = /tmp/bacula-restores
+
}
+
* Directions for [http://www.bacula.org/5.2.x-manuals/en/main/main/Restore_Command.html how to restore]
+
sudo bconsole
+
*restore
+
Then follow the prompts, using the above linked guide to help find the files you want.
+
  
 
== osgeo1 (primary peer1 host) ==
 
== osgeo1 (primary peer1 host) ==

Revision as of 02:30, 9 February 2017

Backups

backup.osgeo.org

This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".

  • Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
  • access is via SAC shell group.
  • download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
  • /backup/bacula/ contains the Bacula storage device
  • /backup/rsync/ is for rsync backups
  • /backup/longterm/ is for long term backups of static material (ie conference web sites)
  • /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
  • /etc/bacula contains Bacula configuration, is under local git repo

OS installation

  • Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
  • Create a second partition on every disk - for use by MD - allocating all the remaining disk space
  • Create an MD RAID5 over all the 'second' partitions for use by LVM
  • Create a LVM volume group on top of the MD physical volume
  • Create volumes "boot" (256 MByte, Ext2, /boot/), "root" (20 GByte, XFS, /root/), "swap" (8 GByte), "var" (20 GByte, XFS, /var/)
  • Install the base operating system, don't forget to set up SSH  ;-)
  • Add volumes "backup" (2 TByte, XFS, /backup/), "mirror" (1 TByte, XFS, /mirror/)

Bacula

See SAC:Bacula

osgeo1 (primary peer1 host)

current cron jobs for backups

/etc/cron.d/backup.cron

#### Backup cron jobs 
# min hour day month dayofweek user command
#
# 2am: Daily rsync to /home/back
05 2 * * * root /root/scripts/daily.sh
# Every 3 hours 
# /etc/mysql-zrm
#33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
# /var/lib/mysql-zrm
#34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
# call backup_trac_svn.sh to backup 
# subversion /var/www/svn/repos
# trac /var/www/trac
45 */3 * * * root /root/scripts/backup_svn_incremental.sh
#45 */3 * * * root /root/scripts/trac_backup.pl
#45 */3 * * * root /root/scripts/backup_trac_svn.sh
# Thisdoes some hourly drupal/mysql optimization
39 * * * * root /root/scripts/drupal_cron.sh

backup script are in /root/scripts

osgeo3 & osgeo4

See: Infrastructure Transition Plan 2010

download.osgeo.org

As noted in Download Server the contents of download.osgeo.org are rsync'ed to osgeo2.

ProjectsVM

Bacula - we need details.

??? How does it work?

svn.osgeo.org

As noted in Subversion the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.