Difference between revisions of "SAC:Backups"

From OSGeo
Jump to: navigation, search
m (current cron jobs for backups)
(OS installation: formatting for readability)
 
(27 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
= Backups =  
 
= Backups =  
  
== osgeo1 (primary peer1 host) ==
+
== backup.osgeo.org (aka osgeo5, mirror) ==
  
    '''TODO''': add in which time zone cron is running - Atlanta [http://www.timeanddate.com/worldclock/city.html?n=25 EST]?
+
This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".
     
+
* Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
      WISH: please don't run heavy backups during European day, but preferably
+
* access is via SAC shell group.
      during Pacific Ocean day  ([http://www.timeanddate.com/worldclock/meetingtime.html?month=12&day=30&year=2007&p1=770&p2=248&p3=-1&p4=-1 optimize backup time])
+
* download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
 +
* /backup/bacula/ contains the Bacula storage device
 +
* /backup/rsync/ is for rsync backups
 +
* /backup/longterm/ is for long term backups of static material (ie conference web sites)
 +
* /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
 +
* /etc/bacula contains Bacula configuration, is under local git repo
  
 +
=== Hardware ===
 +
 +
<pre>
 +
CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
 +
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
 +
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
 +
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
 +
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
 +
PCIe 3.0 x16: No Item Selected
 +
PCIe 2.0 x4 (x8): No Item Selected
 +
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
 +
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Optical Drive: Blanking Panel, No Optical Drive
 +
Front Input: Blanking Panel - No Front Inputs
 +
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
 +
Rail Kit: 1U Sliding Rail Kit
 +
OS: No Item Selected
 +
Warranty: Standard 3-Year Warranty
 +
Notes:
 +
New quote with redundant power supply. Can I get a 2.5 inch spinning
 +
disk instead of SSD in the Front Input slot (500 GB Seagate
 +
Constellation 6 GB/s 2.5")?
 +
</pre>
 +
 +
=== OS installation ===
 +
 +
* Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
 +
* Create a second partition on every disk - for use by MD - allocating all the remaining disk space
 +
* MD RAID5 over all the 'second' partitions for use by LVM - '''8 TB''' usable space
 +
* LVM volume group on top of the MD physical volume
 +
* Volumes
 +
** "boot" (256 MByte, Ext2, /boot/),
 +
** "root" (20 GByte, XFS, /root/),
 +
** "swap" (8 GByte),
 +
** "var" (20 GByte, XFS, /var/)
 +
* Install the base operating system, don't forget to set up SSH  ;-)
 +
* Volumes
 +
** "backup" (2 TByte, XFS, /backup/),
 +
** "mirror" (1 TByte, XFS, /mirror/)
 +
 +
=== Bacula ===
 +
 +
See [[SAC:Bacula]]
 +
 +
== osgeo1 (primary peer1 host) ==
  
 
==== current cron jobs for backups ====
 
==== current cron jobs for backups ====
Line 33: Line 86:
 
backup script are in /root/scripts
 
backup script are in /root/scripts
  
backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental
+
== osgeo3 & osgeo4 ==
  
backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full.  14 copies of the repository are kept. 
+
See: [[Infrastructure Transition Plan 2010]]
  
When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects.  You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. 
+
== download.osgeo.org ==
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
+
  
=== Daily PEER1 Backup ===
+
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to osgeo2.
PEER1 hosting service is doing daily <s>tape</s> backup of entire system using their [http://en.wikipedia.org/wiki/Storage_area_network#SAN_infrastructure SANS] infrastructure. 
+
* This is a service we pay for on both osgeo1 and osgeo2.  Do we need it on both?
+
* At present it is backing up the '''complete file systems''' on both servers.  We must provide them some direction as to the folders to include or exclude, so we maintain a reasonable sized backup.  We then send them the list through their support ticket system.
+
  
==== Backup Folders / Files ====
+
== [[ProjectsVM]] ==
Please list locations to be backed up here, preferably the same on both servers.  Unless we feel we can drop the service on one of them.
+
# /etc - 10MB
+
# /var/www/trac - 550MB
+
# /root - 3MB
+
# /home/back/mailman - 1370MB
+
  
=== Mysql ===
+
Bacula - we need details.
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
+
  
== osgeo2 (test.osgeo.net) ==
+
??? How does it work?
  
* '''Daily PEER1 Backup''' above also applies
+
== svn.osgeo.org ==
  
== download.osgeo.org ==
+
As noted in [[Subversion]] the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.
 
+
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to osgeo2.
+
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 15:06, 15 April 2019

Backups

backup.osgeo.org (aka osgeo5, mirror)

This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".

  • Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
  • access is via SAC shell group.
  • download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
  • /backup/bacula/ contains the Bacula storage device
  • /backup/rsync/ is for rsync backups
  • /backup/longterm/ is for long term backups of static material (ie conference web sites)
  • /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
  • /etc/bacula contains Bacula configuration, is under local git repo

Hardware

CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
PCIe 3.0 x16: No Item Selected
PCIe 2.0 x4 (x8): No Item Selected
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Optical Drive: Blanking Panel, No Optical Drive
Front Input: Blanking Panel - No Front Inputs
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
Rail Kit: 1U Sliding Rail Kit
OS: No Item Selected
Warranty: Standard 3-Year Warranty
Notes:
New quote with redundant power supply. Can I get a 2.5 inch spinning
disk instead of SSD in the Front Input slot (500 GB Seagate
Constellation 6 GB/s 2.5")?

OS installation

  • Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
  • Create a second partition on every disk - for use by MD - allocating all the remaining disk space
  • MD RAID5 over all the 'second' partitions for use by LVM - 8 TB usable space
  • LVM volume group on top of the MD physical volume
  • Volumes
    • "boot" (256 MByte, Ext2, /boot/),
    • "root" (20 GByte, XFS, /root/),
    • "swap" (8 GByte),
    • "var" (20 GByte, XFS, /var/)
  • Install the base operating system, don't forget to set up SSH  ;-)
  • Volumes
    • "backup" (2 TByte, XFS, /backup/),
    • "mirror" (1 TByte, XFS, /mirror/)

Bacula

See SAC:Bacula

osgeo1 (primary peer1 host)

current cron jobs for backups

/etc/cron.d/backup.cron

#### Backup cron jobs 
# min hour day month dayofweek user command
#
# 2am: Daily rsync to /home/back
05 2 * * * root /root/scripts/daily.sh
# Every 3 hours 
# /etc/mysql-zrm
#33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
# /var/lib/mysql-zrm
#34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
# call backup_trac_svn.sh to backup 
# subversion /var/www/svn/repos
# trac /var/www/trac
45 */3 * * * root /root/scripts/backup_svn_incremental.sh
#45 */3 * * * root /root/scripts/trac_backup.pl
#45 */3 * * * root /root/scripts/backup_trac_svn.sh
# Thisdoes some hourly drupal/mysql optimization
39 * * * * root /root/scripts/drupal_cron.sh

backup script are in /root/scripts

osgeo3 & osgeo4

See: Infrastructure Transition Plan 2010

download.osgeo.org

As noted in Download Server the contents of download.osgeo.org are rsync'ed to osgeo2.

ProjectsVM

Bacula - we need details.

??? How does it work?

svn.osgeo.org

As noted in Subversion the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.