Difference between revisions of "SAC:Backups"

From OSGeo
Jump to: navigation, search
(Backups)
(OS installation: formatting for readability)
 
(20 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 
= Backups =  
 
= Backups =  
  
== backup.osgeo.org ==
+
== backup.osgeo.org (aka osgeo5, mirror) ==
  
This the OSUOSL VM used for backups starting in 2010/04. It is used for rsync backups of various things (download.osgeo.org, osgeo1, ...), and backula backups of OSU OSL VMs. 
+
This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".
* Admins: Frank Warmerdam (rsync), Martin Spott (backula)
+
* Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
 
* access is via SAC shell group.
 
* access is via SAC shell group.
 +
* download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
 +
* /backup/bacula/ contains the Bacula storage device
 +
* /backup/rsync/ is for rsync backups
 +
* /backup/longterm/ is for long term backups of static material (ie conference web sites)
 +
* /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
 +
* /etc/bacula contains Bacula configuration, is under local git repo
  
System is still being setup as of 2010/04.
+
=== Hardware ===
  
== osgeo1 (primary peer1 host) ==
+
<pre>
 +
CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
 +
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
 +
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
 +
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
 +
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
 +
PCIe 3.0 x16: No Item Selected
 +
PCIe 2.0 x4 (x8): No Item Selected
 +
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
 +
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Optical Drive: Blanking Panel, No Optical Drive
 +
Front Input: Blanking Panel - No Front Inputs
 +
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
 +
Rail Kit: 1U Sliding Rail Kit
 +
OS: No Item Selected
 +
Warranty: Standard 3-Year Warranty
 +
Notes:
 +
New quote with redundant power supply. Can I get a 2.5 inch spinning
 +
disk instead of SSD in the Front Input slot (500 GB Seagate
 +
Constellation 6 GB/s 2.5")?
 +
</pre>
  
    '''TODO''': add in which time zone cron is running - Atlanta [http://www.timeanddate.com/worldclock/city.html?n=25 EST]?
+
=== OS installation ===
     
+
      WISH: please don't run heavy backups during European day, but preferably
+
      during Pacific Ocean day  ([http://www.timeanddate.com/worldclock/meetingtime.html?month=12&day=30&year=2007&p1=770&p2=248&p3=-1&p4=-1 optimize backup time])
+
  
 +
* Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
 +
* Create a second partition on every disk - for use by MD - allocating all the remaining disk space
 +
* MD RAID5 over all the 'second' partitions for use by LVM - '''8 TB''' usable space
 +
* LVM volume group on top of the MD physical volume
 +
* Volumes
 +
** "boot" (256 MByte, Ext2, /boot/),
 +
** "root" (20 GByte, XFS, /root/),
 +
** "swap" (8 GByte),
 +
** "var" (20 GByte, XFS, /var/)
 +
* Install the base operating system, don't forget to set up SSH  ;-)
 +
* Volumes
 +
** "backup" (2 TByte, XFS, /backup/),
 +
** "mirror" (1 TByte, XFS, /mirror/)
 +
 +
=== Bacula ===
 +
 +
See [[SAC:Bacula]]
 +
 +
== osgeo1 (primary peer1 host) ==
  
 
==== current cron jobs for backups ====
 
==== current cron jobs for backups ====
Line 41: Line 86:
 
backup script are in /root/scripts
 
backup script are in /root/scripts
  
backup_svn_incremental.sh runs every 3 hours and only dumps what has changed in the repository in the last three hours to /home/back/svn_backup/project/incremental
+
== osgeo3 & osgeo4 ==
  
backup-svn_full.sh runs once a day at 1 am and does a 'svnadmin hotcopy' of the repository in /home/back/svn_backup/project/full.  14 copies of the repository are kept. 
+
See: [[Infrastructure Transition Plan 2010]]
  
When adding a new svn repository, make sure to edit both the _full and _incremental scripts to add backups for your projects.  You will also have to create the /home/back/svn_backup/project/{full,incremental} directories or the scripts will not work. 
+
== download.osgeo.org ==
'''rsync_back''' rsyncs www.osgeo.org/home/back with test.osgeo.net/home/back (the second PEER1 server)
+
  
=== Daily PEER1 Backup ===
+
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to osgeo2.
PEER1 hosting service is doing daily backup of identified parts of our system using their [http://en.wikipedia.org/wiki/Storage_area_network#SAN_infrastructure SANS] infrastructure.
+
* This is a service we pay for on both osgeo1 and osgeo2.  Do we need it on both?
+
* At present it is backing up the '''portions identified below''' <s>complete file systems</s> on both servers, so we maintain a reasonable sized backup without incurring storage costs.  Update the list by sending them the list through their support ticket system.
+
  
The backup daemon can be disabled with:
+
== [[ProjectsVM]] ==
/sbin/service dsmcad stop
+
/sbin/service dsmc stop
+
  
and restarted with:
+
Bacula - we need details.
/sbin/service dsmc start
+
/sbin/service dsmcad start
+
  
added note (december 2009): the peer1 admin staff seem determined to restart this service so I have renamed the executables:
+
??? How does it work?
mv /usr/bin/dsmc /usr/bin/dsmc_hidden
+
mv /usr/bin/dsmcad /usr/bin/dsmcad_hidden
+
  
==== Backup Folders / Files ====
+
== svn.osgeo.org ==
Please list locations to be backed up here, preferably the same on both servers.  Unless we feel we can drop the service on one of them.  Storage stats are from early 2008 and out of date by now.
+
# /etc - 10MB
+
# /var/www/trac - 550MB
+
# /root - 3MB
+
# /home/back/mailman - 1370MB
+
# /var/lib/mysql-zrm/daily - 401MB
+
# /usr/local/awstats/data - 52MB
+
# /var/www/- 6GB (SVN 7GB excluded, below)
+
'''running total: < 9GB'''
+
  
===== Exclude =====
+
As noted in [[Subversion]] the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.
# /var/www/svn - 7GB
+
 
+
=== Mysql ===
+
MySQL is being backed up using [http://mysqlbackup.zmanda.com MySQL-zrm]
+
* Daily recovery files saved at: /var/lib/mysql-zrm/daily - other files needed?
+
 
+
=== LDAP ===
+
* Anything needed for this?  i.e. /var/lib/ldap ?
+
 
+
== osgeo2 (test.osgeo.net) ==
+
 
+
* '''Daily PEER1 Backup''' above also applies
+
 
+
== download.osgeo.org ==
+
 
+
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to osgeo2.
+
 
+
== xblade14-2 (buildbot | upload | GDAL | GRASS | remotesensing.org | QGIS | GeoTools | Gallery | FOSS4G2007 ) ==
+
 
+
??? How does it work?
+
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 15:06, 15 April 2019

Backups

backup.osgeo.org (aka osgeo5, mirror)

This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".

  • Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
  • access is via SAC shell group.
  • download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
  • /backup/bacula/ contains the Bacula storage device
  • /backup/rsync/ is for rsync backups
  • /backup/longterm/ is for long term backups of static material (ie conference web sites)
  • /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
  • /etc/bacula contains Bacula configuration, is under local git repo

Hardware

CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
PCIe 3.0 x16: No Item Selected
PCIe 2.0 x4 (x8): No Item Selected
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Optical Drive: Blanking Panel, No Optical Drive
Front Input: Blanking Panel - No Front Inputs
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
Rail Kit: 1U Sliding Rail Kit
OS: No Item Selected
Warranty: Standard 3-Year Warranty
Notes:
New quote with redundant power supply. Can I get a 2.5 inch spinning
disk instead of SSD in the Front Input slot (500 GB Seagate
Constellation 6 GB/s 2.5")?

OS installation

  • Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
  • Create a second partition on every disk - for use by MD - allocating all the remaining disk space
  • MD RAID5 over all the 'second' partitions for use by LVM - 8 TB usable space
  • LVM volume group on top of the MD physical volume
  • Volumes
    • "boot" (256 MByte, Ext2, /boot/),
    • "root" (20 GByte, XFS, /root/),
    • "swap" (8 GByte),
    • "var" (20 GByte, XFS, /var/)
  • Install the base operating system, don't forget to set up SSH  ;-)
  • Volumes
    • "backup" (2 TByte, XFS, /backup/),
    • "mirror" (1 TByte, XFS, /mirror/)

Bacula

See SAC:Bacula

osgeo1 (primary peer1 host)

current cron jobs for backups

/etc/cron.d/backup.cron

#### Backup cron jobs 
# min hour day month dayofweek user command
#
# 2am: Daily rsync to /home/back
05 2 * * * root /root/scripts/daily.sh
# Every 3 hours 
# /etc/mysql-zrm
#33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
# /var/lib/mysql-zrm
#34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
# call backup_trac_svn.sh to backup 
# subversion /var/www/svn/repos
# trac /var/www/trac
45 */3 * * * root /root/scripts/backup_svn_incremental.sh
#45 */3 * * * root /root/scripts/trac_backup.pl
#45 */3 * * * root /root/scripts/backup_trac_svn.sh
# Thisdoes some hourly drupal/mysql optimization
39 * * * * root /root/scripts/drupal_cron.sh

backup script are in /root/scripts

osgeo3 & osgeo4

See: Infrastructure Transition Plan 2010

download.osgeo.org

As noted in Download Server the contents of download.osgeo.org are rsync'ed to osgeo2.

ProjectsVM

Bacula - we need details.

??? How does it work?

svn.osgeo.org

As noted in Subversion the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.