Difference between revisions of "SAC:Backups"

From OSGeo
Jump to navigation Jump to search
(Backup on dedicated hardware)
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
= Backups =  
 
= Backups =  
  
−
== backup.osgeo.org ==
+
== backup.osgeo.org (aka osgeo5, mirror) ==
  
−
This was the OSUOSL VM used for backups starting in 2010/04 and is now upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of OSU OSL VMs and Rsync backups of the public space at "download.osgeo.org".
+
This was the OSUOSL VM used for backups starting in 2010/04 and upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of osgeo6.
−
* Admins: Frank Warmerdam (rsync), Martin Spott (backula)
+
* Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
 
* access is via SAC shell group.
 
* access is via SAC shell group.
 
* download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
 
* download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
Line 11: Line 11:
 
* /backup/longterm/ is for long term backups of static material (ie conference web sites)
 
* /backup/longterm/ is for long term backups of static material (ie conference web sites)
 
* /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
 
* /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
 +
* /etc/bacula contains Bacula configuration, is under local git repo
 +
 +
=== Hardware ===
 +
 +
<pre>
 +
CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
 +
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
 +
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
 +
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
 +
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
 +
PCIe 3.0 x16: No Item Selected
 +
PCIe 2.0 x4 (x8): No Item Selected
 +
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
 +
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
 +
Optical Drive: Blanking Panel, No Optical Drive
 +
Front Input: Blanking Panel - No Front Inputs
 +
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
 +
Rail Kit: 1U Sliding Rail Kit
 +
OS: No Item Selected
 +
Warranty: Standard 3-Year Warranty
 +
Notes:
 +
New quote with redundant power supply. Can I get a 2.5 inch spinning
 +
disk instead of SSD in the Front Input slot (500 GB Seagate
 +
Constellation 6 GB/s 2.5")?
 +
</pre>
  
 
=== OS installation ===
 
=== OS installation ===
Line 16: Line 44:
 
* Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
 
* Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
 
* Create a second partition on every disk - for use by MD - allocating all the remaining disk space
 
* Create a second partition on every disk - for use by MD - allocating all the remaining disk space
−
* Create an MD RAID5 over all the 'second' partitions for use by LVM
+
* MD RAID5 over all the 'second' partitions for use by LVM - '''8 TB''' usable space
−
* Create a LVM volume group on top of the MD physical volume
+
* LVM volume group on top of the MD physical volume
−
* Create volumes "boot" (256 MByte, Ext2, /boot/), "root" (20 GByte, XFS, /root/), "swap" (8 GByte), "var" (20 GByte, XFS, /var/)
+
* Volumes
 +
** "boot" (256 MByte, Ext2, /boot/),  
 +
** "root" (20 GByte, XFS, /root/),  
 +
** "swap" (8 GByte),  
 +
** "var" (20 GByte, XFS, /var/)
 
* Install the base operating system, don't forget to set up SSH  ;-)
 
* Install the base operating system, don't forget to set up SSH  ;-)
−
* Add volumes "backup" (2 TByte, XFS, /backup/), "mirror" (1 TByte, XFS, /mirror/)
+
* Volumes
 +
** "backup" (2 TByte, XFS, /backup/),  
 +
** "mirror" (1 TByte, XFS, /mirror/)
  
 
=== Bacula ===
 
=== Bacula ===
−
Configuration as of June, 2011
 
−
 
−
A Full copy is made every Sat. with incremental backups nightly.
 
−
 
−
This consists of the following directories:
 
−
    File = /boot
 
−
    File = /etc
 
−
    File = /var/www
 
−
    File = /osgeo
 
−
    File = /var/lib/mysql
 
−
    File = /var/lib/postgresql
 
−
    File = /var/lib/dpkg
 
−
    File = /var/lib/aptitude/pkgstates
 
−
From each of the following machines:
 
−
* wiki,secure,web,webextra,backup,qgis,projects,adhoc
 
  
−
trac currently gets trac_backup.zip which is generated on demand
+
See [[SAC:Bacula]]
  
−
==== Administration ====
+
== osgeo3 & osgeo7 & osgeo8 & osgeo9 ==
−
General configuration is managed in /etc/bacula/bacula-dir.conf.
 
  
−
To review the backups and perform a restore you can use the [http://www.bacula.org/manuals/en/console/console/Bacula_Console.html bconsole application]:
+
osgeo3, osgeo7, osgeo8, osgeo9 are production LXD Hosts. osgeo8 and osgeo9 are donated by OSMF.  All servers are are backed up to osgeo4, except for download on osgeo7. These are done nightly or weekly based on frequency of use and size.  The scripts for these are in ansible under osgeo4 deployment and setup to run as a cronjob on osgeo4. download.osgeo.org is replicated to osgeo8 accessible via download8.osgeo.org.
−
  sudo bconsole
 
  
−
Specific configuration of the file/tape storage is in the Storage director /etc/bacula/bacula-sd.conf
+
Using the built in LXD scheduler snapshots of each container are done daily with a retention of 30 days.
−
* Bacula uses the hard drive as a series of Tape drives.
+
This means each container in case of disaster such as virus/exploit/serious misconfiguration can be restored to up to 30 days before.
−
* Each Volume is a predetermined size (~500MB currently), when a volume fills bacula makes the next one.
 
−
* Volumes older than a certain date or when volume increments have reach a configured maximum can be recycled into the available disk pool.
 
  
−
==== Recovery ====
+
In addition the latest snapshots of each container are made on a daily or weekly basis by osgeo4.
−
* If you use bconsole to recover data bacula does the work of finding the files you want and pulling them out of the various volumes.
+
osgeo4 serves as both a dev lxd host (e.g. it has wordpress-dev, wiki-dev, and tracsvn-dev) and a backup lxd host.
−
* Directory for recovered files is set in the bacula-dir.conf under:
 
−
Job {
 
−
  Name = "RestoreFiles"
 
−
  Type = Restore
 
−
  Client = backup.osgeo.osuosl.org-fd
 
−
  FileSet = "Full Set"
 
−
  Storage = File
 
−
  Pool = Default
 
−
  Messages = Standard
 
−
  Where = /tmp/bacula-restores
 
−
}
 
−
* Directions for [http://www.bacula.org/5.0.x-manuals/en/main/main/Restore_Command.html how to restore]
 
−
sudo bconsole
 
−
*restore
 
−
Then follow the prompts, using the above linked guide to help find the files you want.
 
  
−
== osgeo1 (primary peer1 host) ==
+
Smaller containers that change frequently such as secure are snapshot nightly and pushed to osgeo4 using cronjob - /etc/cron.d/osgeo4-lxd-maint on osgeo4. Only one container snapshots of each is kept on osgeo4.
  
−
==== current cron jobs for backups ====
+
Larger containers such as repo (on osgeo3) and download and tracsvn on osgeo7 are snapshotted weekly and pushed to osgeo4.
−
/etc/cron.d/backup.cron
+
All backup containers are suffixed with -backup. So you will see on osgeo4 containers such as download-backup, grass-backup etc.
  
−
#### Backup cron jobs
+
Right now a manual process - backups of key containers are made and stored on backup.osgeo.org in the /backup/osgeo7_containers/image_backups/ folder.
−
# min hour day month dayofweek user command
+
There is currently not enough space there to hold other backups. download is however rsync'ed to http://ftp.osuosl.org/pub/osgeo/download/ .
−
#
 
−
# 2am: Daily rsync to /home/back
 
−
05 2 * * * root /root/scripts/daily.sh
 
−
# Every 3 hours
 
−
# /etc/mysql-zrm
 
−
#33 */3 * * * root /usr/bin/rsync -a --delete /etc/mysql-zrm/ /home/back/etc/mysql-zrm/
 
−
  # /var/lib/mysql-zrm
 
−
#34 */3 * * * root /usr/bin/rsync -a --delete /var/lib/mysql-zrm/ /home/back/mysql-zrm/
 
−
# call backup_trac_svn.sh to backup
 
−
# subversion /var/www/svn/repos
 
−
# trac /var/www/trac
 
−
45 */3 * * * root /root/scripts/backup_svn_incremental.sh
 
−
#45 */3 * * * root /root/scripts/trac_backup.pl
 
−
#45 */3 * * * root /root/scripts/backup_trac_svn.sh
 
−
# Thisdoes some hourly drupal/mysql optimization
 
−
39 * * * * root /root/scripts/drupal_cron.sh
 
−
 
 
−
backup script are in /root/scripts
 
−
 
 
−
== osgeo3 & osgeo4 ==
 
−
 
 
−
See: [[Infrastructure Transition Plan 2010]]
 
  
 
== download.osgeo.org ==
 
== download.osgeo.org ==
  
−
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to osgeo2.
+
As noted in [[Download Server]] the contents of download.osgeo.org are rsync'ed to ftp.osuosl.org/osgeo.
−
 
 
−
== [[ProjectsVM]] ==
 
−
 
 
−
Bacula - we need details.
 
−
 
 
−
??? How does it work?
 
  
 
== svn.osgeo.org ==
 
== svn.osgeo.org ==
Line 114: Line 86:
 
As noted in [[Subversion]] the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.
 
As noted in [[Subversion]] the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.
  
 +
= See also =
 +
 +
* [[SAC Service Status]]
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 07:17, 13 November 2023

Backups

backup.osgeo.org (aka osgeo5, mirror)

This was the OSUOSL VM used for backups starting in 2010/04 and upgraded to dedicated hardware (very) early in 2014. It is used for Bacula backups of osgeo6.

  • Admins: Frank Warmerdam (rsync), Martin Spott (bacula), Sandro Santilli (bacula)
  • access is via SAC shell group.
  • download.osgeo.org rsync mirroring runs nightly and is externally available at download2.osgeo.org.
  • /backup/bacula/ contains the Bacula storage device
  • /backup/rsync/ is for rsync backups
  • /backup/longterm/ is for long term backups of static material (ie conference web sites)
  • /backup/pgdump/ contains PostgreSQL dumps of the Bacula director database
  • /etc/bacula contains Bacula configuration, is under local git repo

Hardware

CPU: Intel Xeon E3-1220v2, 3.1GHz (4-Core, 8MB Cache, 69W) 22nm
RAM: 16GB (2 x 8GB DDR3-1600 ECC Unbuffered 2R DIMMs) Operating at 1600 MT/s Max
NIC: Two Gigabit Ethernet Controllers (Intel 82574L) - Integrated
Management: Integrated IPMI 2.0 & KVM with Dedicated LAN
Controller: 2 Ports 6Gb/s SATA, and 4 Ports 3Gb/s SATA via Intel C216 Chipset
PCIe 3.0 x16: No Item Selected
PCIe 2.0 x4 (x8): No Item Selected
NOTE: Drives will be connected to onboard SATA controller unless otherwise specified
Hot-Swap Drive - 1: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 2: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 3: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Hot-Swap Drive - 4: 3TB Seagate Constellation ES.3 (6Gb/s, 7.2K RPM, 128MB Cache) 3.5-inch SATA
Optical Drive: Blanking Panel, No Optical Drive
Front Input: Blanking Panel - No Front Inputs
Power Supply: Redundant 500W Power Supply with PMBus - 80 PLUS Platinum Certified
Rail Kit: 1U Sliding Rail Kit
OS: No Item Selected
Warranty: Standard 3-Year Warranty
Notes:
New quote with redundant power supply. Can I get a 2.5 inch spinning
disk instead of SSD in the Front Input slot (500 GB Seagate
Constellation 6 GB/s 2.5")?

OS installation

  • Create an empty partition at the beginning of every hard drive, use "Reserved BIOS boot area" on GPT-partitioned disks (typical with 2 GByte disks or bigger) or just "Unused" on DOS-type partition tables. By tradition I reserve 10 MByte on each disk, maybe 1 MByte would be sufficient as well, but these 9 MByte of difference don't cost that much and you're on the safe side ....
  • Create a second partition on every disk - for use by MD - allocating all the remaining disk space
  • MD RAID5 over all the 'second' partitions for use by LVM - 8 TB usable space
  • LVM volume group on top of the MD physical volume
  • Volumes
    • "boot" (256 MByte, Ext2, /boot/),
    • "root" (20 GByte, XFS, /root/),
    • "swap" (8 GByte),
    • "var" (20 GByte, XFS, /var/)
  • Install the base operating system, don't forget to set up SSH ;-)
  • Volumes
    • "backup" (2 TByte, XFS, /backup/),
    • "mirror" (1 TByte, XFS, /mirror/)

Bacula

See SAC:Bacula

osgeo3 & osgeo7 & osgeo8 & osgeo9

osgeo3, osgeo7, osgeo8, osgeo9 are production LXD Hosts. osgeo8 and osgeo9 are donated by OSMF. All servers are are backed up to osgeo4, except for download on osgeo7. These are done nightly or weekly based on frequency of use and size. The scripts for these are in ansible under osgeo4 deployment and setup to run as a cronjob on osgeo4. download.osgeo.org is replicated to osgeo8 accessible via download8.osgeo.org.

Using the built in LXD scheduler snapshots of each container are done daily with a retention of 30 days. This means each container in case of disaster such as virus/exploit/serious misconfiguration can be restored to up to 30 days before.

In addition the latest snapshots of each container are made on a daily or weekly basis by osgeo4. osgeo4 serves as both a dev lxd host (e.g. it has wordpress-dev, wiki-dev, and tracsvn-dev) and a backup lxd host.

Smaller containers that change frequently such as secure are snapshot nightly and pushed to osgeo4 using cronjob - /etc/cron.d/osgeo4-lxd-maint on osgeo4. Only one container snapshots of each is kept on osgeo4.

Larger containers such as repo (on osgeo3) and download and tracsvn on osgeo7 are snapshotted weekly and pushed to osgeo4. All backup containers are suffixed with -backup. So you will see on osgeo4 containers such as download-backup, grass-backup etc.

Right now a manual process - backups of key containers are made and stored on backup.osgeo.org in the /backup/osgeo7_containers/image_backups/ folder. There is currently not enough space there to hold other backups. download is however rsync'ed to http://ftp.osuosl.org/pub/osgeo/download/ .

download.osgeo.org

As noted in Download Server the contents of download.osgeo.org are rsync'ed to ftp.osuosl.org/osgeo.

svn.osgeo.org

As noted in Subversion the subversion repository is mirrored using svn mirroring and does not need to be backed up otherwise.

See also

Retrieved from "http://wiki.osgeo.org/w/index.php?title=SAC:Backups&oldid=131101"