Difference between revisions of "SAC Service Status"

From OSGeo
Jump to: navigation, search
(Telascience Blades: mark dead blades as dead.)
m (NGINX proxy)
 
(243 intermediate revisions by 13 users not shown)
Line 1: Line 1:
 
Infrastructure of OSGeo System Administration Committee ([[SAC]])
 
Infrastructure of OSGeo System Administration Committee ([[SAC]])
  
= OSL =
+
For emergency plans see: [[SAC:Admin and Troubleshooting]]
[[OSL | Open Source Labs]] - 2 physical machines containing ''x'' virtual machines.
+
  
All SAC administrators have LDAP auth to the OSL Virtual Machines. Primary Admins also have ssh key access in case LDAP is down and that will also apply to the physical machines. Worst case scenario use the information on [[OSL | Open Source Labs]] to file a ticket (SAC members only). Direct connection to virtual machines is by appending it's vm alias to .osgeo.osuosl.org.
 
  
Also, community.osgeo.org is a test Drupal instance they hosted for us while evaluating OSL services earlier on.  (See Tyler)
+
= Servers at OSL =
== osgeo3 ==
+
[[OSL | Open Source Labs]] - 4 physical machines containing ''x'' virtual machines.
=== wiki.osgeo.org ===
+
* VM alias is Wiki
+
* http://wiki.osgeo.org is now hosted on this virtual machine.
+
* Primarily administered by Martin.
+
* [http://trac.osgeo.org/osgeo/ticket/103 old migration notes] may be helpful.
+
  
=== Secure ===
 
* Migration of LDAP to this virtual machine planned.
 
See [[Infrastructure Transition Plan 2010#Final_Plan]] for full details.
 
=== TracSvn===
 
* Hosts svn.osgeo.org [[Subversion]] service.
 
* Hosts trac.osgeo.org [[Trac Instances]].
 
  
=== Web ===
+
== Logging into VMS ==
* Transition in Progress
+
Note this does not apply to osgeo4 and osgeo7 lxd hosts.  Refer to those sections for instructions on logging in.
See [[Infrastructure Transition Plan 2010#Final_Plan]] for full details.
+
If VM hanging, see [[OSL]] for how to open a ticket with OSUOSL's support.
  
=== Base ===
+
All [[SAC#Members|SAC administrators]] have LDAP auth to the OSL Virtual Machines.  
* Template VM used when creating new VMs.
+
* Not normally running - make a special request to OSL to start it up.
+
  
=== WebExtra ===
+
To ssh into a server using your LDAP account, you can do the following replacing '''your_osgeo_login''' with your OSGeo login and '''vmname''' with the vm name of the server at OSL.
* See [[WebExtraVM]] for full details.
+
* hosts planet.osgeo.org, mum03.mapserver.org, live.osgeo.org,
+
  
== osgeo4 ==
+
  ssh '''your_osgeo_login'''@'''vmname'''.osgeo.osuosl.org
  
=== backup.osgeo.org ===
+
When prompted for password, use your OSGeo Login password.
* Provides rsync backups of download.osgeo.org
+
* Will provide backula backups of various VMs.
+
* VM alias is Backup
+
* See [[SAC:Backups]] for details.
+
  
=== Projects ===
 
* General purpose home for OSGeo projects which do not garner heavy enough a load to require their own VM.
 
* See [[ProjectsVM]] for full details
 
  
=== QGIS ===
+
[[SAC:Primary Administrators]] also have ssh key access in case LDAP is down and that will also apply to the physical machines. Worst case scenario use the information on [[OSL | Open Source Labs]] to file a ticket (SAC members only). Direct connection to virtual machines is by appending it's vm alias to .osgeo.osuosl.org.
* VM used by the QGIS project.   
+
 
* Shell access only for the [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py?group=qgis QGIS shell access group].  
+
* Live Status via [http://webextra.osgeo.osuosl.org/munin/index.html Munin]
* Details on configuration to follow...
+
 
* Running joomla 1.5.18 and mediawiki
+
== osgeo 7 ==
 +
 
 +
Server added June 2018. Intended to replace osgeo3 and old osgeo4 (before reformat).
 +
[[osgeo7|Configuration Details]]
 +
 
 +
[https://git.osgeo.org/gitea/sac/osgeo7/wiki/_pages Container setup of all the osgeo7 servers is located in https://git.osgeo.org/gitea/sac/osgeo7/wiki/_pages]
 +
 
 +
Running LXD 3 snap based container management -- LXD version 3.17 as of 2019-09-15
 +
 
 +
=== Accessing osgeo7 containers via ssh ===
 +
 
 +
Only the download.osgeo.org is directly exposed ssh via port 22To access the other containers, you can tunnel thru
 +
download.osgeo.org -- You will need an ssh key on download already to do so.  If you do not already have a key installed, put in a [https://trac.osgeo.org/osgeo/newticket SAC Ticket Request]
 +
 
 +
A convenient block to add to your own `.ssh/config` file follows:
 +
 
 +
 
 +
Host osgeo7-*
 +
  ProxyCommand ssh your_osgeo_id@download.osgeo.org -W $(sed -e "s/^osgeo7-//;s/$/.lxd/" <<< "%h"):%p
 +
  IdentityFile "path/to/your/private/key"
 +
 
 +
With the above in place, you can connect to any container using:
 +
 
 +
  ssh your_id@osgeo7-<container_name>
 +
 
 +
 
 +
Services currently on:
 +
 
 +
=== Download ===
 +
'''Container Name:''' download
 +
 
 +
Houses: https://download.osgeo.org, https://bottle.download.osgeo.org download sites
 +
'''Moved 5/5/2019 from osgeo3'''
 +
https://download.osgeo.org Setup detailed in [https://git.osgeo.org/gitea/sac/osgeo7/wiki/Download-Container OSGeo7 Download container]
 +
 
 +
This is new home of download.osgeo.org, it's a debian 9 container (now locked down to only allow ssh key access / ldap auth) for ssh.  It is running nginx instead of apache that the old ran. It has webdav with single local htaccess account geotools for geotools bot use.
 +
 
 +
In order to be able to log in or sftp
 +
 
 +
* You must be a member of the OSGeo shell group which can be granted from another person in shell group - [https://id.osgeo.org/ldap/shell Shell]
 +
* Your ssh public key must be installed in your home drive, which can currently only be done by member of SAC.
 +
 
 +
You can put in a [https://trac.osgeo.org/osgeo/newticket ticket] to request such access.
 +
 
 +
=== WWW ===
 +
'''Container Name:''' wordpress
 +
 
 +
Houses: https://www.osgeo.org, https://2018.foss4g.org, https://2020.europe.foss4g.org websites
 +
 
 +
'''Moved 2019-09-03 from web18a.osgeo.osuosl.org to osgeo7 wordpress container'''
 +
 
 +
Note this is a replica of web18a VM server provided by OSUOSL
 +
 
 +
[https://git.osgeo.org/gitea/sac/osgeo7/wiki/wordpress-container Setup details of wordpress container]
 +
 
 +
=== Secure (LDAP ) ldap.osgeo.org ===
 +
'''Container Name:''' secure
 +
[https://git.osgeo.org/gitea/sac/osgeo7/wiki/secure secure] -- ldap.osgeo.org [[SAC:LDAP]] used for ldap service (a rebuild of old secure.osgeo.osuosl.org) now on Debian 9
 +
 
 +
=== TracSVN (trac, svn, git) ===
 +
This used to be housed on osgeo3 in VM TracSVN VM, and was moved 2019-10-12 to osgeo7 as tracsvn container.
 +
* See [[TracsvnVM]] for full details, and some notes on services running here
 +
 
 +
'''Container Name:''' tracsvn
 +
 
 +
This houses the following:
 +
 
 +
https://git.osgeo.org/gitea
 +
https://trac.osgeo.org
 +
https://svn.osgeo.org
 +
 
 +
=== old-wiki wiki.osgeo.org ===
 +
This used to be housed on osgeo3, and was moved 2019-09-14 to osgeo7 as old-wiki container.
 +
 
 +
'''Container Name:''' old-wiki
 +
 
 +
[https://git.osgeo.org/gitea/sac/osgeo7/wiki/old-wiki-container old wiki container] -- used for wiki service (it is an lxd2pc created image of wiki.osgeo.osuosl.org VM that was on osgeo3)
 +
 
 +
See [[OSGeo Wiki]]
 +
 
 +
=== Nextcloud nextcloud.osgeo.org ===
 +
'''Container Name:''' nextcloud-ubuntu,  collabora (for document/view/editing)
 +
 
 +
Nextcloud - Ubuntu 18.04 LXD/nginx/postgresql 11 container for document sharing similar to dropbox/google drive - nextcloud-ubuntu.lxd - https://nextcloud.osgeo.org [https://git.osgeo.org/gitea/sac/osgeo7/wiki/Nextcloud-Ubuntu-Setup Nextcloud Setup]
 +
 
 +
Collabora - Ubuntu 18.04 LXD container for  LibreOffice/MS Office online document editor currently used exclusively by nextcloud.osgeo.org.  Setup detailed in Nextcloud setup.
 +
 
 +
=== NGINX proxy  ===
 +
'''Container Name:''' nginx
 +
 
 +
Proxy that routes all http/https traffik for the other containers (can be accessed via osgeo7 host lxc or ubuntu@osgeo7-nginx if your key is installed on ubuntu user).
 +
The nginx container holds the letsencrypt https SSL certs for all the containers and handles the renewal of the letsencrypt certs using certbot renew cronjob.
 +
 
 +
=== Dronie Server dronie.osgeo.org ===
 +
'''Container Name:''' dronie-server
 +
* 1.0 server of drone.io runs in an LXD container details [[Dronie]] https://dronie.osgeo.org
 +
 
 +
=== Old Projects ===
 +
'''Container Name:''' old-projects
 +
 
 +
-- this is the old projects.osgeo.osuosl.org migrated from osgeo4 as an lxd container, so more or less the same as it was before, with the exception that all the websites are now proxied thru the nginx container.  Websites like on it are community-review.foss4g.org and spatialreference.org
 +
 
 +
To access you need to go thru download.osgeo.org -> old-projects
  
 
=== Adhoc ===
 
=== Adhoc ===
 +
'''Container Name:''' old-adhoc
 +
 +
[[AdhocVM|old-adhoc]] -- this is the old adhoc.osgeo.osuosl.org migrated 2019-05-08 from osgeo4 as an lxd container.  Used by osgeo-live for there test docs and by grass for earthquake, and mapserver for demo
 +
 +
To access via ssh you should go thru download.osgeo.org -> old-adhoc.lxd
 +
It is accessible via https://adhoc.osgeo.org and http://adhoc.osgeo.osuosl.org
  
 
* VM used for projects for various adhoc purposes.  Risks to system stability that would be unacceptable on the Projects VM may be ok here.  
 
* VM used for projects for various adhoc purposes.  Risks to system stability that would be unacceptable on the Projects VM may be ok here.  
 
* See [[AdhocVM]] for full details, and some notes on services running here.
 
* See [[AdhocVM]] for full details, and some notes on services running here.
 +
* eg http://adhoc.osgeo.osuosl.org/livedvd/docs/en/quickstart/
  
= Peer1 =
+
=== Old Web ===
 +
'''Container Name''': old-web --- the old web.osgeo.osuosl.org (was on osgeo3) currently housing https://id.osgeo.org/ for LDAP management.
 +
* old.www.osgeo.org, osgeo.org
  
== www.osgeo.org (osgeo1) ==
+
=== Old WebExtra ===
  
* [[SAC:Primary Administrators]]
+
'''Container Name:''' old-webextra
* Lots of config information in the [[Migration Documentation]].
+
This is a replica of webextra.osgeo.osuosl.org that was hosted on osgeo3
* [[SAC:Backups]] describes backup strategy.
+
* Web pages (Drupal) - Tyler Mitchell, ... ([[OSGeo Portal Site|info]])
+
* postfix - Tyler Mitchell, ...
+
* Linux updates - Tyler Mitchell, ...
+
* LDAP - Howard Butler
+
* SSL certificate - Shawn Barnes (configuration)
+
* DNS (via Pairnic) - Shawn Barnes
+
* IPTABLES - Shawn Barnes
+
* a few osgeo.org services are monitored by HTTP monitoring scripts, notification goes to Frank Warmerdam, Shawn Barnes and Daniel Morissette - operated by Daniel.
+
* Virtual Hosts (on same server)
+
** lists.osgeo.org (mailman) - Tyler Mitchell ([[SAC:Mailing Lists|info]])
+
** fdo.osgeo.org
+
** mapguide.osgeo.org
+
** grass.osgeo.net - Wolf (drupal trial) &rarr; moved to download.osgeo.org 2/2008 (??)
+
** conference.osgeo.org - [[Conference System]] (also: [[SAC:Setup_OCS]]), Tyler
+
*** foss4g.org (main portal), 2009, 2006 hosted here.  <s>'''2010 is hosted on telascience but temporarily resurrected here during telascience outage 26apr10'''</s> (2010 switched back)
+
** journal.osgeo.org / osgeo.org/ojs - [[Journal System]], Tyler
+
** Redirects for many chapter and other urls handled via /etc/httpd/conf.d/rewrite.conf
+
  
'''Emergency plans:'''
+
Started move on November 29th 2019 and completed December 8th, 2019
 +
* foss4g.org, europe.foss4g.org, planet.osgeo.org various old foss4g.org years, live.osgeo.org, journal.osgeo.org (not sure what this is for should be retired?)
  
See: [[SAC:Admin and Troubleshooting]]
+
== osgeo6 ==
  
== test.osgeo.org (osgeo2) ==
+
* Mail
 +
See [[Osgeo6]] for full details
  
This is the other peer1 host, primary used for backup and development testing of services.
 
  
* /home/back: 6/24 hour backups from osgeo1 (by rsync)
+
== Backup (osgeo5) ==
* /home/other_backups: daily backups from download.osgeo.org by rsync.
+
* Backup now runs on dedicated hardware
* Development drupal instance
+
* Provides Rsync backups of download.osgeo.org
* <s>wiki.osgeo.org</s> is no longer hosted here. (moved to osuosl osgeo3)
+
* Provides Bacula backups of various VMs.
* <s>A temporary qgis.org joomla instance lives here, managed by Werner Macho.</s> (moved to osuosl osgeo4)
+
* See [[SAC:Backups]] for details.
* /var/www/moodle install was a test/demo for education committee, Arnulf, Tyler, Charlie S.
+
* http://planet.osgeo.org - [[PlanetOSGeo#Planet_Configuration|Planet_Configuration]] administered by Mateusz (Tyler as backup)
+
* /var/www/wiki_wiktionary - is http://geodictionary.osgeo.org - admin by Martin, owned by Markus Neteler
+
* /var/www/community - is http://community.osgeo.net - Drupal including a CivicCRM module for Tyler/secretary to track contacts
+
* FOSSGIS wiki - Arnulf, Martin, Dietmar
+
* Private board wiki - Tyler, Board
+
* /var/www/livedvd - copy (from SVN)) of live dvd download page, for use with live.osgeo.org
+
  
== osgeo3 & osgeo4 ==
+
== osgeo4 ==
  
See: [[Infrastructure Transition Plan 2010]]
+
osgeo4 is a real server managed by OSUOSL - can be access via ssh tech_dev@osgeo4.osgeo.osuosl.org -p 2222  (only people with their access keys installed can log in and doesn't allow password access) - password for tech_dev is in the secure container (on osgeo7) / access folder.
  
= Telascience Blades =
+
In August 2019 the server had new power supply put in and replacement disks.  It was reformatted with Ubuntu 18.04.3 to serve as secondary LXD host to osgeo7
 +
zfsutils-linux was installed so lxd can use zfs for storage.
  
''This only describes some of the osgeo oriented systems and is not a full description of the telascience reality in any way''
+
=== sshing into osgeo4 containers ===
 +
Note that all the containers are closed off from direct ssh access except for the hop.osgeo4.osgeo.org.  To access the other containers, you need to hop through hop.
 +
hop container has port 22 open but requires ssh access so users who’ve been granted rights can hop thru it to other containers using hop.osgeo4.osgeo.org as name.
  
== Telascience Disks ==
+
A convenient block to add to your own .ssh/config file follows
  
* [[TelaScience]]
+
Host osgeo4-*
 +
  ProxyCommand ssh your_osgeo_id@hop.osgeo4.osgeo.org -W $(sed -e "s/^osgeo4-//;s/$/.lxd/" <<< "%h"):%p
 +
  IdentityFile "path/to/your/private/key"
  
== HyperCube ==
+
Then to access say the wordpress-dev container, you'd do the below where your_id could be your osgeo id or a local account on that container
  
* Full details on [[HyperCube]]
+
ssh your_id@osgeo4-wordpress-dev
* Used primarily for geodata serving.
+
  
== Lantronix SLC8 ==
+
=== osgeo4 baremetal features ===
 +
It's makeup is as follows:
 +
{| class="wikitable"
 +
|-
 +
! Item !! Settings
 +
|-
 +
| Disks || 6 1.8 TB drives
 +
|-
 +
| Memory || 48 GB
 +
|-
 +
| CPUs || 8 Intel(R) Xeon(R) CPU E5540  @ 2.53GHz (8192kb cache)
 +
|}
 +
<pre>lsblk -i
 +
NAME          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
 +
sda              8:0    0  1.8T  0 disk 
 +
|-sda1          8:1    0  953M  0 part 
 +
| `-md0          9:0    0  952M  0 raid1 /boot
 +
`-sda2          8:2    0 46.6G  0 part 
 +
  `-md1          9:1    0 46.5G  0 raid1
 +
|-lvm-root 253:0    0 37.3G  0 lvm  /
 +
`-lvm-swap 253:1    0  7.5G  0 lvm  [SWAP]
 +
sdb              8:16  0  1.8T  0 disk 
 +
|-sdb1          8:17  0  953M  0 part 
 +
| `-md0          9:0    0  952M  0 raid1 /boot
 +
`-sdb2          8:18  0 46.6G  0 part 
 +
  `-md1          9:1    0 46.5G  0 raid1
 +
|-lvm-root 253:0    0 37.3G  0 lvm  /
 +
`-lvm-swap 253:1    0  7.5G  0 lvm  [SWAP]
 +
sdc              8:32  0  1.8T  0 disk 
 +
sdd              8:48  0  1.8T  0 disk 
 +
sde              8:64  0  1.8T  0 disk 
 +
sdf              8:80  0  1.8T  0 disk
 +
</pre>
  
deviceports
+
<pre>
 +
sdc,sdd,sde,sdf  form a zfs osgeo4_lxd partition (sdc,sdd) mirrors sde,sdf for total lxd capacity of 3.62 TB
 +
</pre>
  
1  B1600-SC0  << blade consoles
+
Nightly backups of osgeo7 containers (secure, nextcloud) will be here and named <container>-backup and be kept in a stopped state.
2 B1600-SC1
+
3 LX50
+
4 Port-4
+
5 SE3510
+
6 Port-6
+
7 V20z
+
  
* ssh sdsu-slc.telascience.org  
+
Services currently on:
 +
=== hop ===
 +
'''Container Name:''' hop - this is the only container with direct ssh access via ssh hop.osgeo4.osgeo.org. To get to other containers, you need to hop thru this one. Requires ssh key access
 +
 
 +
=== osgeo4-nginx ===
 +
'''Container Name:''' osgeo4-nginx ->> all web traffick from other containers on osgeo4 get proxied thru here
 +
 
 +
=== wordpress-dev  ===
 +
'''Container Name:''' wordpress-dev - used primarily for osgeo.org main website development - https://staging.www.osgeo.org, https://dev.www.osgeo.org
 +
 
 +
=== old-wiki-dev  ===
 +
'''Container Name:''' old-wiki-dev - used primarily for experimenting with changes to wiki.osgeo.org like testing out OS and softwere upgrade etc before appying to wiki.osgeo.org. - https://dev.wiki.osgeo.org
 +
 
 +
=== tracsvn-dev  ===
 +
'''Container Name:''' tracsvn-dev - This is a 2019-09-05 lxd2pc image of tracsvn.osgeo.osuosl.org (now on osgeo7 as tracsvn) used primarily for experimenting like testing out OS, git and software upgrade etc before appying to production.
 +
 
 +
=== dronie-client  ===
 +
'''Container Name:''' dronie-client - This is a debian 10 machine, with OSGeo LDAP authentication and a drone-agent docker running.  To be used with https://dronie.osgeo.org
 +
 
 +
 
 +
 
 +
Has the following sites: https://dev.trac.osgeo.org, https://dev.git.osgeo.org/gitea, https://dev.svn.osgeo.org
 +
 
 +
== osgeo3 ==
 +
 
 +
osgeo3 is a real server managed by OSUOSL and running a virtualization solution (ganeti+kvm) to provide Virtual Machines which are the ones SAC can access and manage. SAC can _not_ access osgeo3 directly, but has to contact OSUOSL for that.
 +
 
 +
Each of the sections below represent a Virtual Machine and describes which services are running on it, and by which name they are reachable.
 +
 
 +
=== TracSvnGit ===
 +
**Moved to osgeo7 **
 +
<del>See [[TracsvnVM]] for full details, and some notes on services running here</del>
 +
 
 +
=== Web ===
 +
*** TURNED OFF 2019-11-01 and containerized on osgeo7***
 +
<del>* Drupal instances  for the [[OSGeo_Portal_Site|main OSGeo web site]] and projects like MapGuide and FDO are hosted here (??).</del> DEFUNCT- moved to wordpress on osgeo7 wordpress container
 +
* Virtual Hosts (on same server)
 +
** http://fdo.osgeo.org
 +
** http://mapguide.osgeo.org
 +
* [[SAC:LDAP|LDAP]] Python Admin Scripts in /usr/lib/cgi-bin - accessed via https://id.osgeo.org (this current reside on osgeo7 lxd container called old-web)
 +
See [[Infrastructure Transition Plan 2010#Final_Plan]] for full details.
 +
<del>* [[Infrastructure Transition Plan 2010#osgeo3_.26_osgeo4]]</del>
 +
<del>* Lots of config information in the [[Migration Documentation]].</del>
 +
* [[SAC:Backups]] describes backup strategy.
 +
 
 +
=== Base ===
 +
* Template VM used when creating new VMs.
 +
* Not normally running - make a special request to OSL to start it up.
 +
 
 +
=== WebExtra ===
 +
** Retired December 8th, 2019 -- and moved to osgeo7 as container old-webextra
  
[SDSU-SLC]> connect direct deviceport 1
+
* See [[WebExtraVM]] for full details (server: http://webextra.osgeo.osuosl.org)
Connected to port 1. Escape sequence is ESC A << Important Info
+
* hosts http://planet.osgeo.org, http://mum03.mapserver.org, http://live.osgeo.org
username:
+
* http://foss4g.org (main portal) and archive of old sites 2006-2014
password:
+
* http://conference.osgeo.org - [[Conference System]] (also: [[SAC:Setup_OCS]])
B1600-sc>
+
* http://journal.osgeo.org / osgeo.org/ojs - [[Journal System]]
B1600-sc>console S11
+
* Redirects for many chapter and other urls handled via /etc/httpd/conf.d/rewrite.conf
[Connected with input enabled on fru S11]
+
Escape Sequence is '#.' << Important Info
+
  
[root@xblade11]#
+
= Cloud Hosted Servers and other external under SAC Control =
  
 +
== OSGeo funtoo ==
  
== xblade10-2 ==
+
<del>For lxd experimentation it's an lxd container running other lxd containers and provided by funtoo.org.</del>
*(198.202.74.215) FC4
+
* '''STATUS: This system was compromised and is currently shutdown pending reformatting.'''
+
* [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py LDAP Shell] enabled, yum updated Nov 5th/2006.
+
* [[mapbender at mapbender.telascience.org]] (postgres, mysql running)
+
* [[demo.mapserver.org]] : demo map servers for MapServer project
+
* GeoNetwork opensource website http://geonetwork-opensource.org or http://geonetwork-opensource.telascience.org
+
  
The Plone service should start automatically. In case it doesn't it can be started manually running ''/etc/rc.d/init.d/zopectl start''
+
OSGeo is paying funtoo via treasurer at osgeo.org.
  
** https://198.202.74.215/phpMyAdmin/
+
* [https://git.osgeo.org/gitea/sac/osgeo_funtoo OSGeo Funtoo] osgeo.host.funtoo.org
** https://198.202.74.215/phpPgAdmin/
+
  
== xblade11-2 (buildtest) ==
+
=== LimeSurvey ===
* (198.202.74.216) FC4 (DNS: buildtest.osgeo.org)
+
* '''STATUS: This system was compromised and is currently shutdown pending reformatting.'''
+
* [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py LDAP Shell] enabled, normal system setup done. March 28, 2008
+
* This server got a new disk in February because the old one died, so all previous contents are lost.
+
* Use 87GB /osgeo partition for bulk of work on this system, limited space in /home.
+
* Assigned as a build and regression testing server for Buildbot slaves, and related systems like Hudson for GeoNetwork and Geotools.
+
* '''Hudson does not restart automatically!'''
+
* Details on the [[Hudson Configuration]] on this system.
+
* hosts tiles for the mapserver front page demo [[MapServerTiles]]
+
* host foss4g2009 workshop uploads per [http://trac.osgeo.org/osgeo/ticket/407 #407]
+
* host foss4g2010 workshop uploads per [http://trac.osgeo.org/osgeo/ticket/563 #563]
+
  
== xblade12-2 (Kids GIS) ==
+
* funtoo LXDs currently running:
* (198.202.74.217) FC4
+
** [https://limesurvey.osgeo.org LimeSurvey] -this may be in future migrated to osgeo7 or osgeo4
* '''STATUS: This system was compromised and is currently shutdown pending reformatting.'''
+
* new server: new install, no ldap or remotely mounted home.
+
* This machine is allocated to [[Kids GIS Portal]]
+
  
== xblade13-2 (download) ==
+
== Atlantic.net ==
* (198.202.74.218) FC4
+
  
* * [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py LDAP Shell] enabled
+
* host.postgis.net -p 2222 is an LXD Debian 10 8GB RAM
* download.osgeo.org ([[Download Server]]) - lots of disk space (transitioning to here)
+
* Currenlty running two lxd containers:
* disk also exported to .219 (for shared /osgeo/download directory)
+
    debbie: postgis.net, planet.postgis.net, debbie.postgis.net (jenkins build bot)
* GRASS usage: [http://download.osgeo.org/grass/ source code], [http://download.osgeo.org/grass/grass6_progman/ ProgManual] (built on buildbot .219), [http://download.osgeo.org/grass/grass6_manuals/html63_user/ UserManual] (built on buildbot .219), [http://download.osgeo.org/grass/grass63/binary/linux/ Linux binaries] (built on buildbot .219)
+
    debbie-docker.host.postgis.net - runs docker and serves as a 1.0 agent for dronie.osgeo.org
  
== xblade14-2 (buildbot | upload | GeoTools | Gallery | FOSS4G2007) ==
+
= QGIS off OSGeo =
* (198.202.74.219) FC4
+
Services on separated machines rented and managed by the QGIS project at hetzner
* '''STATUS: This system was compromised and is currently shutdown pending reformatting.'''
+
* [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py LDAP Shell] enabled, yum updated July 19th/2006.
+
* buildbot.osgeo.org: [[OSGeo BuildBot Configuration]].
+
* Using for [[Community Mapbuilder Continuum Builds]].
+
* MapServer
+
** script preparing nightly svn snapshots (/osgeo/mapserver - cronjob under frankw)
+
** mapserver.org: /osgeo/mapserver/mapserver-web, including scripts to recreate things using Sphinx (hobu/jmckenna)
+
* <strike>GRASS (http://grass.osgeo.org/)</strike> (''migrated to projects.osgeo.osuosl.org'')
+
** <strike>/osgeo/grass/grass-web is the svn checkout of the grass website (updated once per hour).</strike>
+
** <strike>/osgeo/grass/rsync is the server which also operated as rsync mirror master (under the 'grass' rsync module)</strike>
+
** <strike>Mediawiki (using mysql)</strike>
+
* GeoTools (http://www.geotools.org)
+
** /osgeo/geotools/geotools-web is GeoTools home using plain HTML
+
  
== xblade15-2 ==
+
* website including documentation http://www.qgis.org
* (198.202.74.220) FC4
+
* website building, documentation building, debian/ubuntu nightlies, plugins.qgis.org
* '''STATUS: This system was compromised and is currently shutdown pending reformatting.'''
+
* issues.qgis.org: redmine
* ldap.telascience.org: Fedora Directory Server. LDAP server.
+
* osgeo.telascience.org: Plone
+
* txtmob.telascience.org: SMS Smart Mob system
+
* gpstrack.telascience.org: Plone GPS / APRS / Cell tracking .... wishing ;)
+
* mediawiki.telascience.org (likely unused since wiki.osgeo.org ended up on osgeo2)
+
* ISO mirroring
+
  
== sparcblade7 ==
+
= Historical servers (not more in use) =
  
* (198.202.74.212)  
+
- [[Telascience Blades (Historical)]]
* HJG still working to setup.
+
* HJG plans to use for a telascience ldap server.
+
* [[User:Warmerda|Frank Warmerdam]] / [[User:Mloskot|Mateusz Loskot]] plans to deploy buildbot slaves (bigendian sparc solaris!)
+
  
== sparcblade8 ==
+
== web18a.osgeo.osuosl.org ==
* (198.202.74.213) Solaris 2.9
+
'''2019-09-03 Production services www.osgeo.org, 2018.foss4g.org moved to wordpress container on [[osgeo7]]
* civicspace.telascience.org: experimental community portal
+
Staging services (staging.www.osgeo.org, dev.www.osgeo.org move to wordpress-dev container on [[osgeo4]]
 +
Grass wordpress is disabled as grass decided to go with another solution, so have grass container on osgeo7'''
 +
(Cloud hosted server on OSUOSL hardware (not ours) )
 +
* Debian 9.3 4GB server, host name: web18a.osgeo.osuosl.org require ssh key to log in.
 +
* Hosts wordpress sites staging.www.osgeo.org,www.osgeo.org, staging.grass.osgeo.org, foss4g2018.osgeo.org
 +
* Setup details on [https://git.osgeo.org/gitea/osgeo/www_apache_configs/wiki/Web18a-setup Web18a setup]
  
== hypersphere ==
 
* (hypersphere.calit2.net)
 
* [http://irc.telascience.org/cgi-bin/irc.cgi Web based IRC client]
 
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 15:04, 12 December 2019

Infrastructure of OSGeo System Administration Committee (SAC)

For emergency plans see: SAC:Admin and Troubleshooting


Servers at OSL

Open Source Labs - 4 physical machines containing x virtual machines.


Logging into VMS

Note this does not apply to osgeo4 and osgeo7 lxd hosts. Refer to those sections for instructions on logging in. If VM hanging, see OSL for how to open a ticket with OSUOSL's support.

All SAC administrators have LDAP auth to the OSL Virtual Machines.

To ssh into a server using your LDAP account, you can do the following replacing your_osgeo_login with your OSGeo login and vmname with the vm name of the server at OSL.

 ssh your_osgeo_login@vmname.osgeo.osuosl.org

When prompted for password, use your OSGeo Login password.


SAC:Primary Administrators also have ssh key access in case LDAP is down and that will also apply to the physical machines. Worst case scenario use the information on Open Source Labs to file a ticket (SAC members only). Direct connection to virtual machines is by appending it's vm alias to .osgeo.osuosl.org.

osgeo 7

Server added June 2018. Intended to replace osgeo3 and old osgeo4 (before reformat). Configuration Details

Container setup of all the osgeo7 servers is located in https://git.osgeo.org/gitea/sac/osgeo7/wiki/_pages

Running LXD 3 snap based container management -- LXD version 3.17 as of 2019-09-15

Accessing osgeo7 containers via ssh

Only the download.osgeo.org is directly exposed ssh via port 22. To access the other containers, you can tunnel thru download.osgeo.org -- You will need an ssh key on download already to do so. If you do not already have a key installed, put in a SAC Ticket Request

A convenient block to add to your own `.ssh/config` file follows:


Host osgeo7-*
  ProxyCommand ssh your_osgeo_id@download.osgeo.org -W $(sed -e "s/^osgeo7-//;s/$/.lxd/" <<< "%h"):%p
  IdentityFile "path/to/your/private/key"

With the above in place, you can connect to any container using:

 ssh your_id@osgeo7-<container_name>


Services currently on:

Download

Container Name: download

Houses: https://download.osgeo.org, https://bottle.download.osgeo.org download sites Moved 5/5/2019 from osgeo3 https://download.osgeo.org Setup detailed in OSGeo7 Download container

This is new home of download.osgeo.org, it's a debian 9 container (now locked down to only allow ssh key access / ldap auth) for ssh. It is running nginx instead of apache that the old ran. It has webdav with single local htaccess account geotools for geotools bot use.

In order to be able to log in or sftp

  • You must be a member of the OSGeo shell group which can be granted from another person in shell group - Shell
  • Your ssh public key must be installed in your home drive, which can currently only be done by member of SAC.

You can put in a ticket to request such access.

WWW

Container Name: wordpress

Houses: https://www.osgeo.org, https://2018.foss4g.org, https://2020.europe.foss4g.org websites

Moved 2019-09-03 from web18a.osgeo.osuosl.org to osgeo7 wordpress container

Note this is a replica of web18a VM server provided by OSUOSL

Setup details of wordpress container

Secure (LDAP ) ldap.osgeo.org

Container Name: secure secure -- ldap.osgeo.org SAC:LDAP used for ldap service (a rebuild of old secure.osgeo.osuosl.org) now on Debian 9

TracSVN (trac, svn, git)

This used to be housed on osgeo3 in VM TracSVN VM, and was moved 2019-10-12 to osgeo7 as tracsvn container.

  • See TracsvnVM for full details, and some notes on services running here

Container Name: tracsvn

This houses the following:

https://git.osgeo.org/gitea https://trac.osgeo.org https://svn.osgeo.org

old-wiki wiki.osgeo.org

This used to be housed on osgeo3, and was moved 2019-09-14 to osgeo7 as old-wiki container.

Container Name: old-wiki

old wiki container -- used for wiki service (it is an lxd2pc created image of wiki.osgeo.osuosl.org VM that was on osgeo3)

See OSGeo Wiki

Nextcloud nextcloud.osgeo.org

Container Name: nextcloud-ubuntu, collabora (for document/view/editing)

Nextcloud - Ubuntu 18.04 LXD/nginx/postgresql 11 container for document sharing similar to dropbox/google drive - nextcloud-ubuntu.lxd - https://nextcloud.osgeo.org Nextcloud Setup

Collabora - Ubuntu 18.04 LXD container for LibreOffice/MS Office online document editor currently used exclusively by nextcloud.osgeo.org. Setup detailed in Nextcloud setup.

NGINX proxy

Container Name: nginx

Proxy that routes all http/https traffik for the other containers (can be accessed via osgeo7 host lxc or ubuntu@osgeo7-nginx if your key is installed on ubuntu user). The nginx container holds the letsencrypt https SSL certs for all the containers and handles the renewal of the letsencrypt certs using certbot renew cronjob.

Dronie Server dronie.osgeo.org

Container Name: dronie-server

Old Projects

Container Name: old-projects

-- this is the old projects.osgeo.osuosl.org migrated from osgeo4 as an lxd container, so more or less the same as it was before, with the exception that all the websites are now proxied thru the nginx container. Websites like on it are community-review.foss4g.org and spatialreference.org

To access you need to go thru download.osgeo.org -> old-projects

Adhoc

Container Name: old-adhoc

old-adhoc -- this is the old adhoc.osgeo.osuosl.org migrated 2019-05-08 from osgeo4 as an lxd container. Used by osgeo-live for there test docs and by grass for earthquake, and mapserver for demo

To access via ssh you should go thru download.osgeo.org -> old-adhoc.lxd It is accessible via https://adhoc.osgeo.org and http://adhoc.osgeo.osuosl.org

Old Web

Container Name: old-web --- the old web.osgeo.osuosl.org (was on osgeo3) currently housing https://id.osgeo.org/ for LDAP management.

  • old.www.osgeo.org, osgeo.org

Old WebExtra

Container Name: old-webextra This is a replica of webextra.osgeo.osuosl.org that was hosted on osgeo3

Started move on November 29th 2019 and completed December 8th, 2019

  • foss4g.org, europe.foss4g.org, planet.osgeo.org various old foss4g.org years, live.osgeo.org, journal.osgeo.org (not sure what this is for should be retired?)

osgeo6

  • Mail

See Osgeo6 for full details


Backup (osgeo5)

  • Backup now runs on dedicated hardware
  • Provides Rsync backups of download.osgeo.org
  • Provides Bacula backups of various VMs.
  • See SAC:Backups for details.

osgeo4

osgeo4 is a real server managed by OSUOSL - can be access via ssh tech_dev@osgeo4.osgeo.osuosl.org -p 2222 (only people with their access keys installed can log in and doesn't allow password access) - password for tech_dev is in the secure container (on osgeo7) / access folder.

In August 2019 the server had new power supply put in and replacement disks. It was reformatted with Ubuntu 18.04.3 to serve as secondary LXD host to osgeo7 zfsutils-linux was installed so lxd can use zfs for storage.

sshing into osgeo4 containers

Note that all the containers are closed off from direct ssh access except for the hop.osgeo4.osgeo.org. To access the other containers, you need to hop through hop. hop container has port 22 open but requires ssh access so users who’ve been granted rights can hop thru it to other containers using hop.osgeo4.osgeo.org as name.

A convenient block to add to your own .ssh/config file follows

Host osgeo4-*
  ProxyCommand ssh your_osgeo_id@hop.osgeo4.osgeo.org -W $(sed -e "s/^osgeo4-//;s/$/.lxd/" <<< "%h"):%p
  IdentityFile "path/to/your/private/key"

Then to access say the wordpress-dev container, you'd do the below where your_id could be your osgeo id or a local account on that container

ssh your_id@osgeo4-wordpress-dev

osgeo4 baremetal features

It's makeup is as follows:

Item Settings
Disks 6 1.8 TB drives
Memory 48 GB
CPUs 8 Intel(R) Xeon(R) CPU E5540 @ 2.53GHz (8192kb cache)
lsblk -i
NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda              8:0    0  1.8T  0 disk  
|-sda1           8:1    0  953M  0 part  
| `-md0          9:0    0  952M  0 raid1 /boot
`-sda2           8:2    0 46.6G  0 part  
  `-md1          9:1    0 46.5G  0 raid1 
	|-lvm-root 253:0    0 37.3G  0 lvm   /
	`-lvm-swap 253:1    0  7.5G  0 lvm   [SWAP]
sdb              8:16   0  1.8T  0 disk  
|-sdb1           8:17   0  953M  0 part  
| `-md0          9:0    0  952M  0 raid1 /boot
`-sdb2           8:18   0 46.6G  0 part  
  `-md1          9:1    0 46.5G  0 raid1 
	|-lvm-root 253:0    0 37.3G  0 lvm   /
	`-lvm-swap 253:1    0  7.5G  0 lvm   [SWAP]
sdc              8:32   0  1.8T  0 disk  
sdd              8:48   0  1.8T  0 disk  
sde              8:64   0  1.8T  0 disk  
sdf              8:80   0  1.8T  0 disk 
sdc,sdd,sde,sdf  form a zfs osgeo4_lxd partition (sdc,sdd) mirrors sde,sdf for total lxd capacity of 3.62 TB

Nightly backups of osgeo7 containers (secure, nextcloud) will be here and named <container>-backup and be kept in a stopped state.

Services currently on:

hop

Container Name: hop - this is the only container with direct ssh access via ssh hop.osgeo4.osgeo.org. To get to other containers, you need to hop thru this one. Requires ssh key access

osgeo4-nginx

Container Name: osgeo4-nginx ->> all web traffick from other containers on osgeo4 get proxied thru here

wordpress-dev

Container Name: wordpress-dev - used primarily for osgeo.org main website development - https://staging.www.osgeo.org, https://dev.www.osgeo.org

old-wiki-dev

Container Name: old-wiki-dev - used primarily for experimenting with changes to wiki.osgeo.org like testing out OS and softwere upgrade etc before appying to wiki.osgeo.org. - https://dev.wiki.osgeo.org

tracsvn-dev

Container Name: tracsvn-dev - This is a 2019-09-05 lxd2pc image of tracsvn.osgeo.osuosl.org (now on osgeo7 as tracsvn) used primarily for experimenting like testing out OS, git and software upgrade etc before appying to production.

dronie-client

Container Name: dronie-client - This is a debian 10 machine, with OSGeo LDAP authentication and a drone-agent docker running. To be used with https://dronie.osgeo.org


Has the following sites: https://dev.trac.osgeo.org, https://dev.git.osgeo.org/gitea, https://dev.svn.osgeo.org

osgeo3

osgeo3 is a real server managed by OSUOSL and running a virtualization solution (ganeti+kvm) to provide Virtual Machines which are the ones SAC can access and manage. SAC can _not_ access osgeo3 directly, but has to contact OSUOSL for that.

Each of the sections below represent a Virtual Machine and describes which services are running on it, and by which name they are reachable.

TracSvnGit

    • Moved to osgeo7 **

See TracsvnVM for full details, and some notes on services running here

Web

      • TURNED OFF 2019-11-01 and containerized on osgeo7***

* Drupal instances for the main OSGeo web site and projects like MapGuide and FDO are hosted here (??). DEFUNCT- moved to wordpress on osgeo7 wordpress container

See Infrastructure Transition Plan 2010#Final_Plan for full details. * Infrastructure Transition Plan 2010#osgeo3_.26_osgeo4 * Lots of config information in the Migration Documentation.

Base

  • Template VM used when creating new VMs.
  • Not normally running - make a special request to OSL to start it up.

WebExtra

    • Retired December 8th, 2019 -- and moved to osgeo7 as container old-webextra

Cloud Hosted Servers and other external under SAC Control

OSGeo funtoo

For lxd experimentation it's an lxd container running other lxd containers and provided by funtoo.org.

OSGeo is paying funtoo via treasurer at osgeo.org.

LimeSurvey

  • funtoo LXDs currently running:
    • LimeSurvey -this may be in future migrated to osgeo7 or osgeo4

Atlantic.net

  • host.postgis.net -p 2222 is an LXD Debian 10 8GB RAM
  • Currenlty running two lxd containers:
   debbie: postgis.net, planet.postgis.net, debbie.postgis.net (jenkins build bot)  
   debbie-docker.host.postgis.net - runs docker and serves as a 1.0 agent for dronie.osgeo.org

QGIS off OSGeo

Services on separated machines rented and managed by the QGIS project at hetzner

  • website including documentation http://www.qgis.org
  • website building, documentation building, debian/ubuntu nightlies, plugins.qgis.org
  • issues.qgis.org: redmine

Historical servers (not more in use)

- Telascience Blades (Historical)

web18a.osgeo.osuosl.org

2019-09-03 Production services www.osgeo.org, 2018.foss4g.org moved to wordpress container on osgeo7 Staging services (staging.www.osgeo.org, dev.www.osgeo.org move to wordpress-dev container on osgeo4 Grass wordpress is disabled as grass decided to go with another solution, so have grass container on osgeo7 (Cloud hosted server on OSUOSL hardware (not ours) )

  • Debian 9.3 4GB server, host name: web18a.osgeo.osuosl.org require ssh key to log in.
  • Hosts wordpress sites staging.www.osgeo.org,www.osgeo.org, staging.grass.osgeo.org, foss4g2018.osgeo.org
  • Setup details on Web18a setup