SAC Service Status

Infrastructure of OSGeo System Administration Committee (SAC)

For emergency plans see: SAC:Admin and Troubleshooting

= Servers at OSL = Open Source Labs - 4 physical machines containing x virtual machines.

Logging into Physical Machines
Currently we have osgeo6 and backup.osgeo.osuosl.org that are physical machines under LDAP control.

Note this does not apply to osgeo3, osgeo4, osgeo7 lxd hosts. Refer to those sections for instructions on logging in. If hanging, see OSL for how to open a ticket with OSUOSL's support.

All SAC administrators have LDAP auth to the OSL Machines.

To ssh into a server using your LDAP account, you can do the following replacing your_osgeo_login with your OSGeo login and vmname with the vm name of the server at OSL.

ssh your_osgeo_login@vmname.osgeo.osuosl.org

When prompted for password, use your OSGeo Login password.

SAC:Primary Administrators also have ssh key access in case LDAP is down and that will also apply to the physical machines. Worst case scenario use the information on Open Source Labs to file a ticket (SAC members only). Direct connection to virtual machines is by appending it's vm alias to .osgeo.osuosl.org.

Logging into LXD Hosts
OSGeo3, OSGeo4, and OSGeo7 are all Ubuntu 18.04 servers running LXD. Note that OSGeo3, OSGeo4, and OSGeo7 are all LXD Hosts running (at this moment all LXC containers). These instructions therefore apply to OSGeo3, OSGeo4, and OSGeo7 Hosts. LXD is a management system for LXC containers and QEMU VMS. LXD has a channel that covers its features.

To directly access the host, you go thru port 2222

ssh tech_dev@server_name.osgeo.osuosl.org -p 2222

Only SAC:Primary Administrators have their ssh key installed under that account. In order to access via KVM of these in event servers do not come up on a reboot, you need to go thru OSU OSL VPN -- details in osgeo7-secure container for each.

Logging into LXD Containers and VMs
A convenient block to add to your ~/.ssh/config to easily login to osgeo's LXD hosted containers and vms is the following:

Host osgeo*-* ProxyCommand ssh hop.$(sed -e "s/-.*//" <<< "%h").osgeo.org -W $(sed -e "s/^osgeo[^-*]-//;s/$/.lxd/" <<< "%h"):%p # this is only needed if you you use different private keys for different servers IdentityFile "path/to/your/private/key"

Then you'll be able to access a LXC Container or QEMU VM on machine `osgeo3` with:

ssh osgeo3-matrix

And one on machine `osgeo7` with:

ssh osgeo7-secure

Note you still need to know where each LXC host is hosted... See successiv section sto know what's on which machine.

osgeo 9
Server added April 2021. Intended to provide additional LXD capacity and backup Configuration Details

osgeo 7
Server added June 2018. Intended to replace osgeo3 and old osgeo4 (before reformat). Configuration Details

Container setup of all the osgeo7 servers is located in https://git.osgeo.org/gitea/sac/osgeo7/wiki/_pages

Running LXD 3 snap based container management -- LXD version 3.17 as of 2019-09-15

Accessing osgeo7 containers via ssh
Only the download.osgeo.org is directly exposed ssh via port 22. To access the other containers, you can tunnel thru download.osgeo.org -- You will need an ssh key on download already to do so. If you do not already have a key installed, put in a SAC Ticket Request

A convenient block to add to your own `.ssh/config` file follows:

Host osgeo7-* ProxyCommand ssh your_osgeo_id@hop.osgeo7.osgeo.org -W $(sed -e "s/^osgeo7-//;s/$/.lxd/" <<< "%h"):%p IdentityFile "path/to/your/private/key"

With the above in place, you can connect to any container using:

ssh your_id@osgeo7-

Troubleshooting: In case of "Permission denied (publickey)." after an update to a modern openSSH version, it might well be that your ssh key (RSH key) is disabled in your client in favour of more modern cyphers.

Ugly workaround: add one line `PubkeyAcceptedKeyTypes ...` in `.ssh/config`, to re-enable RSA keys for now (consider to generate a new key):

vim .ssh/config ... Host * ...    PubkeyAcceptedKeyTypes +ssh-rsa

... but better read e.g. here!

Services currently on:

Download
Container Name: download

Houses: https://download.osgeo.org, https://bottle.download.osgeo.org download sites Moved 5/5/2019 from osgeo3 https://download.osgeo.org Setup detailed in OSGeo7 Download container

This is new home of download.osgeo.org, it's a debian 9 container (now locked down to only allow ssh key access / ldap auth) for ssh. It is running nginx instead of apache that the old ran. It has webdav with single local htaccess account geotools for geotools bot use.

In order to be able to log in or sftp


 * You must be a member of the OSGeo shell group which can be granted from another person in shell group - Shell
 * Once you are in shell group, go to https://id.osgeo.org/ldap/edit and put in your public key.

You should then be able to log into download with your private key.

You can put in a ticket to request such access.

WWW
Container Name: wordpress

Houses: https://www.osgeo.org, https://2018.foss4g.org, https://2020.europe.foss4g.org websites

Moved 2019-09-03 from web18a.osgeo.osuosl.org to osgeo7 wordpress container

Note this is a replica of web18a VM server provided by OSUOSL

Setup details of wordpress container

Secure (LDAP ) ldap.osgeo.org
Container Name: secure secure -- ldap.osgeo.org SAC:LDAP used for ldap service (a rebuild of old secure.osgeo.osuosl.org) now on Debian 9

TracSVN (trac, svn, git)
This used to be housed on osgeo3 in VM TracSVN VM, and was moved 2019-10-12 to osgeo7 as tracsvn container.
 * See TracSVN for full details, and some notes on services running here

Container Name: tracsvn

See TracSVN for info about what it contains (spoiler: trac, svn, gitea)

old-wiki (stopped)
This used to be housed on osgeo3, and was moved 2019-09-14 to osgeo7 as old-wiki container. wiki.osgeo.org moved back to osgeo3 on 2020-05-22 and in wiki container. The wiki container is a complete rebuild with files and database restored and upgraded

Container Name: old-wiki

old wiki container -- used for wiki service (it is an lxd2pc created image of wiki.osgeo.osuosl.org VM that was on osgeo3)

See OSGeo Wiki

Nextcloud nextcloud.osgeo.org
Container Name: nextcloud-ubuntu, collabora (for document/view/editing)

Nextcloud - Ubuntu 18.04 LXD/nginx/postgresql 11 container for document sharing similar to dropbox/google drive - nextcloud-ubuntu.lxd - https://nextcloud.osgeo.org Nextcloud Setup

Collabora - Ubuntu 18.04 LXD container for LibreOffice/MS Office online document editor currently used exclusively by nextcloud.osgeo.org. Setup detailed in Nextcloud setup.

GRASS GIS server
Container Name: grass

Current DNS name: grass.osgeo.org (old CMS retired in 2020 to old.grass.osgeo.org, on osgeo6)

Debian 10 Buster

Web: Apache + Hugo (generated through cronjob from https://github.com/OSGeo/grass-website/), see https://github.com/OSGeo/grass-addons/tree/master/tools/cronjobs_osgeo_lxd

ssh: reachable via jumphost.

NGINX proxy
Container Name: nginx

Proxy that routes all http/https traffick for the other containers (can be accessed via osgeo7 host lxc or ubuntu@osgeo7-nginx if your key is installed on ubuntu user). The nginx container holds the letsencrypt https SSL certs for all the containers and handles the renewal of the letsencrypt certs using certbot renew cronjob. Prometheus server to collect all monitoring logs from OSGeo7 (only accessible by OSGeo3), these get queried via monitor.osgeo.org (running on osgeo3) via grafana server.

Dronie Server dronie.osgeo.org
Container Name: dronie-server
 * 1.0 server of drone.io runs in an LXD container details Dronie https://dronie.osgeo.org

Old Projects
Container Name: old-projects

-- this is the old projects.osgeo.osuosl.org migrated from osgeo4 as an lxd container, so more or less the same as it was before, with the exception that all the websites are now proxied thru the nginx container. Websites like on it are community-review.foss4g.org and spatialreference.org

To access you need to go thru download.osgeo.org -> old-projects

Adhoc
Container Name: old-adhoc

old-adhoc -- this is the old adhoc.osgeo.osuosl.org migrated 2019-05-08 from osgeo4 as an lxd container. Used by osgeo-live for there test docs and by grass for earthquake, and mapserver for demo

To access via ssh you should go thru download.osgeo.org -> old-adhoc.lxd It is accessible via https://adhoc.osgeo.org and http://adhoc.osgeo.osuosl.org


 * VM used for projects for various adhoc purposes. Risks to system stability that would be unacceptable on the Projects VM may be ok here.
 * See AdhocVM for full details, and some notes on services running here.
 * eg http://adhoc.osgeo.osuosl.org/livedvd/docs/en/quickstart/

Old Web
Container Name: old-web --- the old web.osgeo.osuosl.org (was on osgeo3) currently housing https://id.osgeo.org/ for LDAP management.
 * old.www.osgeo.org, osgeo.org
 * id.osgeo.org
 * mapguide.osgeo.org

Old WebExtra
Container Name: old-webextra This is a replica of webextra.osgeo.osuosl.org that was hosted on osgeo3

Started move on November 29th 2019 and completed December 8th, 2019
 * foss4g.org
 * europe.foss4g.org
 * planet.osgeo.org
 * various old foss4g.org years
 * live.osgeo.org
 * journal.osgeo.org (not sure what this is for, should be retired?)

pycsw
Container Name: pycsw


 * https://demo.pycsw.org
 * OGC CSW Reference Implementation and Server demo
 * deployment setup at https://github.com/geopython/demo.pycsw.org
 * running hourly teardown/setup cron via docker-compose
 * migrated from Adhoc VM thanks to SAC (May 2020)

demo.mapserver.org
Container Name: mapserver


 * https://staging.demo.mapserver.org
 * host of many services for the MapServer community: see https://wiki.osgeo.org/wiki/MapServer_at_osgeo7
 * migrated from Adhoc VM thanks to SAC + jmckenna (July 2020)

osgeo6
See Osgeo6 for full details
 * Mail

Backup (osgeo5)

 * Backup now runs on dedicated hardware
 * Provides Rsync backups of download.osgeo.org
 * Provides Bacula backups of various VMs.
 * See SAC:Backups for details.

osgeo4
osgeo4 is a real server managed by OSUOSL - can be access via ssh tech_dev@osgeo4.osgeo.osuosl.org -p 2222 (only people with their access keys installed can log in and doesn't allow password access) - password for tech_dev is in the secure container (on osgeo7) / access folder.

In August 2019 the server had new power supply put in and replacement disks. It was reformatted with Ubuntu 18.04.3 to serve as secondary LXD host to osgeo7 zfsutils-linux was installed so lxd can use zfs for storage.

sshing into osgeo4 containers
Note that all the containers are closed off from direct ssh access except for the hop.osgeo4.osgeo.org. To access the other containers, you need to hop through hop. hop container has port 22 open but requires ssh access so users who’ve been granted rights can hop thru it to other containers using hop.osgeo4.osgeo.org as name.

A convenient block to add to your own .ssh/config file follows where your_id could be your osgeo id or a local account on that container

Host osgeo4-* ProxyCommand ssh your_osgeo_id@hop.osgeo4.osgeo.org -W $(sed -e "s/^osgeo4-//;s/$/.lxd/" <<< "%h"):%p IdentityFile "path/to/your/private/key" User your_id

Then to access say the wordpress-dev container, you'd do the below

ssh osgeo4-wordpress-dev

osgeo4 baremetal features
It's makeup is as follows: lsblk -i NAME          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT sda             8:0    0  1.8T  0 disk `-sda2          8:2    0 46.6G  0 part `-md1         9:1    0 46.5G  0 raid1 |-lvm-root 253:0   0 37.3G  0 lvm   / `-lvm-swap 253:1   0  7.5G  0 lvm   [SWAP] sdb             8:16   0  1.8T  0 disk `-sdb2          8:18   0 46.6G  0 part `-md1         9:1    0 46.5G  0 raid1 |-lvm-root 253:0   0 37.3G  0 lvm   / `-lvm-swap 253:1   0  7.5G  0 lvm   [SWAP] sdc             8:32   0  1.8T  0 disk sdd             8:48   0  1.8T  0 disk sde             8:64   0  1.8T  0 disk sdf             8:80   0  1.8T  0 disk
 * -sda1          8:1    0  953M  0 part
 * `-md0         9:0    0  952M  0 raid1 /boot
 * -sdb1          8:17   0  953M  0 part
 * `-md0         9:0    0  952M  0 raid1 /boot

sdc,sdd,sde,sdf form a zfs osgeo4_lxd partition (sdc,sdd) mirrors sde,sdf for total lxd capacity of 3.62 TB

Nightly backups of osgeo7 containers (secure, nextcloud) will be here and named -backup and be kept in a stopped state.

Services currently on:

hop
Container Name: hop - this is the only container with direct ssh access via ssh hop.osgeo4.osgeo.org. To get to other containers, you need to hop thru this one. Requires ssh key access

osgeo4-nginx
Container Name: osgeo4-nginx ->> all web traffick from other containers on osgeo4 get proxied thru here

old-web-staging
Container Name: old-web-staging - used primarily for experimenting with changes to id.osgeo.org (old-web on osgeo7) like testing out OS and software upgrade etc, changes to LDAP forms and registration, before applying to id.osgeo.org. - https://id.staging.osgeo.org

wordpress-dev
Container Name: wordpress-dev - used primarily for osgeo.org main website development - https://staging.www.osgeo.org, https://dev.www.osgeo.org

wiki-dev
Container Name: wiki-dev - used primarily for experimenting with changes to wiki.osgeo.org like testing out OS and software upgrade etc before appying to wiki.osgeo.org. - https://dev.wiki.osgeo.org

wiki-staging
Container Name: wiki-staging - used primarily for upgrade changes to wiki.osgeo.org like testing out OS and software upgrade etc before applying to wiki.osgeo.org. - https://staging.wiki.osgeo.org. The construction of this container is managed by sac ansible-deployment.

tracsvn-dev
Container Name: tracsvn-dev - This is a 2019-09-05 lxd2pc image of tracsvn.osgeo.osuosl.org (now on osgeo7 as tracsvn) used primarily for experimenting like testing out OS, git and software upgrade etc before appying to production. -- https://dev.git.osgeo.org, https://dev.tracsvn.osgeo.org

dronie-client
Container Name: dronie-client - This is a debian 10 machine, with OSGeo LDAP authentication and a drone-agent docker running. To be used with https://dronie.osgeo.org

gallery
Container Name: gallery - work in progress currently contains lots of pictures of OSGeo events. Investigating various different applications we can use to serve these - currently under consideration are photophrism and nextcloud (both installed on this server). Talk to Brian Hamlin if you want to help out. https://gallery.osgeo.org, https://nextcloud.gallery.osgeo.org

Has the following sites: https://dev.trac.osgeo.org, https://dev.git.osgeo.org/gitea, https://dev.svn.osgeo.org

osgeo3
osgeo3 physical server was rebuilt March 2020 with new disks and power supply, as an Ubuntu 18.04 with LXD. It will be used to run production, but moderately risky things. Refer to SAC:Old-osgeo3 for past history before osgeo3 was rebuilt. osgeo3 is a hosted by OSUOSL - can be accessed via ssh tech_dev@osgeo3.osgeo.osuosl.org -p 2222 (only people with their access keys installed can log in and doesn't allow password access) - password for tech_dev is in the secure container (on osgeo7) / access folder.

sshing into osgeo3 containers
Note that all the containers are closed off from direct ssh access except for the hop.osgeo3.osgeo.org. To access the other containers, you need to hop through hop.osgeo3.osgeo.org. hop container has port 22 open but requires ssh access, you need to be added to shell group and have your ssh keys registered on your profile https://id.osgeo.org/ldap/edit). Users who’ve been granted rights can hop thru it to other containers using hop.osgeo3.osgeo.org as name. Other containers may or may not allow password access. It's up to the those who manage the internal containers.

A convenient block to add to your own .ssh/config file follows Host hop.osgeo3.osgeo.org IdentityFile "path/to/your/private/key"
 * 1) this is only needed if you you use different private keys for different servers

Host osgeo3-* ProxyCommand ssh your_osgeo_id@hop.osgeo3.osgeo.org -W $(sed -e "s/^osgeo3-//;s/$/.lxd/" <<< "%h"):%p IdentityFile "path/to/your/private/key"

Then to access say the nexus container, you'd do the below where your_id could be your osgeo id or a local account on that container

ssh your_id@osgeo3-nexus

Baremetal setup of osgeo3
It's makeup is as follows: lsblk -i NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT loop0            7:0    0 91.4M  1 loop /snap/core/8689 loop1            7:1    0   67M  1 loop /snap/lxd/13901 sda              8:0    0  1.8T  0 disk `-sda2           8:2    0  1.8T  0 part |-osgeo3-root 253:0   0  800G  0 lvm  / `-osgeo3-swap 253:1   0   32G  0 lvm  [SWAP] sdb              8:16   0  1.8T  0 disk `-sdb9           8:25   0    8M  0 part sdc              8:32   0  1.8T  0 disk `-sdc9           8:41   0    8M  0 part sdd              8:48   0  1.8T  0 disk `-sdd9           8:57   0    8M  0 part sde              8:64   0  1.8T  0 disk `-sde9           8:73   0    8M  0 part sr0             11:0    1 1024M  0 rom
 * -sda1           8:1    0  953M  0 part /boot
 * -sdb1           8:17   0  1.8T  0 part
 * -sdc1           8:33   0  1.8T  0 part
 * -sdd1           8:49   0  1.8T  0 part
 * -sde1           8:65   0  1.8T  0 part

sdb,sdc,sdd,sdd form a zfs osgeo3_lxd partition (sdb,sdc) mirrors sdd,sdf for total lxd capacity of 3.62 TB sda (composed of 2 1.8TB disks) is under hardware RAID1 - root lvm partition is 800GB, 1TB unallocated for future use

Services currently on:

hop
Container Name: hop - this is the only container with direct ssh access via ssh hop.osgeo3.osgeo.org. To get to other containers, you need to hop thru this one. Requires ssh key access

osgeo3-nginx
Container Name: nginx proxy all web-traffick via the (using IP: 140.211.15.6 web-osgeo3.osgeo.osuosl.org), the second ip is not in use, but may be used for a secondary nginx, to temporarily host osgeo7/osgeo4 containers in case of major hardware upgrades

nexus (repo.osgeo.org, docker.osgeo.org)
Container Name: nexus - see SAC:Repo this is a debian 10 lxd container running docker 19. It currently has one docker container running within it called nexus -- exposed as repo.osgeo.org on nginx.

dronie-client
Container Name: dronie-client - this is a debian 10 lxd container running docker. Currently has just one running docker osgeo-drone-agent to serve as a client for dronie-server (dronie.osgeo.org running on osgeo7)

matrix
Container Name: matrix - debian10 lxd container with ldap/ssh.

See SAC:MatrixSynapse for full detail.

wiki
Container Name: wiki (wiki.osgeo.org) - debian10 lxd container with ldap/ssh (MediaWiki 1.34, 10.3.22-MariaDB, PHP 7.3)

Note that this is a rebuild of wiki.osgeo.org that used to be on osgeo7 old-wiki container. Move 2020-05-22 Setup steps are at wiki container setup

monitor
Container Name: monitor - debian10 lxd container with ldap/ssh. https://monitor.osgeo.org (houses grafana dashboard (for all servers) and prometheus server for osgeo3 containers and pulls basic container metrics using node exporters pulled via prometheus servers. Requirs ldap to log into the web console.

Configuring servers for monitoring is detailed Git Prometheus Config

limesurvey
Container Name: limesurvey - debian 10, PostgreSQL 13, PHP 7.4 with ldap/ss. https://limesurvey.osgeo.org Setup detailed on limesurvey container

WebExtra

 * Retired December 8th, 2019 -- and moved to osgeo7 as container old-webextra


 * See WebExtraVM for full details (server: http://webextra.osgeo.osuosl.org)
 * hosts http://planet.osgeo.org, http://mum03.mapserver.org, http://live.osgeo.org
 * http://foss4g.org (main portal) and archive of old sites 2006-2014
 * http://conference.osgeo.org - Conference System (also: SAC:Setup_OCS)
 * http://journal.osgeo.org / osgeo.org/ojs - Journal System
 * Redirects for many chapter and other urls handled via /etc/httpd/conf.d/rewrite.conf

= Cloud Hosted Servers and other external under SAC Control =

Future Hosting Plans for Windows / Mac Building
SAC Shared Building Services

Atlantic.net
debbie: debian 10 postgis.net, planet.postgis.net, debbie.postgis.net (jenkins build bot) debbie-docker.host.postgis.net - runs docker and serves as a 1.0 agent for dronie.osgeo.org
 * host.postgis.net -p 2222 is an LXD Ubuntu 18.04 16GB RAM/ 6 vCPU, 350GB data, 250GB block storage
 * Currenlty running two lxd containers:

= QGIS off OSGeo = Services on separated machines rented and managed by the QGIS project at hetzner


 * website including documentation http://www.qgis.org
 * website building, documentation building, debian/ubuntu nightlies, plugins.qgis.org
 * issues.qgis.org: redmine

= Historical servers (not more in use) =

- Telascience Blades (Historical)

web18a.osgeo.osuosl.org
NO LONGER USED - turned off '''2019-09-03 Production services www.osgeo.org, 2018.foss4g.org moved to wordpress container on osgeo7 Staging services (staging.www.osgeo.org, dev.www.osgeo.org move to wordpress-dev container on osgeo4 Grass wordpress is disabled as grass decided to go with another solution, so have grass container on osgeo7''' (Cloud hosted server on OSUOSL hardware (not ours) )
 * Debian 9.3 4GB server, host name: web18a.osgeo.osuosl.org require ssh key to log in.
 * Hosts wordpress sites staging.www.osgeo.org,www.osgeo.org, staging.grass.osgeo.org, foss4g2018.osgeo.org
 * Setup details on Web18a setup

OSGeo funtoo
For lxd experimentation it's an lxd container running other lxd containers and provided by funtoo.org.

OSGeo is paying funtoo via treasurer at osgeo.org.


 * OSGeo Funtoo osgeo.host.funtoo.org

Migrated to osgeo3 2020-11-28
 * funtoo LXDs currently running:
 * LimeSurvey -this may be in future migrated to osgeo7 or osgeo3