Difference between revisions of "Download Server"

From OSGeo
Jump to navigation Jump to search
(details about webdav setup)
 
(16 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
There is a need for a "download server" for OSGeo that provides for bulk downloads of software. 
+
The OSGeo Download Server runs in a container of [[osgeo7]] host, see [[Download Container]]
  
−
= Configuration =
 
  
−
The OSU OSL "Download" VM is being used (140.211.15.132). 
+
Houses the following sites:
  
−
* It is known as "download.osgeo.org", and is configured to respond to that.
+
  - https://upload.osgeo.org
−
* It is also known as "ftp.remotesensing.org", for historical reasons.
+
  - https://bottle.download.osgeo.org
−
* The upload.osgeo.org DNS name also points to it.
 
−
* The downloadable tree is found in /osgeo/download on the server.
 
−
* The virtual host declaration is found in /etc/apache2/sites-available/download2.osgeo.org.conf.
 
−
* Logs are in /var/log/apache2/download_access_log and /var/log/apache2/download_error_log.
 
  
−
= Access Log Viewer (awstats) =
+
The https://download.osgeo.org site goes thru [[osgeo8]] and [[osgeo9]], but data is pulled from this container
  
−
'''awstats has not been reenabled yet on the new VM'''
+
* It is also the official hop container for osgeo7.  Accessible via ssh hop.osgeo7.osgeo.org
 +
*  <del>It has webdav with single local htaccess account geotools for geotools bot use.</del>
  
−
To maintain/view the Apache access logs of download server, the debian [http://awstats.sf.net/ awstats] package has been installed on the server.
+
In order to be able to log in or sftp
−
with site configuration in /etc/awstats/awstats.download.osgeo.org.conf.  It's updated every 4 hours by /etc/cron.d/awstats. See them now at [http://download.osgeo.org/stats/]
 
  
−
= Management =
+
* You must be a member of the OSGeo shell group which can be granted from another person in shell group - [https://id.osgeo.org/ldap/shell Shell]
 +
* Once you are in shell group, go to https://id.osgeo.org/ldap/edit and put in your public key.
  
−
It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files.  Scp, or sftp can be used to bring files onto the server.  Someone from each project will need their LDAP OSGeo Userid [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py shell enabled], for ssh/scp access.
+
You should then be able to log into download (and all other hop servers on the other hosts) with your private key.
  
−
Any administrator can create new project directories, and chown them to a project representative.
+
You can put in a [https://trac.osgeo.org/osgeo/newticket ticket] to request such access.
  
−
Contact Frank Warmerdam for overall configuration questions or about problems.
+
= Configuration =
  
−
= End Users =
+
The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to [[SAC_Service_Status#Download|SAC_Service_Status Download Container]]
  
−
End users should be referred to downloads similarly to:
+
* download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load.
 +
* The upload.osgeo.org DNS name only points to the osgeo7 download container
 +
* The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo)
 +
* The virtual host declaration is found in  /etc/nginx/sites-available
 +
* Logs are in /var/log/nginx
  
−
  http://download.osgeo.org/gdal/gdal-1.3.2.tar.gz
+
* bottle.download.osgeo.org (Mac Bottles) is also hosted on this server and in folder /osgeo/bottle mirrored at https://ftp.osuosl.org/pub/osgeo/bottle
  
−
Directory indexing is left on so requesting a directory will give a file list.
+
= Management =
  
−
= FTP Service =
+
It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files.  Scp, or sftp can be used to bring files onto the server.  Someone from each project will need to add their SSH Public Key(s) into the [https://id.osgeo.org/ldap/edit LDAP OSGeo Userid], for ssh/scp access. 
 +
Any administrator can create new project directories, and chown them to a project representative.
  
−
There is FTP service configured on the '''download.osgeo.org''' blade. The FTP access is served by [http://vsftpd.beasts.org/ vsftpd] - secure and fastest FTP server for UNIX-like systems.
+
= End Users =
  
−
The vsftpd daemon configuration is the default one provided with Debian and is suitable for anonymous ftp only.
+
End users should be referred to downloads similarly to:
−
 
 
−
* root directory: /osgeo/download
 
−
* '''no''' access for local users
 
−
* '''no''' write, mkdir or upload privileges
 
−
 
 
−
== Controlling FTP Service ==
 
−
 
 
−
In order to control the ''vsftpd'' server, you need to:
 
−
* belong to sudoers
 
−
* login in to the download.osgeo.org host using SSH client.
 
−
 
 
−
Here you can see how to issue two basic commands:
 
−
 
 
−
* Start
 
  
−
  $ sudo /etc/init.d/vsftpd start
+
  https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz
  
−
* Stop
+
Directory indexing is left on so requesting a directory will give a file list.
−
 
 
−
$ sudo /etc/init.d/vsftpd stop
 
−
 
 
−
* Restart
 
−
 
 
−
$ sudo /etc/init.d/vsftpd restart
 
  
 
= Rsync =  
 
= Rsync =  
−
 
−
'''Rsync support is not yet re-enabled'''
 
  
 
The /osgeo/download area is exported for anonymous rsync read access as a module named "download".  The following can be used to pull a local copy of the gdal data tree for instance:
 
The /osgeo/download area is exported for anonymous rsync read access as a module named "download".  The following can be used to pull a local copy of the gdal data tree for instance:
  
−
  rsync -av download.osgeo.org::download/gdal/data .
+
  rsync -av upload.osgeo.org::download/gdal/data .
  
 
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.
 
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.
Line 77: Line 56:
 
= Webdav =
 
= Webdav =
  
−
Some projects using Maven as their build tool require webdav in order to work. Here's what's required to get that setup:
+
Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup  
−
 
+
for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools
−
*Enable relevant apache modules
 
−
sudo a2enmod dav
 
−
sudo a2enmod dav_fs
 
−
sudo a2enmod ldap
 
−
sudo a2enmod authnz_ldap
 
−
 
 
−
*Add configuration for project specific location and LDAP group filtering
 
−
*Add configuration for apache ldap auth
 
−
In the vhost config (examples from svn server)
 
−
Include includes.d/webdav/<project>.conf
 
−
In the project.conf (example from tracsvn server
 
−
Include includes.d/ldap_auth.inc
 
  
 
= Backup =  
 
= Backup =  
  
−
'''This is not yet re-enabled.'''
+
Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab.
−
 
+
/home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed
−
Nightly backups of the /osgeo/download tree are managed using rsync to the /osgeo/backup/rsync/download.osgeo.org  
+
(NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).
−
/home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org if needed.
 
  
−
There is also an rsync to iweb.gdal.org (Frank's personal server in Montreal) in case of extreme emergency, and this is found at iweb.gdal.org/download.
+
There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup.
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]
 +
[[Category:Services]]

Latest revision as of 01:06, 21 September 2025

The OSGeo Download Server runs in a container of osgeo7 host, see Download Container


Houses the following sites: 

 - https://upload.osgeo.org
 - https://bottle.download.osgeo.org

The https://download.osgeo.org site goes thru osgeo8 and osgeo9, but data is pulled from this container

  • It is also the official hop container for osgeo7. Accessible via ssh hop.osgeo7.osgeo.org
  • It has webdav with single local htaccess account geotools for geotools bot use.

In order to be able to log in or sftp

  • You must be a member of the OSGeo shell group which can be granted from another person in shell group - Shell
  • Once you are in shell group, go to https://id.osgeo.org/ldap/edit and put in your public key.

You should then be able to log into download (and all other hop servers on the other hosts) with your private key.

You can put in a ticket to request such access.

Configuration

The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to SAC_Service_Status Download Container

  • download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load.
  • The upload.osgeo.org DNS name only points to the osgeo7 download container
  • The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo)
  • The virtual host declaration is found in /etc/nginx/sites-available
  • Logs are in /var/log/nginx

Management

It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files. Scp, or sftp can be used to bring files onto the server. Someone from each project will need to add their SSH Public Key(s) into the LDAP OSGeo Userid, for ssh/scp access. Any administrator can create new project directories, and chown them to a project representative.

End Users

End users should be referred to downloads similarly to: 

https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz

Directory indexing is left on so requesting a directory will give a file list.

Rsync

The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance: 

rsync -av upload.osgeo.org::download/gdal/data .

This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.

Webdav

Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools

Backup

Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab. /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed (NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).

There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup.

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Download_Server&oldid=134279"