Difference between revisions of "Download Server"

From OSGeo
Jump to navigation Jump to search
 
(39 intermediate revisions by 9 users not shown)
Line 1: Line 1:
−
There is a need for a "download server" for OSGeo that provides for bulk downloads of software.
+
There is a need for a "download server" for OSGeo that provides for bulk downloads of software.   
−
 
 
−
We are hesitant to handle this on our primary server hosted by Peer1 because of the risk of exceeding our bandwidth allotment and getting charged a lot extra.  For that reason the download server is being hosted on a telascience blade.  Telascience has extensive bandwidth and is not charging OSGeo for it.   
 
  
 
= Configuration =  
 
= Configuration =  
  
−
For now the 198.202.74.219 blade is being used as a download server. 
+
The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to [[SAC_Service_Status#Download|SAC_Service_Status Download Container]]
−
 
 
−
* It is known as "download.osgeo.org", and is configured to respond to that.
 
−
* It is also known as "ftp.remotesensing.org", for historical reasons.
 
−
* The downloadable tree is found in /osgeo/download on the server.
 
−
* The virtual host declaration is found in /etc/httpd/conf.d/sites/download.osgeo.org.conf.
 
−
* Logs are in /etc/httpd/logs/download_access_log and /etc/httpd/logs/download_error_log.
 
−
 
 
−
= Access Log Viewer =
 
  
−
To maintain/view the Apache access logs of download server, [http://awstats.sf.net/ awstats] has been installed on the server.  
+
* download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load.
−
Logs can be seen by visiting [http://download.osgeo.org/logs/ http://download.osgeo.org/logs/] The details of awstats installation are
+
* The upload.osgeo.org DNS name only points to the osgeo7 download container
 +
* The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo)
 +
* The virtual host declaration is found in  /etc/nginx/sites-available
 +
* Logs are in /var/log/nginx
  
−
* awstats version 6.6-1 installed using rpm
+
* bottle.download.osgeo.org (Mac Bottles) is also hosted on this server and in folder /osgeo/bottle mirrored at  https://ftp.osuosl.org/pub/osgeo/bottle
−
* application root dir /usr/local/awstats/
 
−
* location of cgi executable /usr/local/awstats/wwwroot/cgi-bin/awstats.pl
 
−
* site configuration file /etc/awstats/awstats.www.download.osgeo.org.conf (create such file for any other host to be added in future)
 
−
* Data dir (dir where stats data is stored) /osgeo/download/logs
 
−
* Updation is done daily 1am (server time) using a cronjob by executing /osgeo/scripts/update_logs.sh (to add more sites in future, add entry to this script)
 
  
 
= Management =  
 
= Management =  
  
−
It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files.  Scp, or sftp can be used to bring files onto the server.  Someone from each project will need a shell account on the server.   
+
It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files.  Scp, or sftp can be used to bring files onto the server.  Someone from each project will need their LDAP OSGeo Userid [https://www.osgeo.org/cgi-bin/auth/ldap_shell.py shell enabled], for ssh/scp access.   
  
 
Any administrator can create new project directories, and chown them to a project representative.
 
Any administrator can create new project directories, and chown them to a project representative.
−
 
−
Contact Frank Warmerdam for overall configuration questions or about problems.
 
  
 
= End Users =  
 
= End Users =  
Line 37: Line 23:
 
End users should be referred to downloads similarly to:
 
End users should be referred to downloads similarly to:
  
−
  http://download.osgeo.org/gdal/gdal-1.3.2.tar.gz
+
  https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz
  
−
Directory indexing is left on so requesting a directory will give a file list.  
+
Directory indexing is left on so requesting a directory will give a file list.
  
−
= FTP Service =
+
= Rsync =  
  
−
There is FTP service configured on the '''download.osgeo.org''' blade. The FTP access is served by [http://vsftpd.beasts.org/ vsftpd] - secure and fastest FTP server for UNIX-like systems.
+
The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance:
  
−
The vsftpd daemon configuration includes as minimal set of options turned on as possible:
+
rsync -av upload.osgeo.org::download/gdal/data .
  
−
* anonymous '''only''' access
+
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.
−
* root directory: /osgeo/download
 
−
* '''no''' access for local users
 
−
* '''no''' write, mkdir or upload privileges
 
−
* session owned by '''non-privileged''' user (there has been created a dedicated local user)
 
−
* custom banner message: ''Welcome to OSGeo Foundation FTP service.''
 
−
* maximum number of connected clients: '''50'''
 
−
* maximum number of connections per IP address: '''2'''
 
−
* user and group identifiers in are hidden from directories listings, and replaced with '''ftp'''
 
  
−
= Rsync =  
+
= Webdav =
  
−
The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance:
+
Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup
−
 
+
for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools
−
rsync -av download.osgeo.org::download/gdal/data .
 
−
 
 
−
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  /etc/xinetd.d/rsync was also updated to set set disable=yes to disable=no in the hopes the daemon will restart when the blade is restarted (not tested).
 
  
 
= Backup =  
 
= Backup =  
  
−
Nightly backups of the /osgeo/download tree are managed using rsync to the /home/other_backups/download directory on the secondary osgeo server (test.osgeo.net). This is done by a cron job running as user back on osgeo2 (see /home/other_backups for details).
+
Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab.
−
 
+
/home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed
−
= Issues =
+
(NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).
  
−
* There is only 10GB of free disk space on this blade, and in general the telascience blades have rather limited disk space, so these systems offer limited scalability for the amount of software available.
+
There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup.
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]
 +
[[Category:Services]]

Latest revision as of 21:53, 10 April 2022

There is a need for a "download server" for OSGeo that provides for bulk downloads of software.

Configuration

The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to SAC_Service_Status Download Container

  • download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load.
  • The upload.osgeo.org DNS name only points to the osgeo7 download container
  • The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo)
  • The virtual host declaration is found in /etc/nginx/sites-available
  • Logs are in /var/log/nginx

Management

It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files. Scp, or sftp can be used to bring files onto the server. Someone from each project will need their LDAP OSGeo Userid shell enabled, for ssh/scp access.

Any administrator can create new project directories, and chown them to a project representative.

End Users

End users should be referred to downloads similarly to: 

https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz

Directory indexing is left on so requesting a directory will give a file list.

Rsync

The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance: 

rsync -av upload.osgeo.org::download/gdal/data .

This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.

Webdav

Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools

Backup

Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab. /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed (NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).

There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup.

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Download_Server&oldid=128186"