Difference between revisions of "Download Server"

From OSGeo
Jump to: navigation, search
(Access Log Viewer (awstats))
(12 intermediate revisions by 4 users not shown)
Line 1: Line 1:
There is a need for a "download server" for OSGeo that provides for bulk downloads of software.
+
There is a need for a "download server" for OSGeo that provides for bulk downloads of software.   
 
+
We are hesitant to handle this on our primary server hosted by Peer1 because of the risk of exceeding our bandwidth allotment and getting charged a lot extra.  For that reason the download server is being hosted on a telascience blade.  Telascience has extensive bandwidth and is not charging OSGeo for it.   
+
  
 
= Configuration =  
 
= Configuration =  
  
For now the 198.202.74.218 blade is being used as a download server.   
+
The OSU OSL "Download" VM is being used (140.211.15.132).   
  
 
* It is known as "download.osgeo.org", and is configured to respond to that.
 
* It is known as "download.osgeo.org", and is configured to respond to that.
* It is also known as "ftp.remotesensing.org", for historical reasons.
 
 
* The upload.osgeo.org DNS name also points to it.  
 
* The upload.osgeo.org DNS name also points to it.  
 
* The downloadable tree is found in /osgeo/download on the server.
 
* The downloadable tree is found in /osgeo/download on the server.
* The virtual host declaration is found in /etc/httpd/apach2/sites-available/download.osgeo.org.conf.
+
* /osgeo/download/download6 sub tree is nfs mounted from osgeo6 (currently osgeo4w and qgis in /osgeo/download point to a corresponding directory in download6)
 +
* The virtual host declaration is found in /etc/apache2/sites-available/download2.osgeo.org.conf.
 
* Logs are in /var/log/apache2/download_access_log and /var/log/apache2/download_error_log.
 
* Logs are in /var/log/apache2/download_access_log and /var/log/apache2/download_error_log.
  
 
= Access Log Viewer (awstats) =
 
= Access Log Viewer (awstats) =
  
'''NOTE: awstats is currently missing from the rebuilt download server - Aug 1st, 2010.'''
+
To maintain/view the Apache access logs of download server, the debian [http://awstats.sf.net/ awstats] package has been installed on the server.  
 
+
with site configuration in /etc/awstats/awstats.download.osgeo.org.conf. It's updated every 4 hours by /etc/cron.d/awstats. See them now at [http://download.osgeo.org/stats/]
To maintain/view the Apache access logs of download server, [http://awstats.sf.net/ awstats] has been installed on the server.  
+
Logs can be seen by visiting [http://download.osgeo.org/logs/ http://download.osgeo.org/logs/] The details of awstats installation are
+
 
+
* awstats version 6.6-1 installed using rpm
+
* application root dir /usr/local/awstats/
+
* location of cgi executable /usr/local/awstats/wwwroot/cgi-bin/awstats.pl
+
* site configuration file /etc/awstats/awstats.www.download.osgeo.org.conf (create such file for any other host to be added in future)
+
* Data dir (dir where stats data is stored) /osgeo/download/logs
+
* Updation is done daily 1am (server time) using a cronjob by executing /osgeo/scripts/update_logs.sh (to add more sites in future, add entry to this script)
+
  
 
= Management =  
 
= Management =  
Line 48: Line 37:
 
There is FTP service configured on the '''download.osgeo.org''' blade. The FTP access is served by [http://vsftpd.beasts.org/ vsftpd] - secure and fastest FTP server for UNIX-like systems.
 
There is FTP service configured on the '''download.osgeo.org''' blade. The FTP access is served by [http://vsftpd.beasts.org/ vsftpd] - secure and fastest FTP server for UNIX-like systems.
  
The vsftpd daemon configuration includes as minimal set of options turned on as possible:
+
The vsftpd daemon configuration is the default one provided with Debian and is suitable for anonymous ftp only.
  
* anonymous '''only''' access
 
 
* root directory: /osgeo/download
 
* root directory: /osgeo/download
 
* '''no''' access for local users
 
* '''no''' access for local users
 
* '''no''' write, mkdir or upload privileges
 
* '''no''' write, mkdir or upload privileges
* session owned by '''non-privileged''' user (there has been created a dedicated local user)
 
* custom banner message: ''Welcome to OSGeo Foundation FTP service.''
 
* maximum number of connected clients: '''50'''
 
* maximum number of connections per IP address: '''2'''
 
* user and group identifiers in are hidden from directories listings, and replaced with '''ftp'''
 
  
 
== Controlling FTP Service ==
 
== Controlling FTP Service ==
Line 86: Line 69:
 
  rsync -av download.osgeo.org::download/gdal/data .
 
  rsync -av download.osgeo.org::download/gdal/data .
  
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  /etc/xinetd.d/rsync was also updated to set set disable=yes to disable=no in the hopes the daemon will restart when the blade is restarted (not tested).
+
This is controlled by the rsync configuration file at /etc/rsyncd.conf.  It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.
  
= Backup =  
+
= Webdav =
  
Nightly backups of the /osgeo/download tree are managed using rsync to the /osgeo/backup/rsync/download.osgeo.org
+
Some projects using Maven as their build tool require webdav in order to work. Here's what's required to get that setup:
/home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org if needed.
+
 
 +
* Enable relevant apache modules
 +
sudo a2enmod dav
 +
sudo a2enmod dav_fs
 +
sudo a2enmod ldap
 +
sudo a2enmod authnz_ldap
 +
 
 +
* Add configuration for project specific location and LDAP group filtering
 +
* Add configuration for apache ldap auth
 +
In the vhost config (examples from svn server)
 +
Include includes.d/webdav/<project>.conf
 +
In the project.conf (example from tracsvn server
 +
Include includes.d/ldap_auth.inc
 +
 
 +
= Backup =
  
There is also an rsync to iweb.gdal.org (Frank's personal server in Montreal) in case of extreme emergency, and this is found at iweb.gdal.org/download.
+
Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab.
 +
/home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org if needed
 +
(NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]
 +
[[Category:Services]]

Revision as of 10:41, 20 October 2017

There is a need for a "download server" for OSGeo that provides for bulk downloads of software.

Configuration

The OSU OSL "Download" VM is being used (140.211.15.132).

  • It is known as "download.osgeo.org", and is configured to respond to that.
  • The upload.osgeo.org DNS name also points to it.
  • The downloadable tree is found in /osgeo/download on the server.
  • /osgeo/download/download6 sub tree is nfs mounted from osgeo6 (currently osgeo4w and qgis in /osgeo/download point to a corresponding directory in download6)
  • The virtual host declaration is found in /etc/apache2/sites-available/download2.osgeo.org.conf.
  • Logs are in /var/log/apache2/download_access_log and /var/log/apache2/download_error_log.

Access Log Viewer (awstats)

To maintain/view the Apache access logs of download server, the debian awstats package has been installed on the server. with site configuration in /etc/awstats/awstats.download.osgeo.org.conf. It's updated every 4 hours by /etc/cron.d/awstats. See them now at [1]

Management

It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files. Scp, or sftp can be used to bring files onto the server. Someone from each project will need their LDAP OSGeo Userid shell enabled, for ssh/scp access.

Any administrator can create new project directories, and chown them to a project representative.

Contact Frank Warmerdam for overall configuration questions or about problems.

End Users

End users should be referred to downloads similarly to:

http://download.osgeo.org/gdal/gdal-1.3.2.tar.gz

Directory indexing is left on so requesting a directory will give a file list.

FTP Service

There is FTP service configured on the download.osgeo.org blade. The FTP access is served by vsftpd - secure and fastest FTP server for UNIX-like systems.

The vsftpd daemon configuration is the default one provided with Debian and is suitable for anonymous ftp only.

  • root directory: /osgeo/download
  • no access for local users
  • no write, mkdir or upload privileges

Controlling FTP Service

In order to control the vsftpd server, you need to:

  • belong to sudoers
  • login in to the download.osgeo.org host using SSH client.

Here you can see how to issue two basic commands:

  • Start
$ sudo /etc/init.d/vsftpd start
  • Stop
$ sudo /etc/init.d/vsftpd stop
  • Restart
$ sudo /etc/init.d/vsftpd restart

Rsync

The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance:

rsync -av download.osgeo.org::download/gdal/data .

This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.

Webdav

Some projects using Maven as their build tool require webdav in order to work. Here's what's required to get that setup:

  • Enable relevant apache modules
sudo a2enmod dav
sudo a2enmod dav_fs
sudo a2enmod ldap
sudo a2enmod authnz_ldap
  • Add configuration for project specific location and LDAP group filtering
  • Add configuration for apache ldap auth

In the vhost config (examples from svn server)

Include includes.d/webdav/<project>.conf

In the project.conf (example from tracsvn server

Include includes.d/ldap_auth.inc

Backup

Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab. /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org if needed (NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).