Difference between revisions of "Download Server"
(→Access Log Viewer (awstats): awstats now installed from 'standard' debian package) |
(→Rsync) |
||
| (21 intermediate revisions by 8 users not shown) | |||
| Line 1: | Line 1: | ||
| − | There is a need for a "download server" for OSGeo that provides for bulk downloads of software | + | There is a need for a "download server" for OSGeo that provides for bulk downloads of software. |
| − | |||
| − | |||
= Configuration = | = Configuration = | ||
| − | + | The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to [[SAC_Service_Status#Download|SAC_Service_Status Download Container]] | |
| − | * | + | * download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load. |
| − | + | * The upload.osgeo.org DNS name only points to the osgeo7 download container | |
| − | * The upload.osgeo.org DNS name | + | * The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo) |
| − | * The downloadable tree is found in /osgeo/download on the server. | + | * The virtual host declaration is found in /etc/nginx/sites-available |
| − | * The virtual host declaration is found in /etc/ | + | * Logs are in /var/log/nginx |
| − | * Logs are in /var/log/ | ||
| − | + | * bottle.download.osgeo.org (Mac Bottles) is also hosted on this server and in folder /osgeo/bottle mirrored at https://ftp.osuosl.org/pub/osgeo/bottle | |
| − | |||
| − | |||
| − | |||
= Management = | = Management = | ||
| Line 24: | Line 18: | ||
Any administrator can create new project directories, and chown them to a project representative. | Any administrator can create new project directories, and chown them to a project representative. | ||
| − | |||
| − | |||
= End Users = | = End Users = | ||
| Line 31: | Line 23: | ||
End users should be referred to downloads similarly to: | End users should be referred to downloads similarly to: | ||
| − | + | https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | Directory indexing is left on so requesting a directory will give a file list. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
= Rsync = | = Rsync = | ||
| Line 69: | Line 31: | ||
The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance: | The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance: | ||
| − | rsync -av | + | rsync -av upload.osgeo.org::download/gdal/data . |
This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it. | This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it. | ||
| + | |||
| + | = Webdav = | ||
| + | |||
| + | Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup | ||
| + | for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools | ||
= Backup = | = Backup = | ||
| − | Nightly backups of the /osgeo/download tree are managed using rsync to | + | Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab. |
| − | /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org if needed. | + | /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed |
| + | (NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org). | ||
| − | There is | + | There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup. |
[[Category:Infrastructure]] | [[Category:Infrastructure]] | ||
| + | [[Category:Services]] | ||
Latest revision as of 21:53, 10 April 2022
There is a need for a "download server" for OSGeo that provides for bulk downloads of software.
Configuration
The download container on osgeo7, ssh via upload.osgeo.org (requires SSH key) Refer to SAC_Service_Status Download Container
- download.osgeo.org should not be used as it is a round robin that floats between osgeo7, osgeo8, and osgeo9 to balance load.
- The upload.osgeo.org DNS name only points to the osgeo7 download container
- The downloadable tree is found in /osgeo/download on the server. (mirrored at https://ftp.osuosl.org/pub/osgeo)
- The virtual host declaration is found in /etc/nginx/sites-available
- Logs are in /var/log/nginx
- bottle.download.osgeo.org (Mac Bottles) is also hosted on this server and in folder /osgeo/bottle mirrored at https://ftp.osuosl.org/pub/osgeo/bottle
Management
It is intended that each interested project should have a directory under /osgeo/download with appropriate permissions so a project member can manage the subdirectories and files. Scp, or sftp can be used to bring files onto the server. Someone from each project will need their LDAP OSGeo Userid shell enabled, for ssh/scp access.
Any administrator can create new project directories, and chown them to a project representative.
End Users
End users should be referred to downloads similarly to:
https://download.osgeo.org/gdal/gdal-1.3.2.tar.gz
Directory indexing is left on so requesting a directory will give a file list.
Rsync
The /osgeo/download area is exported for anonymous rsync read access as a module named "download". The following can be used to pull a local copy of the gdal data tree for instance:
rsync -av upload.osgeo.org::download/gdal/data .
This is controlled by the rsync configuration file at /etc/rsyncd.conf. It is possible that rsyncd will not automatically restart on reboot in which case "rsyncd --daemon" as root may be necessary to start it.
Webdav
Some projects using Maven as their build tool require webdav in order to work. There is a standalone webdav in nginx setup for geoserver/geotools use using a htpasswd file in /etc/nginx/auth/geotools
Backup
Nightly backups of the /osgeo/download tree are managed using rsync to /mirror/rsync/download.osgeo.org on backup.osgeo.org by root's crontab. /home/other_backups/download directory backup.osgeo.org by a nightly cronjob, and this mirror is available at the url download2.osgeo.org and http://ftp.osuosl.org/pub/osgeo/download/ if needed (NOTE: that rsync does not delete file on backup.osgeo.org, if they were removed from download.osgeo.org).
There is a weekly backup of the whole download container backed up as stopped container on osgeo4 and called download-backup.