OSGeoLive AddProject

From OSGeo
Revision as of 11:07, 26 November 2017 by Darkblueb (talk | contribs) (changeset - category to OSGeoLive)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

What gets into OSGeoLive?

We have an emphasis of quality, and we are space constrained on OSGeoLive and so are selective about what is included.

Priority for selecting OSGeoLive projects to package is:

  1. OSGeo Projects
  2. OSGeo Projects in incubation
  3. Stable GeoSpatial Open Source Software being presented in workshops, tutorials, or presentations at FOSS4G
  4. Other Stable GeoSpatial Open Source Software with established project communities and a wide user base
  5. It will be ready for the next release in accordance with our schedule

Application Questions

Projects need to provide:

  • Project Champion
Someone who will liaise between the OSGeoLive team and the project and ensures the application remains up to date on OSGeoLive.
  • A base #Install_Script for installing onto an Ubuntu Linux based system.
  • #Documentation in line with OSGeoLive templates
  • Someone who will run and verify the applications works in development releases
  • Project Status to be maintained.

If you think you can cover all these criteria for your project, then send an email to the OSGeoLive email list asking if your project is appropriate. When emailing the list, please address the questions in the Live_GIS_Disc_Apply form.

Which version to use?

  • Stable, always! LiveGIS users are usually seeing OSGeo newbies and will be turned off by any software bugs.

Install Script

Each project is to write a shell script which installs and configures your stable application into the OSGeoLive Lubuntu.

All install scripts for building the LiveGIS disc shall be licensed as LGPL.

Scripts are stored here: https://github.com/OSGeo/OSGeoLive/tree/master/bin

Ideally, your application will already be packaged as a .deb file, probably in UbuntuGIS. Then the main part of the script uses apt-get, such as install_mapserver.sh, which is primarily 

apt-get install mapserver

Projects that haven't been packaged for Ubuntu yet are slightly more complicated:

The uDig package is a good example: install_udig.sh

  • 3rd Party Repositories: Create an <app>.list file with your repository in sources.list.d/ in the git, in your script copy the file to the local machine like
cp -f ../sources.list/ubuntugis.list /etc/apt/sources.list.d/

don't forget to add the repository key like so 

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68436DDF

The script should create an icon for desktop and menu. See section under: #### install desktop icon #### in install_qgis.sh.

When finished, ask the OSGeoLive team to reference your script from main.sh, and add your icon from install_icons_and_menus.sh

Packaging conventions

  • sudo: Assume the script is to be executed as root. This means that sudo is only needed if you need to execute a command as e.g. the database user.
This means that ~/ refers to the directory /home/root/, so you should use /home/user/ explicitly or its equivalent.
Preferred method is to put the USER_NAME and USER_HOME variables at the top, so these can be easily changed in all scripts if needed.
USER_NAME="user"
USER_HOME="/home/$USER_NAME"
cp package_readme.txt "$USER_HOME"
  • Called twice: Assume that the script will be executed multiple times, (as people may re-run the install process to get it right).
Make sure that nothing is corrupted if the script runs a second time.
  • No Prompts: Try not to have any interactive steps which prompt the user for an answer, as we want the scripts to be run automatically. In particular, try running the script a second time, and see if you get prompted to overwrite files.
  • /tmp: Copy all downloads into /tmp/build_<package>/
Don't delete the /tmp/build_<package> directory afterwards. If the script is re-run, then time and bandwidth is saved by not having to download large files again. Do not do code checkouts or place temporary files in the starting build directory, which is ./bin/ in the live disc's git tree. See the "BUILD_DIR" environment variable used in a number of scripts if you later have to refer back to the ../app-conf/ or ../app-data/ relative paths in the live disc's git tree.
  • Icons: Where appropriate, create an icon on the desktop which loads the application. The file should be called <packagename>.desktop and placed in /home/user/Desktop/ for further processing. The icons will be sorted and the Geospatial menu will be auto-generated from here.
Install <packagename>.png icon files to the /usr/share/icons/ directory.
see install_qgis.sh for an example
  • Browser bookmarks: If the application is launched from the browser, then include the application in Firefox bookmarks.
(Todo: How do we do this?)
  • wget: When downloading large files using wget, use the "-c" continue command, as it will not re-download files if they already exist, and will continue if one is partially downloaded:
wget -c package.tar.gz
Do not use -c if there is a chance the file of that name will change on the server. Add "--progress=dot:mega" to the wget command line if the file is at all large.
Tiny files should use wget's -nv (non-verbose) flag.
In a similar manner, use "--quiet" when calling qsql.
  • mkdir: Use -p (create parents too) option when creating a directory, as it doesn't complain about creating the directory a second time.
mkdir -p <dir>
  • Binary files: Do not put large (>5mb) binary files into the source repository. Due to a preservation of history, once binary files are added they can never be removed, and forever-more clog up the backend database. A source code management tool is not a data warehouse. There is some space for (>5mb) static data files (e.g. sample data) at http://download.osgeo.org/livedvd/data/. Just ask on the mailing list if you'd like to house something there.

Packaging considerations

The criteria used to select applications for the LiveGIS follow.

Technical limitations

Space considerations

  • DVD: approx 4.2gb compressed
  • USB stick: For a 4gb persistent USB stick with some room left over for user-workspace, 3.2-3.5gb compressed. (This pretty much works out to the DVD version minus the preloaded Mac and Windows installers)
  • Virtual Machines: theoretically unlimited, but in practice 20gb uncompressed

So smaller apps (<30mb) of lesser priority may easily slip in under the ire radar, but an external project that wants to use a 25% of the available disc space would be the first to be compromised away.

Memory considerations

Both Virtual Machines, and a LiveDVD images are likely to be constrained by limited memory. So to reduce memory usage. Disk image size is not of major concern, as we can just distribute less data.

The following principles should be followed.

  • Do not start applications upon power up. (Ie, don't start deamons, allow users to start them instead).
  • Set up examples which, by default, don't depend on other applications. Less applications open, means less memory. Ie, Have GeoServer access a shapefile instead of PostGIS.
  • Try to avoid scenarios which write data to disk, as disk space in the Live DVD is stored in RAM, and is not cleared afterward.

File permissions and filesystem user IDs can get garbled during the ISO build process. Therefore a special method is used to ensure that data files which must be read-write can be accessed.

To install a writable data directory or files, change its group to "users" and set the file(s) as group-writable: 

adduser $USER_NAME users
chmod -R g+w /usr/local/share/<packagename>/data/
chown -R root.users /usr/local/share/<packagename>/data/

and symlink into the user's home directory if necessary.

Setting files to have universal write permission is strongly discouraged. Do not use "chmod 777 -R".

Shared resources

The disc is a busy place, so we must all play nice and take care not step on each others toes.

Changes to the base system

Please don't customize any system level infrastructure which other projects may be relying on. Often there is an /etc/$foo.d/ directory for you to put config additions into. Always put non-.deb packaged custom things in /usr/local not /usr!

Port numbers

To avoid overlaps and conflicts with other packages, please avoid putting web services on common port numbers like 8000 or 8080. Choose a random port between 8000 and 8080, search through the whole documentation for this port number (e.g. "grep -R 8042 doc bin") and only use this number if it isn't mentioned anywhere else. Then add your new port number to your quick guide so that others will find it.

Rasdaman is using ports 7001 (dispatcher), 7002, 7003, etc. (worker processes). The number of workers is fixed in the etc/rasmgr.conf file, at least one is required.

Directory Structure

  1. /tmp Temporary files (e.g., downloaded archives) go into /tmp. Please create a separate folder for your project.
  2. /usr/lib application are usually installed into /usr/lib
  3. /usr/bin things that get executed by the user such as startup scripts or links to them should go into /usr/bin
  4. /etc/init.d startup/shutdown scripts for services (e.g., postgres, apache, tomcat) are stored in /etc/init.d
  5. /usr/local/share sample data and documentation goes into /usr/local/share/<project>
  6. /etc config files are stored in /etc
  7. /home/user user specific config files or working directories can go into /home/user/. However, keep files in /home/user as small as possible as this folder is loaded into memory in the Live DVD. Symlinks into /usr/local can be useful here.
  8. /var/www/html. This is the directory references. For user specific documentation, a symbolic link from this directory/<project> can point to /usr/local/share/<project>
  • You are strongly encouraged to use /usr/local/ and /var/local/ for storing non-packaged content.

.deb Packaging

Currently many packages are installed from bash scripts. The major release goal for the next versions of the Osgeo live DVD is to make sure that all programs can be installed from debian packages.

This offers many advantages: packages can be uninstalled and updated to newer versions without having to build the complete live dvd. This also makes testing more easy: one can just update to the last live dvd rather than having to install all software. Ideally, applications should be packaged in UbuntuGIS or DebianGIS repositories, or in a Launchpad PPA (e.g. available from the UbuntuGIS repository). This would mean that any ubuntu or debian user can profit from the packaging works for the osgeo live dvd.

Even though packaging can be overwhelming at first, it eventually boils down to creating a few files in a debian subdirectory from a place where the sources of your application are. A quick and easy way to generate a template is by running dh_make (from the dh-make package) in the directory where your sources are. This will create the necessary files to build a package which you will have to edit before packaging for real.


Many tutorials exist on the web that can assist building debian packages, so I will not repeat all of them here. https://wiki.debian.org/IntroDebianPackaging provides a good start.

If you are new to debian packaging probably the best you can do is ask on the mailing lists of UbuntuGIS or DebianGIS whether someone can provide a template to start your own package.

Packaging Python

Instead of using "pip", "easy_install" or other Python installation methods it is preferred to install the python-pkgname package if it exists. By integrating the python libraries into the dpkg system-files database we can quickly find files and make sure that there are not two different versions from two different sources competing and conflicting. Fortunately it is very easy to create a .deb from projects on Pypi.

  • First check that the software is not already on the disc with "dpkg -S name" and "locate name".
  • Next check if it exists as a package, just not currently installed. For example:
apt-cache search fiona
  • If needed, install the packaging tools:
sudo apt-get install python-stdeb build-essential python-all-dev
  • Run pypi-install to download, build, and install the .deb. For example:
sudo pypi-install Fiona

Add the "--keep" command line option to retain a copy of the .deb and .changes file in a subdirectory of /tmp, e.g. for upload to UbuntuGIS's repository.

See also this mailing list thread with further details on manually packaging using the same tools.

Example Datasets

In order to reduce disk space, and to provide consistency between projects, we strongly encourage all projects to make use of the common datasets packaged on OSGeo-Live by the load_gisdata.sh script.

All datasets should be installed into /usr/local/share/data, or if installed elsewhere, there should be a symbolic link into /usr/local/share/data. Data should not be installed in the /home/user directory, as I understand that data in the HOME directory is copied into RAM. (TBD find confirmation of this).

There is a symbolic link from /home/user/data to /usr/local/share/data to make it easy to find from a user's home directory. (Documentation should reference /home/user/data/... ).

Datasets are also available via http://localhost/data/ and http://live.osgeo.org/data/ (which is a symbolic link from /var/www/data to /usr/local/share/data).

Natural Earth

The Natural Earth dataset is the preferred dataset for all examples.

  • 1:10m Simple Populated Places. Shapefiles are in /user/data/natural_earth/* and have also been loaded into PostGIS.
    • admin_0_countries
    • admin_1_states_provinces
    • geography_marine_polys
    • geography_regions_elevated_points
    • geography_regions_polys
    • lakes
    • land
    • ocean
    • populated_places_simple
    • rivers_lake_centerlines
    • urban_areas
  • Raster basemap -- Cross Blended Hypso with Shaded Relief and Water 1:50 million, stored in /user/data/natural_earth/HYP_50M_SR_W/*

OpenStreetMap

OpenStreetMap data is loaded into /usr/local/share/data/osm/feature_city.osm

North Carolina

The OSGeo-Geodata collection of geotiffs and shapefiles for Wake County (North Carolina) has been tentatively added for the 5.0 release.

The North Carolina layers can be found in /usr/local/share/data/north_carolina/ (symlinked to ~/data/north_carolina)

Others

If you need another type of data for your examples, please discuss this with us.

Documentation

OSGeo-Live documentation is published at:

On OSGeo-Live
http://localhost/ which is stored in /var/www/
Stable release
http://live.osgeo.org
Nightly build
http://adhoc.osgeo.osuosl.org/livedvd/docs/en/index.html

Application Overview

The Application Overview is a one page marketing page targeted at a someone considering using GeoSpatial Open Source, who may be a GeoSpatial User, a Technical person without much geospatial expertise, or manager with limited GeoSpatial or technical experience. Published Overviews are linked from here: http://live.osgeo.org/en/overview/overview.html

The source Overviews are written in RST wiki format and built using sphinx. Source documents are to be stored here: https://github.com/OSGeo/OSGeoLive-doc/tree/master/en/overview/<application>_overview.rst . They should follow the structure and style of the reference Overview, which has lots of helpful comments, at https://github.com/OSGeo/OSGeoLive-doc/blob/master/en/overview/postgis_overview.rst

The Overviews are currently published as HTML pages. In future we intend to also create 1 page PDFs, (to hand out as a flier at conferences, and as a page in the OSGeo Book of projects), and as Word/OpenOffice (to be cut and pasted into project proposals and articles which usually use these formats).

Overviews also require:

  • an image for the application, usually a screen shot or collage of screen shots, as per Screen Shot (don't use GIF format, please use PNG instead).
  • A project logo as per #Logo (don't use GIF format, please use PNG instead).

Try printing the html page from the browser, and ensure that the page is less than one page long. (The less words you write, the more people will read your documentation.)

Once finished, note your project's overview status="draft" in the OSGeo-Live project status at: http://spreadsheets.google.com/ccc?key=0Al9zh8DjmU_RdGIzd0VLLTBpQVJuNVlHMlBWSDhKLXc&hl=en_GB#gid=5 and email live-demo@lists.osgeo.org noting your documentation is ready for review. Once reviewed, the status is moved to "complete" and is ready for translation.

Application Quick Start

The Application Quick Start defines detailed steps, with screen shots, for a new user to run one of the application's core functions. It should be able to be run in 5 to 10 minutes.

Try to make use of a friendly, conversational writing style, similar to the way a tutor would talk a class through an example. The quickstarts for R and GeoServer do this very well.

Source documentation is written in RST wiki format, is built using sphinx, and is stored here: https://github.com/OSGeo/OSGeoLive-doc/tree/master/en/quickstart/<application>_quickstart.rst

The reference quickstart, which includes structure guidelines and with tips on how to fill in the quickstart is here: https://github.com/OSGeo/OSGeoLive-doc/blob/master/en/quickstart/udig_quickstart.rst

In order to reduce disk space, and maintain consistency between applications, we request that all examples make use of the common #Example_Datasets. If the datasets don't cover your requirements, discuss this with us and we may add another suitable common dataset.

Applications are expected to include screen shots for each major step in the quickstart, as per Screen Shot.

Once finished, email live-demo@lists.osgeo.org noting your documentation is ready for review.

Quickstart Review Checklist


Someone experienced with the project
  • Is the quickstart applicable for the version of the application installed on OSGeo-Live?
  • If sample data is used, is it one of the OSGeo-Live datasets?
  • Is there a screenshot for each step?
  • Where appropriate, have screenshots been labeled (usually with numbers) to link them back to the documentation?
  • Once finished, update Quickstart status to "6.5 Draft" and "Quickstart reviewed by:" to "osgeolive-version <your name>" in Status Spreadsheet.
  • If not already included, add the editor's name to the top of the .rst file:
:Author: Your Name
QA Tester
  • Has the quickstart been run from start to finish?
  • How long did the quickstart take to run? (Should be between 5 and 15 minutes)
  • Has every step been explained in the quickstart?
  • Once finished, update Quickstart status to "6.5 Reviewed" and "Quickstart reviewed by:" to "osgeolive-version <your name>" in Status Spreadsheet.
  • If not already included, add the testers name to the top of the .rst file:
:Reviewer: Your Name
Reviewer for readability and consistency
  • Would the quickstart be understandable for a novice? Are all technical terms and acronyms explained?
  • Is the language clear, and concise?
  • Does the first paragraph explain what this Quickstart is going to cover?
  • Are there steps for one scenario?
  • Does the quickstart include "Things to try" and/or "What next" sections?
  • Once finished, update Quickstart status to "6.5 Final" and "Quickstart reviewed by:" to "osgeolive-version <your name>" in Status Spreadsheet.
  • If not already included, add the reviewer's name to the top of the .rst file:
:Reviewer: Your Name

The Project's logo is used in the headers of documentation (like Project Overviews, QuickStarts, Powerpoint presentations etc, and as icons on the OSGeo-Live Desktop. Hence there are a few different requirements for the logo.

Header logo
  • Stored here: https://github.com/OSGeo/OSGeoLive-doc/tree/master/images/project_logos
  • Filename should be "logo-<projectname>.png" (apparently PNG is required for PDF docs)
  • Should have a transparent background (not white)
  • Preferably stored in size 125x125 pixels (for use in Project Overview and QuickStarts)
  • For PNGs please run a program like pngcrush or optipng to reduce the size of the image without degrading quality. Here's an example using optipng:
optipng -o5 image.png
  • Preferably also available as SVG, stored as "logo-<projectname>.svg"
Desktop logo
We are still to define the requirements for creating a desktop logo. They will likely include:
  • A 32x32-pixel XPM icon for use by the Debian menus
  • A 48x48-pixel PNG icon for use by freedesktop.org menus

Screen Shot

Project overviews include an image, which is usually a screen shot, or collage of screen shorts. Quickstarts include screen shots for each significant step.

  • Screen shots can be taken using Shutter (on linux) or Greenshot (on windows).
  • Screen shots should be taken from a 1024x768 display, and should be created in PNG format. (Apparently PNG is required for PDF documentation.)
  • Images are to be stored here: https://github.com/OSGeo/OSGeoLive-doc/tree/master/images/screenshots/1024x768
  • For Quickstarts, consider marking up the image to explain the current steps. Eg: Add circled numbers: 1, 2, 3 or draw an oval around buttons being described. This is very easy to do using the "Edit" tab in the Shutter program, which provides these drawing icons to add. Tutorials augmented with detailed and pertinent images make it easy for the reader to follow what is going on. Extra time spent on an image pays off big in comprehension (you need more than just a course screen dump). Lines, numbering, highlights, boxes and annotations all help direct a user's focus to those areas which are important.
  • Screenshots with large areas of constant color (menus, etc.) should be in PNG format, screenshots containing large areas of imagery (satellite images, shaded relief DEMs, etc.) should be in JPEG format.
  • For PNGs please run a program like pngcrush to reduce the size of the image without degrading quality.

Extra Documentation

Some projects link to extra documentation, stored as PDFs or similar, or alternatively provide example applications. These should be referenced in documentation via a relative link such than it can viewed from: 

 http://localhost/<application>/... or http://live.osgeo.org/<application>/...

This should typically be achieved by creating a symbolic link from /var/www/<application> to /usr/local/share/doc/<application>

Publishing Documentation

The following web service is useful for checking reST formatting: http://rst.ninjs.org/

For a more comprehensive test, set up the reST processor, sphinx:

(you will have to have Sphinx 0.6.4 or greater installed, it's in the python-sphinx Debian package)
 sudo apt-get install python-sphinx

 cd osgeo/livedvd/gisvm/trunk/doc/
 make html

You can then view the results in: 

osgeo/livedvd/gisvm/trunk/doc/_build/*.html

Translations

For details on how to translate, refer to Live GIS Translate

Notebooks

OSGeo-Live provides a set of digital notebooks to explore several open source solution for geospatial data analysis, with the aim of bridging together the several software libraries already installed on the OSGeo-live to perform complex geo-data-science workflows. The notebooks are developed in the Jupyter environments which is heavily based on the IPython notebook project, are written id different languages (bash, python, R) and are organized in a series of "topic-oriented" geospatial notebooks.

Astatic HTML rendering of the notebooks shipped within the OSGeo-Live is available at: OSGeoLive-Notebooks

Notebook-Guidelines

The intended use of the notebooks on the OSGeo-live is to include rich descriptive narrative to a data processing workflow, including the possibilities to interact with data through a nicely done interactive widget system and finally enabling the printing of publication-ready reports (see latex inline rendering and pdf export).

On the OSGeo-live, we are aiming to use the same Example_Datasets among projects this way different software projects can be more easily compared, say for their usage complexity or for output quality/performance comparison.

Notebooks shipped within the OSGeo-live must:

  • be the result of original work specifically designed to run on the OSGeo-live
  • describe a workflow related to geospatial data analysis where code and software documentation are merged together by a rich descriptive narrative.
  • use dataset already present on the OSGeo-live dis (Notebooks developer are free to generate novel data i.e.: from querying available database or using using numerical tools to generate synthetic dataset)
  • avoid the use of network resources to allow offline use (alert the user otherwise)
  • when applicable use hyperlinks to the official project documentation (i.e.: if use a GRASS GIS command, the notebook developer is invited to ad add reference to the official documentation available on the OSGeo-Live)
Retrieved from "http://wiki.osgeo.org/w/index.php?title=OSGeoLive_AddProject&oldid=111211"