AfricaEvents29November2008

...back to Africa_Local_Chapter

What?

Our first OSGeo local chapter meet up! We will chew the cud, show you some cool stuff and start to get people familiar with the brilliant FOSS GeoSpatial products out there!

We'll go through installing some software and getting it to a point where you can have fun with it.

How Much?

There will be a door charge of R50 to cover costs & refreshments. That and an open mind are all you need to bring. If you're staying for the QGIS session in the afternoon you can bring your laptop (Windows, Linux or Mac!)

Where?

St Davids Marist Inanda College
36 Rivonia Rd
Inanda
Sandton

Map and Directions

NB: We'll be in the Prep School computer lab.

When?

• November 29th
• Presentation and workshop segment: 9am to 12pm
• QGIS Bug Party! 12h30 onwards

Who

Your friendly presenters for the day will be:

• Tim Sutton (tim at linfiniti.com) - QGIS hacker and FOSS GIS junkie extraordinaire. Tim eats Weetbix for breakfast because it sounds like Linux. He once rewrote Nkosi Sikelela in C++ because he was feeling patriotic.
• Graeme McFerren - (graeme.mcferren at gmail.com) - GIS researcher and FOSS4G user. Mmm, aspire to be like Tim then.
• Gavin Fleming (gavinjfleming at gmail dot com) - I enjoy my daily FOSS4G fix and telling people how great it was. I'm a 'geoinformation scientist' at Mintek and love to see cutting edge and useful geospatial applications emerging.
• Brendon Wolff-Piggott - (brendon at integratedgeodata.co.za) - Water resources consultant and FOSS GIS enthusiast. I wish there were more hours in the day to keep up with FOSS4G!

RSVP

Let gavinjfleming at gmail dot com know if you're coming as numbers are limited.

QGIS Bug Party

As a preview release there are still various bugs that need resolving and various elements (e.g. translation) that may be incomplete. On Saturday the 29th in Johannesburg, South Africa, members from the Africa OSGeo chapter will be holding a QGIS Bug Hunt Party. We would like to invite people from around the world to virtually participate. Each person attending will get a free copy of QGIS 1.0.0 Preview 2 absolutely free of charge! Sorry no shrink wrap copies available. Preview 2 is the second in a series of preview releases we are making before we release the Stable QGIS 1.0.0 with long term support. We are going to great lengths to polish up QGIS 1.0 for you (inside and out) and we really appreciate all the feedback we have received thus far. Its virtually impossible to ship any software completely bug free, but with your help we can add just a little more shine to the polish! We need your help to:

• test on different machine configurations (bring your laptops!)
• validate existing bugs on each platform
• mark bugs that can no longer be replicated as ready for closure
• create new tickets for bugs that have not been reported yet (please search first to avoid posting duplicates!)
• mark bugs that are duplicates
• provide additional diagnostics so that developers can replicate issues

The idea is not to try to *fix* the bugs, but rather to do a QA session on the bugs in the bug tracker. In the process you will learn how to interact with an Open Source project the Right Way i.e. filing good quality bug reports.

I will start off with a simple show and tell covering:

• where to find the QGIS (and other OSGeo projects bug tracking sites)
• creating an account so you can file non-anonymous bugs and feedback on any OSGeo project bug tracker
• how the bug tracking system works - what the various input fields in the QGIS tracker mean

The approach will be to start by going through any bugs that have not been revised recently, adding additional status information. If you think the bug is fixed, add a comment:

Works for me now, recommend closure, <Your Name>

If you can replicate the bug, add a comment like:

Can replicate, OS: Ubuntu 8.10, QGIS Preview 2, <Your Name>

Use your initiative to deal with tickets that don't fit into the categories above, following a similar approach.

After that I will turn you loose on the queue and a shiny copy of QGIS preview 2 and let you have at it. Party is over when the last man falls asleep at the keyboard! Ok probably we will finish at between 4:30pm and 5pm if my wife has anything to do with it!

We look forward to seeing you at the QGIS Bug Party!!!

Programme in a nutshell

• Be there in time for a 9am start
• Morning session: Introducing QGIS, PostGIS, GeoServer, uDig
• Afternoon session: QGIS Bug Party

Note: There will be frequent rest breaks and lots of time to ask questions!

1. Welcome, Introduction and Overview: Starting up (5 minutes)

What am I looking at here? What do I need to do to get started?

2. QGIS: Opening and viewing spatial data. Your first map. (15 minutes)

What can I look at? How do I set up a simple map? Toolbars and Plug-ins.

3. PostGIS: Why use PostGIS? Simple example of loading data. (15 minutes)

Why use a spatial database? What does one look like? How do I populate a spatial database and where does my data live?

4. QGIS: Looking at PostGIS data from QGIS. (15 minutes)

View a PostGIS database from QGIS. Setting up data definitions for PostGIS sources. Loading data into PostGIS with the Spit Loader (no, please don't touch the screen!)

5. PostGIS: More than a place to dump data... (15 minutes)

PostGIS is not just good for storing data, one can manipulate it too! GIS can be done inside the database, which is rather nice when you think about it.

6. uDig: Why uDig? GIS and the Web. (15 minutes)

Getting GIS data ready for presentation on the Web! Editing SLDs.

7. GeoServer: Tell everyone about it! (15 minutes)

Displaying GIS data on the web. Grouped WMS. Scale-dependent rendering.

8. And more QGIS.

Detail on symbology. Topological editing. QML, SPD and Openlayers. (15 minutes)

9. Show off session

Here we can do some whizz bang demo stuff that we don't have time to do in workshop format but that will give you some hints as to what else can be achieved.

10. General

Questions and answers. What you can do next. Resources. Possibly some in-depth examples based on frequent questions. What would you like to see at future workshops?

11. Your assessment of the workshop

Tell us what you thought!

Workshop Notes

Lets make a map - First steps with QGIS

First we show you

• Open QGIS using the icon on the top Gnome Panel
• From the Layer menu, select Add Vector Layer

 Layer -> Add Vector Layer

• A dialog box will appear
• Navigate to

 /home/workshop/gisdata/za/brits/SAExplorer/

• Add the following layer :

 Provinces.shp

• Experiment with the map navigation toolbar to zoom in / out, pan, select etc features.
• Open the Layer properties dialog. There are several ways you can do this:

 1) Select the layer in the legend and then from the menu do Layer -> Properties
 or
 2) Right click on the layer in the legend and choose Properties from the context menu
 or
 3) Double click on the layer in the legend

• Familiarise yourself with the various panels in the vector properties dialog.
• Select the Symbology tab (second from the left)
• From the Style Options panel, choose a pleasing fill colour for the provinces then

 Click OK

Now you try!

Add the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)

 Transport_NationalRoads.shp
 Transport_Railways.shp
 RSATowns.shp
 Dams.shp

Zoom and pan around a little to explore your data Symbolise each layer with an appropriate colour to make the most awe inspiring map.

Save your project using

File -> Save

And put it into

/home/workshop/gisdata/brits.qgs

What have I learned?

In ten minutes you now know how to add vector layers into QGIS, pan and zoom to navigate the map, set the colour of vectors using the Single Symbol option in the Layer Properties.

Shapefiles shmapefiles....real (wo)men store their data in PostGIS

First we show you

• Create a new PostgreSQL database. To do this, open the terminal application (icon to the right of Firefox icon on the top menu bar). Now at the prompt type the following command:

 createdb workshop

• Verify your database exists (once again type this into the terminal)

 psql -l

• You should see something like this:

workshop@heron:~$ psql -l
       List of databases
  Name    |  Owner   | Encoding 
-----------+----------+----------
postgres  | postgres | UTF8
template0 | postgres | UTF8
template1 | postgres | UTF8
workshop  | workshop | UTF8
(4 rows)

• away from the terminal for a while - lets look at what a non-spatial, vanilla PostgreSQL database looks like. Open pgAdmin 3 (the icon with a blue elephant face)
• Double-click on the icon that has a red cross next to some text "gis (localhost:5432)" and press OK if another screen comes up. The database cluster called gis expands and you should see two databases: 'postgres' and 'workshop'. Expand workshop -> Schemas -> public

Specifically, let us look at Functions and Tables - don't look to deeply, this is just to show the differences after PostGIS is installed on the database (you will see later). Specifically, there is nothing in the db at the moment. So minimise pgAdmin and lets crack on with getting some data in!

• Install the PostGIS spatial data extensions (again from the terminal).

Hint copy and paste these from the IRC channel or the wiki page!

createlang plpgsql workshop
psql workshop < /usr/share/postgresql-8.3-postgis/lwpostgis.sql
psql workshop < /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql

• Yegads! What did I just do? Firstly, you installed the plpgsql procedural language extensions into your PostgreSQL database. Then you ran two sql scripts which create the PostGIS spatial extensions to your database. You only need to do this once when you create a new spatial database, so don't worry about remembering those commands off by heart.
• Lets upload a shapefile into the database....once more into the black hole (terminal window).

cd /home/workshop/gisdata/za/brits/SAExplorer
shp2pgsql -I -s 4326 Dams.shp dams | psql workshop

• Erk! What does that all mean? Lets break it down:

cd /home/workshop/gisdata/za/brits/SAExplorer - go to our data directory
shp2pgsql - an application that loads shapefiles into a PostGIS database
-I - an option to shp2pgsql telling it to create a spatial index
-s - the coordinate reference system to use, expressed as an EPSG number. 4326 is the same as Lat/Long WGS84
Dams.shp - the filename of the shapefile to load
dams - the table name that the data should be loaded into
| psql workshop - send the results of the shp2pgsql into the psql application, workshop database

• Lets take a peek inside our database now and see what we have - back to pgAdmin, click on the 'workshop' database icon and then refresh it (red/green circular arrows icon).

Three major things to look at:

Hundreds of Functions for working with spatial data: 
* geometry constructors, 
* spatial predicates, 
* geometry editors, 
* geometry aggregators, 
* geometry accessors, 
* measurements, 
* linear referencing
* and outputting  of spatial data. 

2 important meta-tables: (expand the Tables object)
* geometry_columns (for describing the different spatial datasets) - right-click -> 
* and spatial_ref_sys(a list of projections/coordinate systems and different representations thereof)

A 'dams' table which, when expanded should show a bunch of Columns, Constraints (e.g. Primary Key), Indexes, Rules and Triggers. 
* Note the geometry type of column 'the_geom'. 
* Also note the spatial index present - this allows for optimal retrieval of data based on the spatial properties of the data.

Now you try!

Load the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)

Hydrology_DWAFRivers.shp -> call the table 'rivers'
Transport_Mainroads.shp -> call the table 'roads'
ProvincialBoundary.shp -> call the table 'provinces'

And to show that this can scale a bit, let us load in the background a dataset of over 350 000 points (which we will use a bit later)

* cd /home/workshop/gisdata/za/fire/modis_af/
* shp2pgsql -I -s 4326 modis_af.shp fires | psql workshop  &> /dev/null (puts output into background)

This takes a while so lets move on.

What have I learnt?

• PostgreSQL is an enterprise-ready relational database management system - and it's Free and Open Source Software
• You can create a new database using the createdb command.
• PostGIS is a add-on to PostgreSQL that lets you store spatial data and carry out spatial analysis within the database.
• In order to set up PostGIS you need to run a couple of SQL scripts
• shp2pgsql is a command line application that lets you load shapefiles into your spatial datastore

Using PostGIS data from QGIS

First we show you

Once your data is in PostGIS, its easy to share across your organisation. Because it's in a database you get row level locking (instead of file locking in a shapefile), so many users can be editing and viewing data at the same time. There are many clients that let you visualise your PostGIS data..lets take a look at how we do it in QGIS.

• Open QGIS
• Start a new project

 File -> New Project

• Add a PostGIS layer

 Layer -> Add PostGIS Layer

• From the connections list, choose workshop

 Click connect

• In the table list, click on dams

 Click Add

Now you try!

• Add the following layers from your database

rivers
roads
provinces

• set some nice colors for your layers - notice how the process is exactly the same as when setting colours for a shapefile.

• save your project into

/home/workshop/gisdata/brits_pg.qgs

What have I learned?

• You can visualise PostGIS spatial database tables inside QGIS
• To add a PostGIS layer, use the Layer -> Add PostGIS Layer menu option.
• Once a PostGIS layer, you can treat it just like any other (e.g. shapefile) layer
• Your presenters are truly amazing....however would I have figured this stuff out for my self :-)

The voema of PostGIS - not jsut a place to store data!

Okay, we will go a bit deeper into the woods here! We want to show that with some SQL (I won't say simple) you can do some complex - and repeatable - spatial analysis right inside your database, and then visualise the results in various clients.

First we show you

We will use pgAdmin (your great PostgreSQL friend) to get this happening. Click on the Tables object of your workshop db. Then Tools -> Query tool from the menu. This should present a dialog box with 3 panels.

We are going to do a simple intersection to get started. In the top left panel paste

SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE (roads.the_geom && rivers.the_geom AND intersects(roads.the_geom, rivers.the_geom));

and click Query -> Execute This should take a few seconds, then you will see a result returned into the Data Output grid. Something like 3338 rows are returned from the original roads dataset of 11964.

So what is going on here?

You are performing a spatial overlay with a SQL query, asking the database to return records from the roads table that spatially intersect records from the rivers table.

• the && operator is a fast bounding box 'overlaps' test that takes advantage of the spatial indexes of the table to narrow the search space for the true intersection test.
• the intersects part of the query tests the geometries of the subset selected above for spatial intersection and returns a result.

To show some other functionality, paste the following text into the query window and execute:

ALTER TABLE roads DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'roads';COMMIT; UPDATE roads SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE roads ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

We are being good South African GIS types and using our Haartebeesthoek94 Datum. Lets do the same for rivers:

ALTER TABLE rivers DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'rivers';COMMIT; UPDATE rivers SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE rivers ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

This looks fairly intense, but the main points are the changing of the data in the geometry_columns meta-table and then updating the geometries in each table to a different CRS (4148, or Haartebeesthoek94). We demonstrate the use of the UPDATE SQL command, used to change data in the database, and the use of the ST_Transform function of PostGIS, which does the work. Now lets try the original query again:

SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE st_intersects(roads.the_geom, rivers.the_geom);

...which returns the same number of features, but in our own CRS. Note that we have used the ST_Intersects function rather than the Intersects function as previously shown. ST_ type functions are more standards compliant, so should be used in place of the older style functions. Ideally we would project the data as appropriate, using the very same technique, but substituting 4148 for the EPSG code for our projection of choice. This is a demo though, folks, so I will get away with murder... Lets quickly repeat the process for the other datasets we will use in a minute:

ALTER TABLE provinces DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'provinces';COMMIT; UPDATE provinces SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE provinces ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
ALTER TABLE fires DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'fires';COMMIT; UPDATE fires SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE fires ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

So, while that goes ahead (it will take a minute or so to transform 350k points), let us think out another analysis. Lets make it a bit tricky. Okay, we want to create a table for serving up data to the outside world (Gavin will do this later) that has all the intense fires in Gauteng or NorthWest Province, that have occurred within 3000 metres of a main road. There are many ways to skin this cat in PostGIS, so I will pick a method and explain why I did so. First, a convenience table is created:

SELECT roads.gid, ST_Buffer(roads.the_geom,0.027) AS geom INTO roads_buffer FROM roads, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov WHERE ST_Within(roads.the_geom, prov.the_geom);

This is a table of polygons representing ~3000 metre road buffers, and note the use of the function to populate the geometry_columns meta-table with our new layer.

SELECT DISTINCT fires.gid, fires.frp, fires.acqdatetim, fires.the_geom INTO fires_analysis_3000m FROM fires, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov, roads_buffer  WHERE fires.frp > 200 AND ST_Within(fires.the_geom, prov.the_geom) AND ST_Within(fires.the_geom, roads_buffer.geom);

The main feature of this query is the longish WHERE clause, which tests weather the fire is intense, whether it falls within Gauteng or NW Province and whether the fire occurred within appx 3000 metres of one of our main roads.

There are not too many fires that fall out of this analysis, but it shows how we have relatively easily worked with and reduced a large dataset to a few points. These queries can be further tuned to take advantage of spatial and other indexes, but that is perhaps for another time.

The final thing we will do is prepare this layer for use on the web...

INSERT INTO geometry_columns(f_table_catalog,f_table_schema,f_table_name,f_geometry_column,coord_dimension,srid,"type") VALUES(,'public','fires_analysis_3000m','the_geom',2,4148,'MULTIPOINT');

Normal meta-table stuff

ALTER TABLE fires_analysis_3000m ADD PRIMARY KEY (gid);

And if we wanted to we could add a spatial index:

CREATE INDEX fires_Analysis_3000m_the_geom_gist ON fires_Analysis_3000m USING gist(the_geom);

uDig

First we show you

Once your data is in PostGIS, its easy to share across your organisation. Because it's in a database you get row level locking (instead of file locking in a shapefile), so many users can be editing and viewing data at the same time. There are many clients that let you visualise your PostGIS data..lets take a look at how we do it in QGIS.

• Open QGIS
• Start a new project

 File -> New Project

• Add a PostGIS layer

 Layer -> Add PostGIS Layer

• From the connections list, choose workshop

 Click connect

• In the table list, click on dams

 Click Add

Now you try!

• Add the following layers from your database

rivers
roads
provinces

• set some nice colors for your layers - notice how the process is exactly the same as when setting colours for a shapefile.

• save your project into

/home/workshop/gisdata/brits_pg.qgs

What have I learned?

• You can visualise PostGIS spatial database tables inside QGIS
• To add a PostGIS layer, use the Layer -> Add PostGIS Layer menu option.
• Once a PostGIS layer, you can treat it just like any other (e.g. shapefile) layer
• Your presenters are truly amazing....however would I have figured this stuff out for my self :-)

GeoServer

First we show you

Once your data is in PostGIS, its easy to share across your organisation. Because it's in a database you get row level locking (instead of file locking in a shapefile), so many users can be editing and viewing data at the same time. There are many clients that let you visualise your PostGIS data..lets take a look at how we do it in QGIS.

• Open QGIS
• Start a new project

 File -> New Project

• Add a PostGIS layer

 Layer -> Add PostGIS Layer

• From the connections list, choose workshop

 Click connect

• In the table list, click on dams

 Click Add

Now you try!

• Add the following layers from your database

rivers
roads
provinces

• set some nice colors for your layers - notice how the process is exactly the same as when setting colours for a shapefile.

• save your project into

/home/workshop/gisdata/brits_pg.qgs

What have I learned?

• You can visualise PostGIS spatial database tables inside QGIS
• To add a PostGIS layer, use the Layer -> Add PostGIS Layer menu option.
• Once a PostGIS layer, you can treat it just like any other (e.g. shapefile) layer
• Your presenters are truly amazing....however would I have figured this stuff out for my self :-)

Quick Qml with QGIS

First we show you

If you have used ArcView before, you might have used the avl feature which lets you store the symbology settings for a layer. QGIS has a similar feature called 'qml' which lets you create default symbology settings for a layer, that will be applied when ever that layer is loaded. We are also going to take a quick peek at how to create more complex symbology for a layer.

• Open QGIS if its not already running.
• Add the dams layer from the Postgresql 'Workshop' database
• Double click on the layer entry in the legend
• Activate the symbology tab
• Now change the legend type to Graduated Symbol
• Change the classification field to shape_area
• Set the number of classes to 5
• Click the classify button
• Click the Apply button

You should see the shapes in the main window have now adopted a range of scintillating colours! However is you add that layer to another QGIS project you would normally have to do all this work again! Qml to the rescue.

• Save your symbology settings by clicking the save as default button

Ok lets test it out!

• Close QGIS
• Reopen QGIS
• Add the Dams layer from the Postgresql 'Workshop' database

If the planets are correctly aligned, your dams will appear all beautifully coloured according to your personal preference.

If you want to you can also save your settings as non-default, and them restore them by selecting a .qml file again later. Refer to the Load style and Save style buttons on the vector properties dialog.

Now you try!

Ok give it a whirl, create default appearances for your other layers stored in your PostGIS database. Also try experimenting with the other legend types such as Continuous Colour and Unique Value. Once you have set your default appearances, close QGIS and then reopen it. Now add all your PostGIS layers to the blank project and they should all magically appear with your preferred appearance.

What have I learned?

The Layer Properties dialog in QGIS will let you specify the default appearance for a layer.