Difference between revisions of "AfricaEvents29November2008"

From OSGeo
Jump to navigation Jump to search
 
(124 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
...back to [[Africa_Local_Chapter]]
 +
 
= What? =
 
= What? =
  
Our first OSGEO local chapter meet up! We will chew the cud, show you some cool stuff and start to get people familiar with the brilliant FOSS GeoSpatial products out there!
+
Our first OSGeo local chapter meet up! We will chew the cud, show you some cool stuff and start to get people familiar with the brilliant FOSS GeoSpatial products out there!
 +
 
 +
We'll go through installing some software and getting it to a point where you can have fun with it.
 +
 
 +
= How Much? =
  
 +
There will be a door charge of R50 to cover costs & refreshments. That and an open mind are all you need to bring. If you're staying for the QGIS session in the afternoon you can bring your laptop (Windows, Linux or Mac!)
  
 
= Where? =
 
= Where? =
  
Gavin can you add travel details + lat/lon for the location? Embed an openlayers map here?
+
St Davids Marist Inanda College
 +
36 Rivonia Rd
 +
Inanda
 +
Sandton
 +
[http://www.stdavids.co.za/about_directions.asp Map and Directions]
  
 +
NB: We'll be in the Prep School computer lab.
  
 
= When? =
 
= When? =
  
 
* November 29th
 
* November 29th
* Presentation and workshop segment: 10am to 1pm
+
* Presentation and workshop segment: 9am to 12pm
* QGIS Bug Party!
+
* QGIS Bug Party! 12h30 onwards
  
 
= Who =
 
= Who =
Line 19: Line 31:
 
Your friendly presenters for the day will be:
 
Your friendly presenters for the day will be:
  
* '''Tim Sutton''' (tim at linfiniti.com) - QGIS hacker and FOSS GIS junkie extraordinaire. Tim eats Weetbix for breakfast because it sounds like Linux. He once rewrote Nkosi Sikileli in C++ because he was feeling patriotic.
+
* '''Tim Sutton''' (tim at linfiniti.com) - QGIS hacker and FOSS GIS junkie extraordinaire. Tim eats Weetbix for breakfast because it sounds like Linux. He once rewrote Nkosi Sikelela in C++ because he was feeling patriotic.
 
* '''Graeme McFerren''' - (graeme.mcferren at gmail.com) - GIS researcher and FOSS4G user. Mmm, aspire to be like Tim then.
 
* '''Graeme McFerren''' - (graeme.mcferren at gmail.com) - GIS researcher and FOSS4G user. Mmm, aspire to be like Tim then.
* '''Gavin Fleming''' - add your email and witty bio here
+
* '''Gavin Fleming''' (gavinjfleming at gmail dot com) - I enjoy my daily FOSS4G fix and telling people how great it was. I'm a 'geoinformation scientist' at Mintek and love to see cutting edge and useful geospatial applications emerging.
* '''Brendon Wolff-Piggott''' - (brendon at integratedgeodata.co.za) - Water resources consultant and FOSS GIS enthusiast.  I wish there were more hours in the day to keep up with FOSS4G..
+
* '''Brendon Wolff-Piggott''' - (brendon at integratedgeodata.co.za) - Water resources consultant and FOSS GIS enthusiast.  I wish there were more hours in the day to keep up with FOSS4G!
 
 
= Presentation and workshop segment =
 
  
== Install postgis  :  Gavin ==
+
= RSVP =
* create user account
+
Let gavinjfleming at gmail dot com know if you're coming as numbers are limited.  
* create a db
 
* load plpgsql, lwpostgis, spatial_ref_sys
 
* use shp2pgsql to load a shapefile
 
* perform some basic spatial queries
 
* show inserting geometries
 
* show extracting geometries
 
 
 
== Install geoserver :  Graeme ==
 
* install java 1.6 from apt
 
* grab standalone geoserver binary
 
* extract to opt
 
* export JAVA_HOME=/usr
 
* start geoserver
 
* create a datastore
 
* create a couple of feature types using pg backend
 
* create a WMS layergroup with a few layers
 
* demonstrate the result in geoserver browser
 
 
 
== Install uDig :  Gavin ==
 
* Connect to Geoserver wfs
 
* create nice sld for layer
 
* paste into geoserver and reload
 
* Install QGIS: Tim
 
* Pull in WMS layer from geoserver
 
* Pull up postgis layer
 
* Demonstrate editing and topological editing
 
* Demonstrate thematic mapping options
 
* Demonstrate attribute actions
 
* Demonstrate plugin manager
 
 
 
== Install QGIS : Brendon ==
 
 
 
* Pull in WMS layer from geoserver
 
* Pull up postgis layer
 
* Demonstrate editing and topological editing
 
* Demonstrate thematic mapping options
 
* Demonstrate attribute actions
 
* Demonstrate plugin manager
 
  
  
Line 71: Line 43:
 
= QGIS Bug Party =
 
= QGIS Bug Party =
  
As a preview release there are still various bugs that need resolving and various elements (e.g. translation) that may be incomplete. On Saturday the 29th in Johannesburg, South Africa, members from the Africa OSGEO chapter will be holding a QGIS Bug Hunt Party. We would like to invite people from around the world to virtually participate. Each person attending will get a free copy of QGIS 1.0.0 Preview 2 absolutely free of charge! Sorry no shrink wrap copies available. Preview 2 is the second in a series of preview releases we are making before we release the Stable QGIS 1.0.0 with long term support.  
+
As a preview release there are still various bugs that need resolving and various elements (e.g. translation) that may be incomplete. On Saturday the 29th in Johannesburg, South Africa, members from the Africa OSGeo chapter will be holding a QGIS Bug Hunt Party. We would like to invite people from around the world to virtually participate. Each person attending will get a free copy of QGIS 1.0.0 Preview 2 absolutely free of charge! Sorry no shrink wrap copies available. Preview 2 is the second in a series of preview releases we are making before we release the Stable QGIS 1.0.0 with long term support.  
 
We are going to great lengths to polish up QGIS 1.0 for you (inside and out) and we really appreciate all the feedback we have received thus far. Its virtually impossible to ship any software completely bug free, but with your help we can add just a little more shine to the polish! We need your help to:
 
We are going to great lengths to polish up QGIS 1.0 for you (inside and out) and we really appreciate all the feedback we have received thus far. Its virtually impossible to ship any software completely bug free, but with your help we can add just a little more shine to the polish! We need your help to:
  
Line 84: Line 56:
  
 
I will start off with a simple show and tell covering:
 
I will start off with a simple show and tell covering:
* where to find the QGIS (and other OSGEO projects bug tracking sites)
+
* where to find the QGIS (and other OSGeo projects bug tracking sites)
* creating an account so you can file non-anonymous bugs and feedback on any OSGEO project bug tracker
+
* creating an account so you can file non-anonymous bugs and feedback on any OSGeo project bug tracker
 
* how the bug tracking system works - what the various input fields in the QGIS tracker mean
 
* how the bug tracking system works - what the various input fields in the QGIS tracker mean
  
Line 99: Line 71:
  
  
After that I will turn you lose on the queue and a shiny copy of QGIS preview 2 and let you have at it. Party is over when the last man falls asleep at the keyboard! Ok probably we will finish at between 4:30pm and 5pm if my wife has anything to do with it!
+
After that I will turn you loose on the queue and a shiny copy of QGIS preview 2 and let you have at it. Party is over when the last man falls asleep at the keyboard! Ok probably we will finish at between 4:30pm and 5pm if my wife has anything to do with it!
  
  
Line 105: Line 77:
  
  
 +
= Programme in a nutshell =
 +
 +
* Be there in time for a 9am start
 +
* Morning session: Introducing QGIS, PostGIS, GeoServer, uDig
 +
* Afternoon session: QGIS Bug Party
 +
 +
Note: There will be frequent rest breaks and lots of time to ask questions!
 +
 +
'''1. Welcome, Introduction and Overview: Starting up (5 minutes)'''
 +
 +
What am I looking at here? What do I need to do to get started?
 +
 +
'''2. QGIS: Opening and viewing spatial data. Your first map. (15 minutes)'''
 +
 +
What can I look at? How do I set up a simple map?  Toolbars and Plug-ins.
 +
 +
'''3. PostGIS: Why use PostGIS? Simple example of loading data. (15 minutes)'''
 +
 +
Why use a spatial database? What does one look like? How do I populate a spatial database and where does my data live?
 +
 +
'''4. QGIS: Looking at PostGIS data from QGIS.  (15 minutes)'''
 +
 +
View a PostGIS database from QGIS.  Setting up data definitions for PostGIS sources.  Loading data into PostGIS with the Spit Loader (no, please don't touch the screen!)
 +
 +
'''5. PostGIS: More than a place to dump data... (15 minutes)'''
 +
 +
PostGIS is not just good for storing data, one can manipulate it too! GIS can be done inside the database, which is rather nice when you think about it.
 +
 +
'''6. uDig: Why uDig?  GIS and the Web.  (15 minutes)'''
 +
 +
Getting GIS data ready for presentation on the Web! Editing SLDs.
 +
 +
'''7. GeoServer: Tell everyone about it!  (15 minutes)'''
 +
 +
Displaying GIS data on the web.  Grouped WMS.  Scale-dependent rendering.
 +
 +
'''8. And more QGIS.'''
 +
 +
Detail on symbology.  Topological editing. QML, SPD and Openlayers.  (15 minutes)
 +
 +
'''9. Show off session'''
 +
 +
Here we can do some whizz bang demo stuff that we don't have time to do in  workshop format but that will give you some hints as to what else can be achieved.
 +
 +
'''10. General'''
 +
 +
Questions and answers.  What you can do next.  Resources.  Possibly some in-depth examples based on frequent questions.  What would you like to see at future workshops?
 +
 +
'''11. Your assessment of the workshop'''
 +
 +
Tell us what you thought!
 +
 +
= Workshop Notes =
 +
 +
== Welcome, introductions and overview ==
 +
 +
== Lets make a map - First steps with QGIS ==
 +
 +
=== First we show you ===
 +
 +
* Open QGIS using the icon on the top Gnome Panel
 +
* From the Layer menu, select '''Add Vector Layer'''
 +
  Layer -> Add Vector Layer
 +
* A dialog box will appear
 +
* Navigate to
 +
  /home/workshop/gisdata/za/brits/SAExplorer/
 +
* Add the following layer :
 +
  Provinces.shp
 +
* Experiment with the '''map navigation toolbar''' to zoom in / out, pan, select etc features.
 +
* Open the '''Layer properties dialog'''. There are several ways you can do this:
 +
  1) Select the layer in the legend and then from the menu do Layer -> Properties
 +
  or
 +
  2) Right click on the layer in the legend and choose Properties from the context menu
 +
  or
 +
  3) Double click on the layer in the legend
 +
* Familiarise yourself with the various panels in the vector properties dialog.
 +
* Select the '''Symbology''' tab (second from the left)
 +
* From the '''Style Options''' panel, choose a pleasing '''fill colour''' for the provinces then
 +
  Click OK
 +
 +
Or you can:
 +
* Select the '''Legend Type''' as ''Unique Value''. 
 +
* Set the '''Classification field''' to ''PROVNAME''
 +
* Click on the '''Classify''' button to set '''different colours for each province'''.
 +
  Click OK
 +
 +
=== Now you try! ===
 +
 +
'''Add''' the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)
 +
 +
  Transport_NationalRoads.shp
 +
  Transport_Railways.shp
 +
  RSATowns.shp
 +
  Dams.shp
 +
 +
'''Zoom and pan''' around a little to explore your data'''
 +
'''Symbolise''' each layer with an appropriate colour to make the most awe inspiring map.
 +
 +
Activate the '''Identify features''' icon on the middle toolbar to identify the attribute values associated with a spatial feature for the layer highlighted in the Legend Box.
 +
 +
Activate the '''Select Features''' icon next to Identify Features to select one or more features. 
 +
 +
Open the attribute table to view all feature attributes. Note that selected features will have highlighted rows in the attribute table.  You can do this using the neighbouring '''Open Attribute Table''' icon, or by selecting the Open Attribute Table option on the context menu for the layer in the legend using right-click on the mouse.
 +
 +
You can also select rows in the attribute table by:
 +
* Left-clicking on the desired rows.
 +
* Using the '''Search''' box at the bottom-left of the form.
 +
* Selecting the '''Advanced''' button and opening the '''Search Query Builder'''.
 +
 +
Save your project using
 +
File -> Save
 +
 +
And put it into
 +
/home/workshop/gisdata/brits.qgs
 +
 +
=== What have I learned? ===
 +
 +
In ten minutes you now know how to '''add vector layers''' into QGIS, '''pan and zoom''' to navigate the map, and set the colour of vectors using the '''Single Symbol''' or '''Unique Values''' options in the '''Layer Properties'''.
 +
 +
You have also learnt how to use the interactive '''Identify''' and '''Select''' functions, '''opened attribute tables''' and found out how to '''search''' them.
 +
 +
== More on QGIS Mapping ==
 +
 +
=== First we show you ===
 +
 +
Add the vector layer '''Education_Schools2000.shp''' from the SAExplorer directory.  Notice how dense the points are!
 +
 +
* Right-click on the schools layer in the Legend frame.
 +
* Select '''Properties'''
 +
* Select the '''General tab'''
 +
* Check '''Use scale dependent rendering'''
 +
* Set the maximum field to 1000000
 +
 +
* Zoom out to view the whole country.    '''Notice that the schools no longer appear on the map!'''
 +
* Now start zooming in, keeping an eye on the scale shown in the '''scale box''' at the bottom of the screen.
 +
 +
Notice how the schools points reappear when the scale decreases below 1:1000000
 +
 +
* From the Layer menu, select '''Add Raster Layer'''
 +
  Layer -> Add Raster Layer
 +
* A dialog box will appear
 +
* Navigate to
 +
  /home/workshop/gisdata/za/gauteng/
 +
* Add the following layer :
 +
SOU-8_satellite.tif
 +
 +
Notice that the image is largely in shades of greyish-brown, but that a dam can be seen towards the bottom of the image.
 +
 +
* Open the Properties for the raster layer, in a similar way to that used for vector layers.
 +
Layer -> Properties
 +
* Select the '''Transparency''' tab.
 +
* Using the slider, set transparency to 25%
 +
* Click OK
 +
Notice how the you can now see the vector layers behind the satellite image.
 +
 +
=== Now you try! ===
 +
 +
If you have not already done so, add the vector layers Transport_NationalRoads.shp and Transport_Mainroads.shp from /home/workshop/gisdata/za/brits/SAExplorer.
 +
Set scale-dependent rendering so that the national roads display at all scales, while the main roads are only shown when you are zoomed in.
 +
 +
Go to the satellite image properties, Symbology tab.  Try the different "Render as" options.  Then when you have found the one that suits you, have a look at the "Contast Enhancement" options.  Compare "No Stretch" and "Stretch to MinMax". 
 +
 +
Try "Render as Single Band Grey".  Set the "Custom min/max values" radio button.  Change the Min and Max values.  Go to "Contract Enhancement" again and set it to "Stretch and Clip to MinMax", and see the effect.
 +
 +
Add raster images of 1: 50 000 map sheets from " /home/workshop/gisdata/za/brits/toposheet images"
 +
You will need to set the Files of Type box to '''JPEG JFIF''' to see them, before you can select them.
 +
Set their transparency to a value that allows you to see the vector features behind them.  Zoom in and look at how the vector features and image features match (or not!).
 +
 +
Notice how you can zoom right in to vector data, and a point remains a point and a line remains a line.  When you zoom in to the satellite image it rapidly appears "blocky" compared to the 50 000 map sheets. Zoom in enough and even the map sheet raster blocks can be seen.  This is a basic difference between vector and raster data.
 +
 +
=== What have I learned? ===
 +
 +
You have learnt to set scale-dependent rendering so that data sets are displayed at appropriate scales, how to add raster data and set raster display options.
 +
 +
== Shapefiles shmapefiles....real (wo)men store their data in PostGIS ==
 +
 +
 +
=== First we show you ===
 +
 +
* '''Create''' a new PostgreSQL database. To do this, open the terminal application (icon to the right of Firefox icon on the top menu bar). Now at the prompt type the following command:
 +
  createdb workshop
 +
 +
* '''Verify''' your database exists (once again type this into the terminal)
 +
  psql -l
 +
* You should see something like this:
 +
workshop@heron:~$ psql -l
 +
        List of databases
 +
  Name    |  Owner  | Encoding
 +
-----------+----------+----------
 +
postgres  | postgres | UTF8
 +
template0 | postgres | UTF8
 +
template1 | postgres | UTF8
 +
workshop  | workshop | UTF8
 +
(4 rows)
 +
* away from the terminal for a while - lets look at what a non-spatial, vanilla PostgreSQL database looks like. Open pgAdmin 3  (the icon with a blue elephant face)
 +
* Double-click on the icon that has a red cross next to some text "gis (localhost:5432)" and press OK if another screen comes up. The database cluster called gis expands and you should see two databases: 'postgres' and 'workshop'. Expand workshop -> Schemas -> public
 +
Specifically, let us look at Functions and Tables - don't look to deeply, this is just to show the differences after PostGIS is installed on the database (you will see later). Specifically, there is nothing in the db at the moment. So minimise pgAdmin and lets crack on with getting some data in!
 +
 +
* Install the PostGIS spatial data extensions (again from the terminal).
 +
'''Hint''' copy and paste these from the IRC channel or the wiki page!
 +
createlang plpgsql workshop
 +
psql workshop < /usr/share/postgresql-8.3-postgis/lwpostgis.sql
 +
psql workshop < /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql
 +
 +
* Yegads! What did I just do? Firstly, you installed the '''plpgsql''' procedural language extensions into your PostgreSQL database. Then you ran two sql scripts which create the '''PostGIS''' spatial extensions to your database. You only need to do this '''once''' when you create a new spatial database, so don't worry about remembering those commands off by heart.
 +
* Lets upload a shapefile into the database....once more into the black hole (terminal window).
 +
cd /home/workshop/gisdata/za/brits/SAExplorer
 +
shp2pgsql -I -s 4326 Dams.shp dams | psql -d workshop
 +
* Erk! What does that all mean? Lets break it down:
 +
cd /home/workshop/gisdata/za/brits/SAExplorer - go to our data directory
 +
shp2pgsql - an application that loads shapefiles into a PostGIS database
 +
-I - an option to shp2pgsql telling it to create a spatial index
 +
-s - the coordinate reference system to use, expressed as an EPSG number. 4326 is the same as Lat/Long WGS84
 +
Dams.shp - the filename of the shapefile to load
 +
dams - the table name that the data should be loaded into
 +
| psql workshop - send the results of the shp2pgsql into the psql application, workshop database
 +
 +
 +
* Lets take a peek inside our database now and see what we have - back to pgAdmin, click on the 'workshop' database icon and then refresh it (red/green circular arrows icon).
 +
Three major things to look at:
 +
 +
Hundreds of Functions for working with spatial data:
 +
* geometry constructors,
 +
* spatial predicates,
 +
* geometry editors,
 +
* geometry aggregators,
 +
* geometry accessors,
 +
* measurements,
 +
* linear referencing
 +
* and outputting  of spatial data.
 +
 +
2 important meta-tables: (expand the Tables object)
 +
* geometry_columns (for describing the different spatial datasets) - right-click ->
 +
* and spatial_ref_sys(a list of projections/coordinate systems and different representations thereof)
 +
 +
A 'dams' table which, when expanded should show a bunch of Columns, Constraints (e.g. Primary Key), Indexes, Rules and Triggers.
 +
* Note the geometry type of column 'the_geom'.
 +
* Also note the spatial index present - this allows for optimal retrieval of data based on the spatial properties of the data.
 +
 +
=== Now you try! ===
 +
 +
'''Load''' the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)
 +
 +
Hydrology_DWAFRivers.shp -> call the table 'rivers'
 +
Transport_Mainroads.shp -> call the table 'roads'
 +
ProvincialBoundary.shp -> call the table 'provinces'
 +
 +
And to show that this can scale a bit, let us load in the background a dataset of over 350 000 points (which we will use a bit later)
 +
* cd /home/workshop/gisdata/za/fire/modis_af/
 +
* shp2pgsql -I -s 4326 modis_af.shp fires | psql -d workshop  &> /dev/null (puts output into background)
 +
* shp2pgsql -I -s -D 4326 modis_af.shp fires | psql -d workshop (simpler and faster)
 +
 +
=== Super duper advanced ===
 +
 +
What if I want to load a batch of shapefiles in a directory in 1 go? Try this nifty bash script one liner!
 +
 +
for FILE in `ls *.shp`; do TABLE=`echo $FILE | sed 's/\.shp//g'`; echo $TABLE;shp2pgsql -I -s 4326 $FILE $TABLE | psql -d workshop; done
 +
 +
 +
This takes a while so lets move on.
 +
 +
=== What have I learnt? ===
 +
 +
* PostgreSQL is an enterprise-ready relational database management system - and it's Free and Open Source Software
 +
* You can create a new database using the '''createdb''' command.
 +
* '''PostGIS''' is a add-on to PostgreSQL that lets you store spatial data and carry out spatial analysis within the database.
 +
* In order to set up PostGIS you need to run a couple of SQL scripts
 +
* '''shp2pgsql''' is a command line application that lets you load shapefiles into your spatial datastore
 +
 +
== Using PostGIS data from QGIS ==
 +
 +
 +
=== First we show you ===
 +
 +
Once your data is in PostGIS, its easy to share across your organisation. Because it's in a database you get row level locking (instead of file locking in a shapefile), so many users can be editing and viewing data at the same time. There are many clients that let you visualise your PostGIS data..lets take a look at how we do it in QGIS.
 +
 +
* Open QGIS
 +
* Start a new project
 +
  File -> New Project
 +
* Add a PostGIS layer
 +
  Layer -> Add PostGIS Layer
 +
* From the connections list, choose '''workshop'''
 +
  Click connect
 +
* In the table list, click on '''dams'''
 +
  Click Add
 +
 +
=== Now you try! ===
 +
 +
* Add the following layers from your database
 +
rivers
 +
roads
 +
provinces
 +
 +
* set some nice colors for your layers - notice how the process is exactly the same as when setting colours for a shapefile.
 +
 +
* save your project into
 +
 +
/home/workshop/gisdata/brits_pg.qgs
 +
 +
=== What have I learnt? ===
 +
 +
* You can '''visualise''' PostGIS spatial database tables inside QGIS
 +
* To add a PostGIS layer, use the '''Layer -> Add PostGIS Layer''' menu option.
 +
* Once a PostGIS layer, you can treat it just like any other (e.g. shapefile) layer
 +
* Your presenters are truly amazing....however would I have figured this stuff out for my self :-)
 +
 +
== More on PostGIS Data in QGIS ==
 +
 +
=== First we show you ===
 +
* Click on the '''Blue Elephant''' icon in the middle toolbar row to start SPIT loader.
 +
* Select the PostgreSQL Connection as "workshop".
 +
* Click on Connect to verify your connection
 +
* Now click "Add" - the "Add Shapefile" dialog box comes up.
 +
 +
Navigate to /home/workshop/gisdata/za/brits/SAExplorer.
 +
 +
* Select "Rsatowns.shp"
 +
 +
Click "Open".  PostGIS scans the file and lists it in the display frame.
 +
 +
Click "OK".  QGIS loads the shapefile data into PostGIS with no need to use the terminal!
 +
 +
* Use Layer -> Add PostGIS Layer to load the layer from PostGIS directly into QGIS. 
 +
 +
* Now add the rivers PostGIS layer to QGIS to see another advantage of using PostGIS layers.
 +
* Open the Layer properties form. 
 +
* On the General tab, click "Query Builder" near the bottom right-hand corner. This opens the Query Builder.
 +
* Click Field "RIVER_CLAS".
 +
* Click the "All" button under the "Values" frame.
 +
* Build the Query
 +
"RIVER_CLAS" LIKE 'Perennial' OR "RIVER_CLAS" LIKE 'Perennial river'
 +
* Click "Test" to check that it will work.
 +
* Click "OK" twice to apply it.
 +
 +
Now look at the display - the query has not merely highlighted the selected features, it has filtered the non-selected features out so that they are not displayed. 
 +
 +
=== Now you try ===
 +
 +
* Try loading other layers into PostGIS using the Spit loader.
 +
* Load these into QGIS, and use the "Query Builder" to define the data sets that you want to see.
 +
* You could define the non-perennial rivers as a separate data set, and use scale-dependent rendering to show them only when you zoom right in.
 +
 +
=== What I have learnt ===
 +
 +
How to rapidly load data into postgis using the spit loader.  How to set up data queries to filter spatial data display from postgis.
 +
 +
== The Zen of PostGIS - not just a pretty place to store data! ==
 +
 +
Okay, we will go a bit deeper into the woods here! We want to show that with some SQL (I won't say simple) you can do some complex - and repeatable - spatial analysis right inside your database, and then visualise the results in various clients.
 +
 +
=== First we show you ===
 +
 +
We will use pgAdmin (your great PostgreSQL friend) to get this happening. Click on the Tables object of your workshop db. Then Tools -> Query tool from the menu. This should present a dialog box with 3 panels.
 +
 +
We are going to do a simple intersection to get started.
 +
In the top left panel paste
 +
SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE (roads.the_geom && rivers.the_geom AND intersects(roads.the_geom, rivers.the_geom));
 +
and click Query -> Execute
 +
This should take a few seconds, then you will see a result returned into the Data Output grid. Something like 3338 rows are returned from the original roads dataset of 11964.
 +
 +
So what is going on here?
 +
 +
You are performing a spatial overlay with a SQL query, asking the database to return records from the roads table that spatially intersect records from the rivers table.
 +
* the && operator is a fast bounding box 'overlaps' test that takes advantage of the spatial indexes of the table to narrow the search space for the true intersection test.
 +
* the intersects part of the query tests the geometries of the subset selected above for spatial intersection and returns a result.
 +
 +
To show some other functionality, paste the following text into the query window and execute:
 +
ALTER TABLE roads DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'roads';COMMIT; UPDATE roads SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE roads ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
 +
We are being good South African GIS types and using our Haartebeesthoek94 Datum. Lets do the same for rivers:
 +
ALTER TABLE rivers DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'rivers';COMMIT; UPDATE rivers SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE rivers ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
 +
 +
This looks fairly intense, but the main points are the changing of the data in the geometry_columns meta-table and then updating the geometries in each table to a different CRS (4148, or Haartebeesthoek94). We demonstrate the use of the UPDATE SQL command, used to change data in the database, and the use of the ST_Transform function of PostGIS, which does the work. Now lets try the original query again:
 +
SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE st_intersects(roads.the_geom, rivers.the_geom);
 +
...which returns the same number of features, but in our own CRS. Note that we have used the ST_Intersects function rather than the Intersects function as previously shown. ST_ type functions are more standards compliant, so should be used in place of the older style functions.
 +
Ideally we would project the data as appropriate, using the very same technique, but substituting 4148 for the EPSG code for our projection of choice. This is a demo though, folks, so I will get away with murder...
 +
Lets quickly repeat the process for the other datasets we will use in a minute:
 +
ALTER TABLE provinces DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'provinces';COMMIT; UPDATE provinces SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE provinces ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
 +
ALTER TABLE fires DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'fires';COMMIT; UPDATE fires SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE fires ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
 +
 +
So, while that goes ahead (it will take a minute or so to transform 350k points), let us think out another analysis. Lets make it a bit tricky. Okay, we want to create a table for serving up data to the outside world (Gavin will do this later) that has all the intense fires in Gauteng or NorthWest Province, that have occurred within 3000 metres of a main road. There are many ways to skin this cat in PostGIS, so I will pick a method and explain why I did so.
 +
First, a convenience table is created:
 +
SELECT roads.gid, ST_Buffer(roads.the_geom,0.027) AS geom INTO roads_buffer FROM roads, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov WHERE ST_Within(roads.the_geom, prov.the_geom);
 +
This is a table of polygons representing ~3000 metre road buffers, and note the use of the function to populate the geometry_columns meta-table with our new layer.
 +
SELECT DISTINCT fires.gid, fires.frp, fires.acqdatetim, fires.the_geom INTO fires_analysis_3000m FROM fires, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov, roads_buffer  WHERE fires.frp > 200 AND ST_Within(fires.the_geom, prov.the_geom) AND ST_Within(fires.the_geom, roads_buffer.geom);
 +
The main feature of this query is the longish WHERE clause, which tests weather the fire is intense, whether it falls within Gauteng or NW Province and whether the fire occurred within appx 3000 metres of one of our main roads.
 +
 +
There are not too many fires that fall out of this analysis, but it shows how we have relatively easily worked with and reduced a large dataset to a few points. These queries can be further tuned to take advantage of spatial and other indexes, but that is perhaps for another time.
 +
 +
The final thing we will do is prepare this layer for use on the web...
 +
INSERT INTO geometry_columns(f_table_catalog,f_table_schema,f_table_name,f_geometry_column,coord_dimension,srid,"type") VALUES('','public','fires_analysis_3000m','the_geom',2,4148,'MULTIPOINT');
 +
Normal meta-table stuff
 +
ALTER TABLE fires_analysis_3000m ADD PRIMARY KEY (gid);
 +
And if we wanted to we could add a spatial index:
 +
CREATE INDEX fires_Analysis_3000m_the_geom_gist ON fires_Analysis_3000m USING gist(the_geom);
 +
 +
=== Now you try! ===
 +
 +
Well, no you do not - but you can ask questions...
 +
 +
=== What have I learned? ===
 +
 +
Well, hopefully a fair bit. You can, with a bit of practice and reading up on the matter, get PostGIS to do some rather interesting and powerful things. We have just seen a database do GIS... Some of you may be in the know about such matters, but hopefully this shows some possibilities to others.
 +
 +
== uDig ==
 +
 +
 +
=== First we show you ===
 +
 +
uDig ('user-friendly desktop internet GIS') is another excellent desktop GIS. let's check out the basics
 +
 +
* Open uDig
 +
* start adding layers (note the variety of data that can be used)
 +
** Add a PostGIS layer
 +
** add a shapefile
 +
* theme the layers
 +
* check out the table view, catalogue and some functionality
 +
 +
There's no need to save projects - you can just close uDig and it will save automatically. Note
 +
 +
=== Now you try! ===
 +
 +
* Add the following layers from your database
 +
rivers, dams, schools, roads, more if you like
 +
 +
* theme the school layer with nice symbols and labels - we're going to try scale-dependent rendering
 +
**zoom in closer than 1:50000 and set the max scale to 50000
 +
**on the theme dialog, click XML
 +
**select the <rule></rule> section and copy it to a text editor.
 +
**zoom out beyond 50000, go back to 'theme' and simplify your symbols, removing the labels.
 +
**set the min scale to 50000
 +
**click 'XML'
 +
**paste the previous 'rule' into the XML, before or after the one you see.
 +
**click OK! now zoom in and out past to see the result.
 +
 +
=== What have I learned? ===
 +
 +
* Viewing and rendering different formats in uDig
 +
* Theming with SLD (Styled layer descriptor), the OGC standard.
 +
* scale-dependent rendering
 +
** with a point - we're going to come back to uDig!
 +
 +
== GeoServer ==
 +
 +
=== First we show you ===
 +
 +
cd
 +
export JAVA_HOME=/usr
 +
cd Apps/geoserver
 +
bin/startup.sh
 +
 +
* Start geoserver (normally set to start automatically and run all the time)
 +
* All admin is done via the web
 +
* set up a namespace
 +
* set up data stores
 +
* set up feature types
 +
 +
=== Now you try! ===
 +
 +
* start GeoServer
 +
* click Config -> Data
 +
* Set Namespace to any name and any url
 +
* Set Datastore to your postGIS database
 +
* define a FeatureType for schools, choose a point style
 +
'''Styling'''
 +
*Go to Data -> Style -> new
 +
**give your style a name and paste in your school SLD from uDig
 +
**submit
 +
**Go back to FeatureTypes and Edit your schools Feature, applying your new style
 +
**Check out the results in the GeoServer demo pages
 +
 +
'''Group layers'''
 +
* Set up a group in the 'WMS' section
 +
 +
 +
===Back to uDig
 +
*bring in a WMS layer from http://localhost/geoserver:8080/wms
 +
*bring in the same layer as WFS
 +
**theme it
 +
**save it as a shapefile
 +
*try this with QGIS too (and with ArcGIS back at the office!)
 +
 +
=== What have I learned? ===
 +
 +
* you can use GeoServer to serve your spatial data from any source as WMS, WFS, KML and more!
 +
* You can use WMS and WFS layers in a desktop GIS
 +
* impress your colleagues by sending them kml links from your geoserver so they can view them live in Google Earth!
 +
 +
== More Advanced QGIS Use ==
 +
Here are some more things you can do with QGIS!  Don't be put off by the heading of this section, we'll take you through step-by-step as before..
 +
 +
=== ''Quick Qml with QGIS'' ===
 +
 +
=== First we show you ===
 +
 +
If you have used  ArcView before, you might have used the avl feature which lets you store the symbology settings for a layer. QGIS has a similar feature called 'qml' which lets you create default symbology settings for a layer, that will be applied when ever that layer is loaded. We are also going to take a quick peek at how to create more complex symbology for a layer.
 +
 +
 +
* Open QGIS if its not already running.
 +
* Add the dams layer from the Postgresql 'Workshop' database
 +
* Double click on the layer entry in the legend
 +
* Activate the symbology tab
 +
* Now change the legend type to '''Graduated Symbol'''
 +
* Change the classification field to '''shape_area'''
 +
* Set the '''number of classes''' to 5
 +
* Click the '''classify''' button
 +
* Click the '''Apply''' button
 +
 +
You should see the shapes in the main window have now adopted a range of scintillating colours! However is you add that layer to another QGIS project you would normally have to do all this work again! Qml to the rescue.
 +
 +
* Save your symbology settings by clicking the '''save as default''' button
 +
 +
Ok lets test it out!
 +
 +
* Close QGIS
 +
* Reopen QGIS
 +
* Add the Dams layer from the Postgresql 'Workshop' database
 +
 +
If the planets are correctly aligned, your dams will appear all beautifully coloured according to your personal preference.
 +
 +
If you want to you can also save your settings as non-default, and them restore them by selecting a .qml file again later. Refer to the '''Load style''' and '''Save style''' buttons on the vector properties dialog.
 +
 +
=== Now you try! ===
 +
 +
Ok give it a whirl, create default appearances for your other layers stored in your PostGIS database. Also try experimenting with the other legend types such as '''Continuous Colour''' and '''Unique Value'''.
 +
Once you have set your default appearances, close QGIS and then reopen it. Now add all your PostGIS layers to the blank project and they should all magically appear with your preferred appearance.
 +
 +
=== What have I learned? ===
 +
 +
The '''Layer Properties''' dialog in QGIS will let you specify the '''default appearance''' for a layer.
 +
 +
=== ''Setting Projections'' ===
 +
Projections are a mathematical way of transforming latitude and longitude coordinates into a form that can be meaningfully used in a map.  Sources to be added.
 +
Some projections in common use in South Africa are the Albers Equal Area, Lamberts Conformal Conic and Gauss (LO) projections.
 +
 +
=== First we show you ===
 +
* Make sure that you are zoomed in to Gauteng.
 +
* From the File menu, select Project Properties.
 +
  File -> Project Properties
 +
* Activate the Coordinate Reference System tab
 +
* Check "Enable "on the fly" CRS Transformation
 +
* Note that the current Coordinate Reference System is WGS84, which is a standard for displaying geographic coordinates.
 +
* Look at the column EPSG - this a standard for identifying coordinate systems.  When you have selected a coordinate system you can always find it again quickly by noting the EPSG code and entering it into the '''Find box''' at the bottom of the form.
 +
* Highlight the ''Projected Coordinate Systems" heading, and double-click to open the list.
 +
* Scroll down and highlight the "Universal Transverse Mercator" heading.
 +
* Select the '''Name''' radio button.
 +
* Enter "33S" in the text box, and press find.  Keep pressing find until you get to "WGS 84/UTM Zone 33". 
 +
Or select the EPSG ID radio button, enter 32733 and press the Find button.    Isn't that easier?
 +
* Notice that the overall shape of the country has changed due to the projection.
 +
* Zoom into Gauteng again.  Move your cursor and look at the coordinates shown at the bottom of the screen.  Notice how they have changed.  They are now in meters rather than decimal degrees.
 +
 +
=== Now You Try! ===
 +
First of all try a related projection such as UTM Zone 34S or UTM 38S, and see what the impact is.  Move your cursor around - how do the coordinates differ from ones shown under UTM 36S?
 +
 +
Then try one of the Stereographic South projections, such as EPSG=3031,  for a very different view of South Africa!
 +
 +
QGIS calculates the effect of the projection before displaying the data and doesn't make any changes, so you can try anything you like!
 +
 +
=== What have I learned? ===
 +
You have learnt how to set an on-the-fly projection for display in QGIS, and how to use EPSG codes to rapidly find a specific projection. Remember, if you are trying to develop a map for use in the field you will need to use a projection specific to South Africa.
 +
 +
=== ''Editing Vector Data'' ===
 +
Have a look at the boundary of the Eastern and Western Cape on the coast.  The rivers and provincial boundary seem to follow each other closely. Now zoom right in and notice how the provincial boundary is much more detailed than the rivers, and they don't follow each other well at all.
 +
 +
QGIS to the rescue!
 +
 +
=== First we show you ===
 +
* Select the rivers layer in the Legend panel on the left.
 +
* Click on the blue Pencil icon (Toggle Editing) on the leftmost toolbar on the bottom row or <menu option>. 
 +
* Now click on the <edit vertices> icon. Notice how the river lines have suddenly sprouted small circles wherever the line changes direction.  These are the vertices - if you have used a drawing program before you might have seen something like this.
 +
* Drag the vertices of the rivers to align with the provincial boundary.
 +
Notice that we don't have enough vertices to do this exactly.
 +
* Click on the add vertices icon.  Now click on the river wherever more vertices are needed. 
 +
* When you have enough vertices go back to "Edit Vertices" and move them where they need to be.
 +
* Now save your changes using by clicking on "Toggle Editing" and selecting "Save" on the dialog box.
 +
 +
=== Now You Try! ===
 +
Go to Gauteng and bring up the JPEG map sheets behind the roads.  Have a look at the roads and notice where the vectors are not aligned with the base map.  Use the skills you have used to correct the roads vectors.  Try other tools like Split Features, Move Features and other to see what they do. 
 +
 +
Then look at the rivers and provincial boundaries again.  See if you can find another place where the two are not aligned.  Experiment with the Snapping Options under Project Properties.  Use the Measure Length tool to decide on reasonable initial tolerance values.
 +
 +
=== What have I learned? ===
 +
You have now learned how to edit vector data in QGIS using the edit toolbar.  Editing data may not be as immediately rewarding as setting up a map onscreen or for printing, but it is important to be able to do this to get rid of errors that will spoil all the effort put into drawing up an impressive map!
  
[[Category:Local Chapters]]
 
 
[[Category:Africa]]
 
[[Category:Africa]]

Latest revision as of 02:07, 1 December 2008

...back to Africa_Local_Chapter

What?

Our first OSGeo local chapter meet up! We will chew the cud, show you some cool stuff and start to get people familiar with the brilliant FOSS GeoSpatial products out there!

We'll go through installing some software and getting it to a point where you can have fun with it.

How Much?

There will be a door charge of R50 to cover costs & refreshments. That and an open mind are all you need to bring. If you're staying for the QGIS session in the afternoon you can bring your laptop (Windows, Linux or Mac!)

Where?

St Davids Marist Inanda College
36 Rivonia Rd
Inanda
Sandton

Map and Directions

NB: We'll be in the Prep School computer lab.

When?

  • November 29th
  • Presentation and workshop segment: 9am to 12pm
  • QGIS Bug Party! 12h30 onwards

Who

Your friendly presenters for the day will be:

  • Tim Sutton (tim at linfiniti.com) - QGIS hacker and FOSS GIS junkie extraordinaire. Tim eats Weetbix for breakfast because it sounds like Linux. He once rewrote Nkosi Sikelela in C++ because he was feeling patriotic.
  • Graeme McFerren - (graeme.mcferren at gmail.com) - GIS researcher and FOSS4G user. Mmm, aspire to be like Tim then.
  • Gavin Fleming (gavinjfleming at gmail dot com) - I enjoy my daily FOSS4G fix and telling people how great it was. I'm a 'geoinformation scientist' at Mintek and love to see cutting edge and useful geospatial applications emerging.
  • Brendon Wolff-Piggott - (brendon at integratedgeodata.co.za) - Water resources consultant and FOSS GIS enthusiast. I wish there were more hours in the day to keep up with FOSS4G!

RSVP

Let gavinjfleming at gmail dot com know if you're coming as numbers are limited.


QGIS Bug Party

As a preview release there are still various bugs that need resolving and various elements (e.g. translation) that may be incomplete. On Saturday the 29th in Johannesburg, South Africa, members from the Africa OSGeo chapter will be holding a QGIS Bug Hunt Party. We would like to invite people from around the world to virtually participate. Each person attending will get a free copy of QGIS 1.0.0 Preview 2 absolutely free of charge! Sorry no shrink wrap copies available. Preview 2 is the second in a series of preview releases we are making before we release the Stable QGIS 1.0.0 with long term support. We are going to great lengths to polish up QGIS 1.0 for you (inside and out) and we really appreciate all the feedback we have received thus far. Its virtually impossible to ship any software completely bug free, but with your help we can add just a little more shine to the polish! We need your help to:

  • test on different machine configurations (bring your laptops!)
  • validate existing bugs on each platform
  • mark bugs that can no longer be replicated as ready for closure
  • create new tickets for bugs that have not been reported yet (please search first to avoid posting duplicates!)
  • mark bugs that are duplicates
  • provide additional diagnostics so that developers can replicate issues

The idea is not to try to *fix* the bugs, but rather to do a QA session on the bugs in the bug tracker. In the process you will learn how to interact with an Open Source project the Right Way i.e. filing good quality bug reports.

I will start off with a simple show and tell covering:

  • where to find the QGIS (and other OSGeo projects bug tracking sites)
  • creating an account so you can file non-anonymous bugs and feedback on any OSGeo project bug tracker
  • how the bug tracking system works - what the various input fields in the QGIS tracker mean

The approach will be to start by going through any bugs that have not been revised recently, adding additional status information. If you think the bug is fixed, add a comment:

Works for me now, recommend closure, <Your Name>

If you can replicate the bug, add a comment like:

Can replicate, OS: Ubuntu 8.10, QGIS Preview 2, <Your Name>

Use your initiative to deal with tickets that don't fit into the categories above, following a similar approach.


After that I will turn you loose on the queue and a shiny copy of QGIS preview 2 and let you have at it. Party is over when the last man falls asleep at the keyboard! Ok probably we will finish at between 4:30pm and 5pm if my wife has anything to do with it!


We look forward to seeing you at the QGIS Bug Party!!!


Programme in a nutshell

  • Be there in time for a 9am start
  • Morning session: Introducing QGIS, PostGIS, GeoServer, uDig
  • Afternoon session: QGIS Bug Party

Note: There will be frequent rest breaks and lots of time to ask questions!

1. Welcome, Introduction and Overview: Starting up (5 minutes)

What am I looking at here? What do I need to do to get started?

2. QGIS: Opening and viewing spatial data. Your first map. (15 minutes)

What can I look at? How do I set up a simple map? Toolbars and Plug-ins.

3. PostGIS: Why use PostGIS? Simple example of loading data. (15 minutes)

Why use a spatial database? What does one look like? How do I populate a spatial database and where does my data live?

4. QGIS: Looking at PostGIS data from QGIS. (15 minutes)

View a PostGIS database from QGIS. Setting up data definitions for PostGIS sources. Loading data into PostGIS with the Spit Loader (no, please don't touch the screen!)

5. PostGIS: More than a place to dump data... (15 minutes)

PostGIS is not just good for storing data, one can manipulate it too! GIS can be done inside the database, which is rather nice when you think about it.

6. uDig: Why uDig? GIS and the Web. (15 minutes)

Getting GIS data ready for presentation on the Web! Editing SLDs.

7. GeoServer: Tell everyone about it! (15 minutes)

Displaying GIS data on the web. Grouped WMS. Scale-dependent rendering.

8. And more QGIS.

Detail on symbology. Topological editing. QML, SPD and Openlayers. (15 minutes)

9. Show off session

Here we can do some whizz bang demo stuff that we don't have time to do in workshop format but that will give you some hints as to what else can be achieved.

10. General

Questions and answers. What you can do next. Resources. Possibly some in-depth examples based on frequent questions. What would you like to see at future workshops?

11. Your assessment of the workshop

Tell us what you thought!

Workshop Notes

Welcome, introductions and overview

Lets make a map - First steps with QGIS

First we show you

  • Open QGIS using the icon on the top Gnome Panel
  • From the Layer menu, select Add Vector Layer
 Layer -> Add Vector Layer
  • A dialog box will appear
  • Navigate to
 /home/workshop/gisdata/za/brits/SAExplorer/
  • Add the following layer :
 Provinces.shp
  • Experiment with the map navigation toolbar to zoom in / out, pan, select etc features.
  • Open the Layer properties dialog. There are several ways you can do this:
 1) Select the layer in the legend and then from the menu do Layer -> Properties
 or
 2) Right click on the layer in the legend and choose Properties from the context menu
 or
 3) Double click on the layer in the legend
  • Familiarise yourself with the various panels in the vector properties dialog.
  • Select the Symbology tab (second from the left)
  • From the Style Options panel, choose a pleasing fill colour for the provinces then
 Click OK

Or you can:

  • Select the Legend Type as Unique Value.
  • Set the Classification field to PROVNAME
  • Click on the Classify button to set different colours for each province.
 Click OK

Now you try!

Add the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)

 Transport_NationalRoads.shp
 Transport_Railways.shp
 RSATowns.shp
 Dams.shp

Zoom and pan around a little to explore your data Symbolise each layer with an appropriate colour to make the most awe inspiring map.

Activate the Identify features icon on the middle toolbar to identify the attribute values associated with a spatial feature for the layer highlighted in the Legend Box.

Activate the Select Features icon next to Identify Features to select one or more features.

Open the attribute table to view all feature attributes. Note that selected features will have highlighted rows in the attribute table. You can do this using the neighbouring Open Attribute Table icon, or by selecting the Open Attribute Table option on the context menu for the layer in the legend using right-click on the mouse.

You can also select rows in the attribute table by:

  • Left-clicking on the desired rows.
  • Using the Search box at the bottom-left of the form.
  • Selecting the Advanced button and opening the Search Query Builder.

Save your project using

File -> Save

And put it into

/home/workshop/gisdata/brits.qgs

What have I learned?

In ten minutes you now know how to add vector layers into QGIS, pan and zoom to navigate the map, and set the colour of vectors using the Single Symbol or Unique Values options in the Layer Properties.

You have also learnt how to use the interactive Identify and Select functions, opened attribute tables and found out how to search them.

More on QGIS Mapping

First we show you

Add the vector layer Education_Schools2000.shp from the SAExplorer directory. Notice how dense the points are!

  • Right-click on the schools layer in the Legend frame.
  • Select Properties
  • Select the General tab
  • Check Use scale dependent rendering
  • Set the maximum field to 1000000
  • Zoom out to view the whole country. Notice that the schools no longer appear on the map!
  • Now start zooming in, keeping an eye on the scale shown in the scale box at the bottom of the screen.
Notice how the schools points reappear when the scale decreases below 1:1000000

  • From the Layer menu, select Add Raster Layer
 Layer -> Add Raster Layer
  • A dialog box will appear
  • Navigate to
 /home/workshop/gisdata/za/gauteng/
  • Add the following layer :
SOU-8_satellite.tif

Notice that the image is largely in shades of greyish-brown, but that a dam can be seen towards the bottom of the image.

  • Open the Properties for the raster layer, in a similar way to that used for vector layers.
Layer -> Properties
  • Select the Transparency tab.
  • Using the slider, set transparency to 25%
  • Click OK

Notice how the you can now see the vector layers behind the satellite image.

Now you try!

If you have not already done so, add the vector layers Transport_NationalRoads.shp and Transport_Mainroads.shp from /home/workshop/gisdata/za/brits/SAExplorer. Set scale-dependent rendering so that the national roads display at all scales, while the main roads are only shown when you are zoomed in.

Go to the satellite image properties, Symbology tab. Try the different "Render as" options. Then when you have found the one that suits you, have a look at the "Contast Enhancement" options. Compare "No Stretch" and "Stretch to MinMax".

Try "Render as Single Band Grey". Set the "Custom min/max values" radio button. Change the Min and Max values. Go to "Contract Enhancement" again and set it to "Stretch and Clip to MinMax", and see the effect.

Add raster images of 1: 50 000 map sheets from " /home/workshop/gisdata/za/brits/toposheet images" You will need to set the Files of Type box to JPEG JFIF to see them, before you can select them. Set their transparency to a value that allows you to see the vector features behind them. Zoom in and look at how the vector features and image features match (or not!).

Notice how you can zoom right in to vector data, and a point remains a point and a line remains a line. When you zoom in to the satellite image it rapidly appears "blocky" compared to the 50 000 map sheets. Zoom in enough and even the map sheet raster blocks can be seen. This is a basic difference between vector and raster data.

What have I learned?

You have learnt to set scale-dependent rendering so that data sets are displayed at appropriate scales, how to add raster data and set raster display options.

Shapefiles shmapefiles....real (wo)men store their data in PostGIS

First we show you

  • Create a new PostgreSQL database. To do this, open the terminal application (icon to the right of Firefox icon on the top menu bar). Now at the prompt type the following command:
 createdb workshop
  • Verify your database exists (once again type this into the terminal)
 psql -l
  • You should see something like this:
workshop@heron:~$ psql -l
       List of databases
  Name    |  Owner   | Encoding 
-----------+----------+----------
postgres  | postgres | UTF8
template0 | postgres | UTF8
template1 | postgres | UTF8
workshop  | workshop | UTF8
(4 rows)
  • away from the terminal for a while - lets look at what a non-spatial, vanilla PostgreSQL database looks like. Open pgAdmin 3 (the icon with a blue elephant face)
  • Double-click on the icon that has a red cross next to some text "gis (localhost:5432)" and press OK if another screen comes up. The database cluster called gis expands and you should see two databases: 'postgres' and 'workshop'. Expand workshop -> Schemas -> public

Specifically, let us look at Functions and Tables - don't look to deeply, this is just to show the differences after PostGIS is installed on the database (you will see later). Specifically, there is nothing in the db at the moment. So minimise pgAdmin and lets crack on with getting some data in!

  • Install the PostGIS spatial data extensions (again from the terminal).

Hint copy and paste these from the IRC channel or the wiki page!

createlang plpgsql workshop
psql workshop < /usr/share/postgresql-8.3-postgis/lwpostgis.sql
psql workshop < /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql
  • Yegads! What did I just do? Firstly, you installed the plpgsql procedural language extensions into your PostgreSQL database. Then you ran two sql scripts which create the PostGIS spatial extensions to your database. You only need to do this once when you create a new spatial database, so don't worry about remembering those commands off by heart.
  • Lets upload a shapefile into the database....once more into the black hole (terminal window).
cd /home/workshop/gisdata/za/brits/SAExplorer
shp2pgsql -I -s 4326 Dams.shp dams | psql -d workshop
  • Erk! What does that all mean? Lets break it down:
cd /home/workshop/gisdata/za/brits/SAExplorer - go to our data directory
shp2pgsql - an application that loads shapefiles into a PostGIS database
-I - an option to shp2pgsql telling it to create a spatial index
-s - the coordinate reference system to use, expressed as an EPSG number. 4326 is the same as Lat/Long WGS84
Dams.shp - the filename of the shapefile to load
dams - the table name that the data should be loaded into
| psql workshop - send the results of the shp2pgsql into the psql application, workshop database


  • Lets take a peek inside our database now and see what we have - back to pgAdmin, click on the 'workshop' database icon and then refresh it (red/green circular arrows icon).

Three major things to look at:

Hundreds of Functions for working with spatial data: 
* geometry constructors, 
* spatial predicates, 
* geometry editors, 
* geometry aggregators, 
* geometry accessors, 
* measurements, 
* linear referencing
* and outputting  of spatial data. 

2 important meta-tables: (expand the Tables object)
* geometry_columns (for describing the different spatial datasets) - right-click -> 
* and spatial_ref_sys(a list of projections/coordinate systems and different representations thereof)
A 'dams' table which, when expanded should show a bunch of Columns, Constraints (e.g. Primary Key), Indexes, Rules and Triggers. 
* Note the geometry type of column 'the_geom'. 
* Also note the spatial index present - this allows for optimal retrieval of data based on the spatial properties of the data.

Now you try!

Load the following layers (also from the /home/workshop/gisdata/za/brits/SAExplorer/ directory)

Hydrology_DWAFRivers.shp -> call the table 'rivers'
Transport_Mainroads.shp -> call the table 'roads'
ProvincialBoundary.shp -> call the table 'provinces'

And to show that this can scale a bit, let us load in the background a dataset of over 350 000 points (which we will use a bit later)

* cd /home/workshop/gisdata/za/fire/modis_af/
* shp2pgsql -I -s 4326 modis_af.shp fires | psql -d workshop  &> /dev/null (puts output into background)
* shp2pgsql -I -s -D 4326 modis_af.shp fires | psql -d workshop (simpler and faster)

Super duper advanced

What if I want to load a batch of shapefiles in a directory in 1 go? Try this nifty bash script one liner!

for FILE in `ls *.shp`; do TABLE=`echo $FILE | sed 's/\.shp//g'`; echo $TABLE;shp2pgsql -I -s 4326 $FILE $TABLE | psql -d workshop; done


This takes a while so lets move on.

What have I learnt?

  • PostgreSQL is an enterprise-ready relational database management system - and it's Free and Open Source Software
  • You can create a new database using the createdb command.
  • PostGIS is a add-on to PostgreSQL that lets you store spatial data and carry out spatial analysis within the database.
  • In order to set up PostGIS you need to run a couple of SQL scripts
  • shp2pgsql is a command line application that lets you load shapefiles into your spatial datastore

Using PostGIS data from QGIS

First we show you

Once your data is in PostGIS, its easy to share across your organisation. Because it's in a database you get row level locking (instead of file locking in a shapefile), so many users can be editing and viewing data at the same time. There are many clients that let you visualise your PostGIS data..lets take a look at how we do it in QGIS.

  • Open QGIS
  • Start a new project
 File -> New Project
  • Add a PostGIS layer
 Layer -> Add PostGIS Layer
  • From the connections list, choose workshop
 Click connect
  • In the table list, click on dams
 Click Add

Now you try!

  • Add the following layers from your database
rivers
roads
provinces
  • set some nice colors for your layers - notice how the process is exactly the same as when setting colours for a shapefile.
  • save your project into
/home/workshop/gisdata/brits_pg.qgs

What have I learnt?

  • You can visualise PostGIS spatial database tables inside QGIS
  • To add a PostGIS layer, use the Layer -> Add PostGIS Layer menu option.
  • Once a PostGIS layer, you can treat it just like any other (e.g. shapefile) layer
  • Your presenters are truly amazing....however would I have figured this stuff out for my self :-)

More on PostGIS Data in QGIS

First we show you

  • Click on the Blue Elephant icon in the middle toolbar row to start SPIT loader.
  • Select the PostgreSQL Connection as "workshop".
  • Click on Connect to verify your connection
  • Now click "Add" - the "Add Shapefile" dialog box comes up.

Navigate to /home/workshop/gisdata/za/brits/SAExplorer.

  • Select "Rsatowns.shp"
Click "Open".  PostGIS scans the file and lists it in the display frame.
Click "OK".  QGIS loads the shapefile data into PostGIS with no need to use the terminal!
  • Use Layer -> Add PostGIS Layer to load the layer from PostGIS directly into QGIS.
  • Now add the rivers PostGIS layer to QGIS to see another advantage of using PostGIS layers.
  • Open the Layer properties form.
  • On the General tab, click "Query Builder" near the bottom right-hand corner. This opens the Query Builder.
  • Click Field "RIVER_CLAS".
  • Click the "All" button under the "Values" frame.
  • Build the Query
"RIVER_CLAS" LIKE 'Perennial' OR "RIVER_CLAS" LIKE 'Perennial river'
  • Click "Test" to check that it will work.
  • Click "OK" twice to apply it.

Now look at the display - the query has not merely highlighted the selected features, it has filtered the non-selected features out so that they are not displayed.

Now you try

  • Try loading other layers into PostGIS using the Spit loader.
  • Load these into QGIS, and use the "Query Builder" to define the data sets that you want to see.
  • You could define the non-perennial rivers as a separate data set, and use scale-dependent rendering to show them only when you zoom right in.

What I have learnt

How to rapidly load data into postgis using the spit loader. How to set up data queries to filter spatial data display from postgis.

The Zen of PostGIS - not just a pretty place to store data!

Okay, we will go a bit deeper into the woods here! We want to show that with some SQL (I won't say simple) you can do some complex - and repeatable - spatial analysis right inside your database, and then visualise the results in various clients.

First we show you

We will use pgAdmin (your great PostgreSQL friend) to get this happening. Click on the Tables object of your workshop db. Then Tools -> Query tool from the menu. This should present a dialog box with 3 panels.

We are going to do a simple intersection to get started. In the top left panel paste

SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE (roads.the_geom && rivers.the_geom AND intersects(roads.the_geom, rivers.the_geom));

and click Query -> Execute This should take a few seconds, then you will see a result returned into the Data Output grid. Something like 3338 rows are returned from the original roads dataset of 11964.

So what is going on here?

You are performing a spatial overlay with a SQL query, asking the database to return records from the roads table that spatially intersect records from the rivers table.

  • the && operator is a fast bounding box 'overlaps' test that takes advantage of the spatial indexes of the table to narrow the search space for the true intersection test.
  • the intersects part of the query tests the geometries of the subset selected above for spatial intersection and returns a result.

To show some other functionality, paste the following text into the query window and execute:

ALTER TABLE roads DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'roads';COMMIT; UPDATE roads SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE roads ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

We are being good South African GIS types and using our Haartebeesthoek94 Datum. Lets do the same for rivers:

ALTER TABLE rivers DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'rivers';COMMIT; UPDATE rivers SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE rivers ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

This looks fairly intense, but the main points are the changing of the data in the geometry_columns meta-table and then updating the geometries in each table to a different CRS (4148, or Haartebeesthoek94). We demonstrate the use of the UPDATE SQL command, used to change data in the database, and the use of the ST_Transform function of PostGIS, which does the work. Now lets try the original query again:

SELECT roads.gid, roads.code, roads.route_nr, roads.shape_leng, roads.the_geom FROM roads, rivers WHERE st_intersects(roads.the_geom, rivers.the_geom);

...which returns the same number of features, but in our own CRS. Note that we have used the ST_Intersects function rather than the Intersects function as previously shown. ST_ type functions are more standards compliant, so should be used in place of the older style functions. Ideally we would project the data as appropriate, using the very same technique, but substituting 4148 for the EPSG code for our projection of choice. This is a demo though, folks, so I will get away with murder... Lets quickly repeat the process for the other datasets we will use in a minute:

ALTER TABLE provinces DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'provinces';COMMIT; UPDATE provinces SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE provinces ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);
ALTER TABLE fires DROP CONSTRAINT enforce_srid_the_geom;UPDATE geometry_columns SET srid = 4148 WHERE f_table_name = 'fires';COMMIT; UPDATE fires SET the_geom = ST_Transform(the_geom,4148);COMMIT; ALTER TABLE fires ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 4148);

So, while that goes ahead (it will take a minute or so to transform 350k points), let us think out another analysis. Lets make it a bit tricky. Okay, we want to create a table for serving up data to the outside world (Gavin will do this later) that has all the intense fires in Gauteng or NorthWest Province, that have occurred within 3000 metres of a main road. There are many ways to skin this cat in PostGIS, so I will pick a method and explain why I did so. First, a convenience table is created:

SELECT roads.gid, ST_Buffer(roads.the_geom,0.027) AS geom INTO roads_buffer FROM roads, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov WHERE ST_Within(roads.the_geom, prov.the_geom);

This is a table of polygons representing ~3000 metre road buffers, and note the use of the function to populate the geometry_columns meta-table with our new layer.

SELECT DISTINCT fires.gid, fires.frp, fires.acqdatetim, fires.the_geom INTO fires_analysis_3000m FROM fires, (SELECT * FROM provinces WHERE prov_code = 'GT' OR prov_code = 'NW') AS prov, roads_buffer  WHERE fires.frp > 200 AND ST_Within(fires.the_geom, prov.the_geom) AND ST_Within(fires.the_geom, roads_buffer.geom);

The main feature of this query is the longish WHERE clause, which tests weather the fire is intense, whether it falls within Gauteng or NW Province and whether the fire occurred within appx 3000 metres of one of our main roads.

There are not too many fires that fall out of this analysis, but it shows how we have relatively easily worked with and reduced a large dataset to a few points. These queries can be further tuned to take advantage of spatial and other indexes, but that is perhaps for another time.

The final thing we will do is prepare this layer for use on the web...

INSERT INTO geometry_columns(f_table_catalog,f_table_schema,f_table_name,f_geometry_column,coord_dimension,srid,"type") VALUES(,'public','fires_analysis_3000m','the_geom',2,4148,'MULTIPOINT');

Normal meta-table stuff

ALTER TABLE fires_analysis_3000m ADD PRIMARY KEY (gid);

And if we wanted to we could add a spatial index:

CREATE INDEX fires_Analysis_3000m_the_geom_gist ON fires_Analysis_3000m USING gist(the_geom);

Now you try!

Well, no you do not - but you can ask questions...

What have I learned?

Well, hopefully a fair bit. You can, with a bit of practice and reading up on the matter, get PostGIS to do some rather interesting and powerful things. We have just seen a database do GIS... Some of you may be in the know about such matters, but hopefully this shows some possibilities to others.

uDig

First we show you

uDig ('user-friendly desktop internet GIS') is another excellent desktop GIS. let's check out the basics

  • Open uDig
  • start adding layers (note the variety of data that can be used)
    • Add a PostGIS layer
    • add a shapefile
  • theme the layers
  • check out the table view, catalogue and some functionality

There's no need to save projects - you can just close uDig and it will save automatically. Note

Now you try!

  • Add the following layers from your database
rivers, dams, schools, roads, more if you like
  • theme the school layer with nice symbols and labels - we're going to try scale-dependent rendering
    • zoom in closer than 1:50000 and set the max scale to 50000
    • on the theme dialog, click XML
    • select the <rule></rule> section and copy it to a text editor.
    • zoom out beyond 50000, go back to 'theme' and simplify your symbols, removing the labels.
    • set the min scale to 50000
    • click 'XML'
    • paste the previous 'rule' into the XML, before or after the one you see.
    • click OK! now zoom in and out past to see the result.

What have I learned?

  • Viewing and rendering different formats in uDig
  • Theming with SLD (Styled layer descriptor), the OGC standard.
  • scale-dependent rendering
    • with a point - we're going to come back to uDig!

GeoServer

First we show you

cd
export JAVA_HOME=/usr
cd Apps/geoserver
bin/startup.sh
  • Start geoserver (normally set to start automatically and run all the time)
  • All admin is done via the web
  • set up a namespace
  • set up data stores
  • set up feature types

Now you try!

  • start GeoServer
  • click Config -> Data
  • Set Namespace to any name and any url
  • Set Datastore to your postGIS database
  • define a FeatureType for schools, choose a point style

Styling

  • Go to Data -> Style -> new
    • give your style a name and paste in your school SLD from uDig
    • submit
    • Go back to FeatureTypes and Edit your schools Feature, applying your new style
    • Check out the results in the GeoServer demo pages

Group layers

  • Set up a group in the 'WMS' section


===Back to uDig

  • bring in a WMS layer from http://localhost/geoserver:8080/wms
  • bring in the same layer as WFS
    • theme it
    • save it as a shapefile
  • try this with QGIS too (and with ArcGIS back at the office!)

What have I learned?

  • you can use GeoServer to serve your spatial data from any source as WMS, WFS, KML and more!
  • You can use WMS and WFS layers in a desktop GIS
  • impress your colleagues by sending them kml links from your geoserver so they can view them live in Google Earth!

More Advanced QGIS Use

Here are some more things you can do with QGIS! Don't be put off by the heading of this section, we'll take you through step-by-step as before..

Quick Qml with QGIS

First we show you

If you have used ArcView before, you might have used the avl feature which lets you store the symbology settings for a layer. QGIS has a similar feature called 'qml' which lets you create default symbology settings for a layer, that will be applied when ever that layer is loaded. We are also going to take a quick peek at how to create more complex symbology for a layer.


  • Open QGIS if its not already running.
  • Add the dams layer from the Postgresql 'Workshop' database
  • Double click on the layer entry in the legend
  • Activate the symbology tab
  • Now change the legend type to Graduated Symbol
  • Change the classification field to shape_area
  • Set the number of classes to 5
  • Click the classify button
  • Click the Apply button

You should see the shapes in the main window have now adopted a range of scintillating colours! However is you add that layer to another QGIS project you would normally have to do all this work again! Qml to the rescue.

  • Save your symbology settings by clicking the save as default button

Ok lets test it out!

  • Close QGIS
  • Reopen QGIS
  • Add the Dams layer from the Postgresql 'Workshop' database

If the planets are correctly aligned, your dams will appear all beautifully coloured according to your personal preference.

If you want to you can also save your settings as non-default, and them restore them by selecting a .qml file again later. Refer to the Load style and Save style buttons on the vector properties dialog.

Now you try!

Ok give it a whirl, create default appearances for your other layers stored in your PostGIS database. Also try experimenting with the other legend types such as Continuous Colour and Unique Value. Once you have set your default appearances, close QGIS and then reopen it. Now add all your PostGIS layers to the blank project and they should all magically appear with your preferred appearance.

What have I learned?

The Layer Properties dialog in QGIS will let you specify the default appearance for a layer.

Setting Projections

Projections are a mathematical way of transforming latitude and longitude coordinates into a form that can be meaningfully used in a map. Sources to be added. Some projections in common use in South Africa are the Albers Equal Area, Lamberts Conformal Conic and Gauss (LO) projections.

First we show you

  • Make sure that you are zoomed in to Gauteng.
  • From the File menu, select Project Properties.
 File -> Project Properties
  • Activate the Coordinate Reference System tab
  • Check "Enable "on the fly" CRS Transformation
  • Note that the current Coordinate Reference System is WGS84, which is a standard for displaying geographic coordinates.
  • Look at the column EPSG - this a standard for identifying coordinate systems. When you have selected a coordinate system you can always find it again quickly by noting the EPSG code and entering it into the Find box at the bottom of the form.
  • Highlight the Projected Coordinate Systems" heading, and double-click to open the list.
  • Scroll down and highlight the "Universal Transverse Mercator" heading.
  • Select the Name radio button.
  • Enter "33S" in the text box, and press find. Keep pressing find until you get to "WGS 84/UTM Zone 33".
Or select the EPSG ID radio button, enter 32733 and press the Find button.    Isn't that easier?
  • Notice that the overall shape of the country has changed due to the projection.
  • Zoom into Gauteng again. Move your cursor and look at the coordinates shown at the bottom of the screen. Notice how they have changed. They are now in meters rather than decimal degrees.

Now You Try!

First of all try a related projection such as UTM Zone 34S or UTM 38S, and see what the impact is. Move your cursor around - how do the coordinates differ from ones shown under UTM 36S?

Then try one of the Stereographic South projections, such as EPSG=3031, for a very different view of South Africa!

QGIS calculates the effect of the projection before displaying the data and doesn't make any changes, so you can try anything you like!

What have I learned?

You have learnt how to set an on-the-fly projection for display in QGIS, and how to use EPSG codes to rapidly find a specific projection. Remember, if you are trying to develop a map for use in the field you will need to use a projection specific to South Africa.

Editing Vector Data

Have a look at the boundary of the Eastern and Western Cape on the coast. The rivers and provincial boundary seem to follow each other closely. Now zoom right in and notice how the provincial boundary is much more detailed than the rivers, and they don't follow each other well at all.

QGIS to the rescue!

First we show you

  • Select the rivers layer in the Legend panel on the left.
  • Click on the blue Pencil icon (Toggle Editing) on the leftmost toolbar on the bottom row or <menu option>.
  • Now click on the <edit vertices> icon. Notice how the river lines have suddenly sprouted small circles wherever the line changes direction. These are the vertices - if you have used a drawing program before you might have seen something like this.
  • Drag the vertices of the rivers to align with the provincial boundary.
Notice that we don't have enough vertices to do this exactly.
  • Click on the add vertices icon. Now click on the river wherever more vertices are needed.
  • When you have enough vertices go back to "Edit Vertices" and move them where they need to be.
  • Now save your changes using by clicking on "Toggle Editing" and selecting "Save" on the dialog box.

Now You Try!

Go to Gauteng and bring up the JPEG map sheets behind the roads. Have a look at the roads and notice where the vectors are not aligned with the base map. Use the skills you have used to correct the roads vectors. Try other tools like Split Features, Move Features and other to see what they do.

Then look at the rivers and provincial boundaries again. See if you can find another place where the two are not aligned. Experiment with the Snapping Options under Project Properties. Use the Measure Length tool to decide on reasonable initial tolerance values.

What have I learned?

You have now learned how to edit vector data in QGIS using the edit toolbar. Editing data may not be as immediately rewarding as setting up a map onscreen or for printing, but it is important to be able to do this to get rid of errors that will spoil all the effort put into drawing up an impressive map!