Training Material for UN Open GIS Spiral 3
General Info
The OSGeo UN Committee promotes the development and use of open source software that meets UN needs and supports the aims of the UN. Following a meeting between OSGeo Board of Directors and the UN GIS team at FOSS4G in Seoul, Korea in September 2015, the Committee has mainly worked on the UN Open GIS Initiative, a project “...to identify and develop an Open Source GIS bundle that meets the requirements of UN operations, taking full advantage of the expertise of mission partners including partner nations, technology contributing countries, international organisations, academia, NGOs, private sector. The strategic approach shall be developed with best and shared principles, standards and ownership in a prioritized manner that addresses capability gaps and needs without duplicating efforts of other Member States or entities. The UN Open GIS Initiative strategy shall collaboratively and cooperatively develop, validate, assess, migrate and implement sound technical capabilities with all the appropriate documentation and training that in the end provides a united effort to improve the effectiveness and efficiency of utilizing Open Source GIS around the world.” (more details at [ https://wiki.osgeo.org/wiki/UnitedNations_Committee]).
Purpose of this document
This educational material is designed as a step-by-step software learning guide for the users of the UN uDig Processing Tool (GXT) for data elaboration. You can visit http://www.mangosystem.com:8080/gxt/udig_en/ for more information. GXT Toolbox for uDig is a plugin that provides geoprocessing functionalities of the GXT engine library. GXT is geoprocessing engine library written in Java to analyze geospatial data. It is based on the GeoTools library and provides OGC Web Processing Service (WPS) through GeoServer. GXT integrates seamlessly with Open Source Java GIS such as uDig GIS, 52° North Web Processing Service. Currently it supports more than 200 algorithms and KOPSS (Korea Planning Support System) Open API. GXT itself is a commercial product (server only) but desktop plugin is free for anyone who use uDig GIS.
The course is structured with content to suit novice, intermediate and advanced users.
Target Audience
The primary target audience is professionals at local, regional, national or international agencies especially those in developing countries.
License
This educational material was written by Silvia Franceschi and Andrea Antonello (HydroloGIS) with the menthorship of HaeKyong Kang of the Korea Research Institute for Human Settlements and Minpa Lee of MangoSystem, within the project of collaboration between the OSGEO foundation and UN institute under the framework of the UN OSGeo Challenge. It is distributed according to the CREATIVE COMMONS deed: Attribution - NoDerivs 2.0. According to this license type you are free to:
- copy, distribute, display and perform the work
- make commercial use of the work
Under the following conditions:
- you must attribute the work in the manner specified by the author or licensor
- you may not alter, transform, or build upon this work.
For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above. This is a human-readable summary of the Legal Code (the full license) that can be consulted at: website.
1. Introduction
This tutorial contains the description of the usage of some commands for environmental analysis of raster and vector data with the uDig Processing Toolbox. The purpose of this quick start document is to introduce the user in the use of the algorithms contained in the Processing Toolbox of uDig for environmental analysis in particular related to ecology and ecosystems identification.
The guide is not comprehensive in documenting the use and functionality of all the Processing Toolbox library. Rather, it is intended as a short introduction to guide a user through the initial steps of running some functions required to obtain information from the maps and to evaluate some main ecological quantities and indexes. The complete documentation on the use of each Processing Toolbox tool is given directly in the Graphical Interface of the commands.
In this tutorial, you will perform the following tasks:
preliminary operations
raster data analysis
NDVI
DTM and DTM derived data
vector data analysis
density
proximity analysis
assign attributes
interpolation on raster
The Korean dataset of the region around Seoul, is used as example. You can download the data from here. All the data used for this quick start are open access in order to give the user the possibility to replicate the same analysis on other places in the world without any cost. The analysis can be done also using more accurate data if available.
1.1. The uDig Processing Toolbox
The entire Processing Toolbox library is divided into 4 main sections:
General Tools: set of generic tools to handle vector data.
Import
Export
Data Creation
Calculation
Graph
Proximity Analysis
Aggregation
Extract
Overlay Analysis
Editing
Utilities
Spatial Statistics: set of tools for statistics on vector layers considering the spatial distriution of the features.
Descriptive Statistics
Point Pattern Analysis
Global Spatial Autocorrelation
Local Spatial Autocorrelation
Spatial Relationship
Spatial Distribution
Raster Tools: set of tools for raster based operations.
Descriptive Statistics
Conversion
Extract
Conditional
Distance
Math
Density
Interpolation
Surface Analysis
Zonal Tools
Utilities
GeoTools Processes: set of tools coming directly from the GeoTools library to manage and analyse directly geometries and vector and raster layers.
Geometry Processes
Vector Processes
Raster Processes
1.2. Preliminary operations
In the following section we illustrate some of the functions available in the Processing Toolbox of uDig using vector layers, satellite images and the DTM in tiff grid format and leading you running the further commands.
First of all get (or download) the Processing Toolbox dataset at processing_toolbox_tutorial_dataset.zip and unzip the content directly in a folder on your PC. The dataset covers an area around the city of Seoul in South Korea and contains three different type of information:
- Landsat8 images LC08_L1TP_115034_20180721_20180731_01_T1.tar.gz: unzip and untar this file to obtain a folder containing all the 11 available bands of the Landsat8 images in WGS84/UTM zone 52N coordinate system EPSG:32652.
- Aster Digital Elevation map (DTM at a resolution of 20 m) 20180814093648_947847400.zip: unzip this file and consider the geotif in LongLat WGS84 coordinate system EPSG:4326.
- Open Street Map vector dataset south-korea-latest-free.shp.zip: unzip this file and consider the 18 shapefiles downloaded from the OSM website. The files are in LongLat WGS84 coordinate system EPSG:4326.
First of all you have to prepare a dedicated Project and Map in uDig. In order to obtain correct results of the processing tools it is recommended to convert all the dataset in a unique coordinate reference system. Since for the current use case the data have different coordinate reference systems, we choose to work with metric coordinate reference system UTM zone 52N, EPSG: 32652.
Before starting with the analysis please do the following preliminary operations:
- open uDig
- select the Catalog view from the available views of the main application (usually this is placed in the lower part of the main window)
- drag and drop the files of the Landsat8 images band 4 and 5, the DTM and the vector layers of landuse, natural points, roads and water ways from your File Manager into the Catalog view
- right click on the Landsat8 geotif and select Add to New Map or drag and drop it in the Layers view: this should open a new Map view with the coordinate reference system of the selected data source, in this case WGS84/UTM zone 52N (please take care that this is the projection of your working View)
- drag and drop the other files directly in the Map view or in the Layers view to visualize them all in the map.
Template:Note uDig automatically reprojects the layers in the projection of the Map view, but only for visualization, the data remain in the original projection.
Figure 1. Create the Map view starting from the layer of the Landsat8 images.
Figure 2. Import of the test dataset in uDig, zoom to all layers.
1.2.1. Set up of the data of the dataset
Usually we use data from different sources therefore, very often the available information are on different coordinate reference systems (CRS) and on different/widen areas. To homogenize the works and assure that all the tools work perfectly it is reccommanded (at least) to reproject all the data in the same CRS and define a working area where to clip all the data.
Reproject reprojects the selected layer in the given CRS. There are two different versions of the tool specific for raster and vector layers available in:
- for raster layers:
Raster Tools → Utilities → Reproject
The tool requires in input:
Input Raster layer: select the raster layer to reproject from the list of the raster layers available in the Map
Target CRS: the target CRS, you can write it in the form of EPSG:32652 or click the
at the end of the line to choose how to select it:CRS form current Map
CRS from layers → then select the layer
select CRS → open the standar uDig window where to select the CRS
Resample Type: default value is NEAREST, other options are BILINEAR and CUBIC
Output Cell Size (optional): the size of the output raster if different from the original
Forced CRS (optional): force the CRS of the input raster map to the one specified here in case the input file misses this information
Output Raster: the path and name of the output raster layer.
Template:Note It is important to fix the resolution of the output raster (Output Cell Size) especially with reprojection between systems using different measurement units (degree vs metric) and in any case to be sure to have squared cells in the output layer. Squared cells are mandatories if you want to use some analysis tools and in particular to use the tools of the HortonMachine library.
Figure 3. Execution of the Reproject command for the raster of DTM.
Template:Note To open the graphical interface of the commands available in the list of the Processing Toolbox double click on the name of the tool you want to run. To run the tool click on the OK button after filling all the required input in the window. To exit the tool once executed, click on Cancel. The tool will run every time you click on the Ok button.
Template:Note The output raster will be visualized all white, use the Styling System of uDig for a better visualization.
- for vector layers:
GeoTools Processes → Vector processes → Reproject
Feature layer: select the vector layer to reproject from the list of the vector layers available in the Map
Forced CRS (optional): force the CRS of the input vector map to the one specified here in case the input file misses this information
Target CRS: the target CRS, you can write it in the form of EPSG:32652 or click the
at the end of the line to choose how to select it:CRS form current Map
CRS from layers → then select the layer
select CRS → open the standar uDig window where to select the CRS
Result: the path and name of the output vector layer.
Figure 4. Execution of the Reproject command for the vector of the landuse.
To go on with the tutorial please reproject using the same tool as for the landuse layer also the vector layers of:
- gis_osm_natural_free_1
- gis_osm_waterways_free_1
- gis_osm_water_a_free_1
- gis_osm_roads_free_1
- gis_osm_landuse_a_free_1.
After reprojecting all the layers you can delete the original layers from the Layers view. To do this you can select all the layers or one layer at a time from the Layers view and select Delete from the context menu of the right mouse click.
Figure 5. Delete some layers from the Layers view.
Clip extracts the features of the selected layer for a defined region.
Before starting with the clipping we should define our working area as a polygon geometry. The standar process to do this operation in uDig is the following:
create a new layer: Layer → Create
define the characteristics of the new layer:
name: area_of_interest
attributes:
name: String
geometry: Polygon
CRS: UTM zone 52N (EPSG: 32684)
click on OK to add the new layer to the project
select the editing tool to Create → Create Rectangle
draw a rectangle in the area around Seoul (not too big but big enough to contain some of the natural points, see the picture).
The following image contains an example of the area of interest.
Figure 6. Example of the layer of the area of interest.
There is a new module in the Processing Toolbox developed to simplify this operation. In fact, the Geometry to Features tool can be used to automatically extract a polygon layer on the Map extent.
General Tools → Import → Geometry to Features
The tool requires in input:
Input Geometry (WKT): the geometry to import, click the
at the end of the line to choose how to select it:Point
LineString
Polygon: and then select the first option Polygon from Map’s Extent
Geometry from Layers…: select the layer or the features to use to create the new layer
CRS (optional): the CRS of the input geometry if different from the one of the current Map
Name (optional): name for the features in the new layer
Single Part (optional): boolean variable to define if it is required to split multipart geometry to single parts, default is No
Result Features: the path and name of the output layer containing the new features.
Figure 7. Execution of the Geometry to Features command to extract the area of interest.
Figure 8. Example of the layer of the area of interest extracted with the command Geometry to Features.
The Processing Toolbox contains some different versions of the clipping tool. You can visualize all of them just typing the word clip in the search box of the Processing Toolbox window.
Figure 9. Search for the different possibilities of clipping operations in the Processing Toolbox.
In particular we are interested in clipping both raster and vector layers and therefore we will use:
- for raster layers:
Raster Tools → Extract → Clip by Extent
The tool requires in input:
Input Raster layer: selects the raster layer to clip from the list of the raster layers available in the Map
Extent: a reference for the boundaries of the clipping area, click the
at the end of the line to choose how to select it:Current Extent of the Map
Full Extent of the Map
Layer’s Extent: selects the layer of the area_of_interest
Output Raster: the path and name of the output raster layer.
Figure 10. Execution of the Clip by Extent command for the projected raster of the DTM.
- for vector layers:
General Tools → Extract → Clip With Polygon Geometry
The tool requires in input:
Input Feature layer: select the vector layer to clip from the list of the vector layers available in the Map
Clip Polygon Geometry (WKT): a reference for the boundaries of the clipping area, click the
at the end of the line to choose how to select it:Point
LineString: selects the boundary from the area_of_interest Layer’s Extent
Polygon: then selects the polygon from the area_of_interest Layer’s Extent
Geometry from Layers…: select the layer and feature ID of the polygon to use for delimiting the new vector layer
Output Features: the path and name of the output vector layer.
Figure 11. Execution of the Clip With Polygon Geometry command for the projected vector of the landuse.
Figure 12. Execution of the Clip With Polygon Geometry command for the projected vector of the landuse, select the geometry picker.
To go on with the tutorial please clip using the same tools as used for DTM and landuse, respectively for raster and vector layers, also the other following layers (take care to use the reprojected ones):
- Landsat8 images of band 4 and 5: LC08_L1TP_116034_20160519_20170324_01_T1_B4.TIF and LC08_L1TP_116034_20160519_20170324_01_T1_B5.TIF
- osm_natural_utm
- osm_waterways_utm
- osm_water_a_utm
- osm_roads_utm.
After clipping all the layers you can delete the original layers from the Layers view.
The final configuration of the application and data should be like the one in the next image.
Figure 13. Configuration of the uDig application after the preliminary operations.
SetNull sets specific values in the raster layer to assign to NoData. This tool works only for raster layers and it allows the user to set the cell with certain values to NoData (not valid). Doing this, these cells will be automatically excluded from further elaborations on the raster maps like statistical analysis or during the evaluation of spatial indexes.
Template:Note To query single values of the raster layers or to display the attributes of specific features of vector layers please use the info tool of uDig available in the Palette of the Map view.
To set the value of 0.0 of the Landsat8 images to NoData you can use the SetNull tool available in:
Raster Tools → Conditional → Set Null
The tool requires in input:
- Input Raster layer: select the raster layer of which to set the NoData values from the list of the raster layers available in the Map
- Band index (optional): optional value of band index to use for multiple band layers, otherwise leave it at 0
- NoData Filert Expression: expression used by the software to identify the NoData values in the layer, click on the at the end of the line to choose how to define it.
- Replace NoData (optional): flag to identify if the tool has to work in the standard or in the opposite way, that means replacing NoData values with a valid value
- New Value (optional): this value is required only if the Replace NoData is activated (Yes) and it represents the new value to assign to previous NoData values
- Output Raster: the path and name of the output raster layer.
Figure 14. Execution of the Set Null command for the raster of the Landsat8 images.
In our example we have to set the value 0.0 of the Landsat8 images to NoData. This expression is very easy and can be written directly in the NoData Filert Expression as:
[Value] = 0.0
In case there is the need to use more advanced expressions to identify the values of a raster map to set to NoData it is possible to use the Query Builder Dialog integrated in the Processing Toolbox just clicking on the 
button and select:
- Layer: only for testing the expression directly in the Query Builder Dialog
- Fields: in case of raster layers the only field available is the Value, double click on the item in this section to add it to the query or type Value directly in the text field below
- Operators: select one of the available operators or directly type it in the text field below
- SQL where clause: the resulting SQL expression used by the software for filtering the values and assigning the NoData.
The Query Builder Dialog gives also the possibility to show in the Values section a sample or all the data contained in the selected raster layer clicking on the Sample, or for all values on All button and to Test the expression inserted clicking the specific button at the bottom of the window.
Figure 15. The Query builder options of the Set Null command for the raster of the Landsat8 images.
To go on with the tutorial please set the null values to zero using the same Set Null tool for both the raster layers of the Landsat8 images, band 4 and 5: LC08_L1TP_116034_T1_B4_utm_seoul.tif and LC08_L1TP_116034_T1_B5_utm_seoul.tif.
2. Analysis of Landsat raster images
In the field of environmental sciences lot of analysis have to be performed on raster data. In this tutorial we will consider two different kind of raster data:
- satellite images (Landsat8)
- digital elevation maps (DTM) and derived products.
Raster data are continuous data in the format of a matrix of cells (or pixels) organized into rows and columns where each cell contains a value representing the content of the raster. The easiest way to observe the contened values is to use a right style, but if you need some more precise information you have to analyse them. The Spatial Toolbox provides useful tools for analyzing and do elaborations of raster layers.
Summary Statistics for Raster is a tool which prints a summary of the main statistics of the selected raster layer. It is available in the section:
Raster Tools → Descriptive Statistics
double click on the Summary Statistics for Raster entry. The required input parameters are:
Input Raster layer: select the raster layer to elaborate from the list of the raster layers available in the Map
Crop Geometry (optional): a reference for "clipping" a custom area of the raster layer if you need to extract the information on a subset of the complete raster; click the
at the end of the line to choose how to select it:Point
LineString: selects the boundary from the area_of_interest Layer’s Extent
Polygon: selects the polygon from the area_of_interest Layer’s Extent
Geometry from Layers…: selects the layer and feature ID of the polygon to use for delimiting the new vector layer
Band Index (optional): optional value of band index to use for multiple band layers, otherwise leave it at 0.
Figure 16. Execution of the Summary Statistics for Raster.
The tools evaluates the following statistics:
- Count: number of valid cells of the raster layer
- Invalid Count: number of not valid cells or unclassified values
- Minimum: minimun value of the raster
- Maximum: maximum value of the raster
- Range: the entire range of the data content in the raster layer (max - min)
- Ranges: the available ranges the data content in the raster layer in case of an elaboration on multiple geometries
- Sum: the sum of the data content in the raster layer
- Mean: the average of the data content in the raster layer
- Variance: the variance of the data content in the raster layer
- Standard Deviation: the standard deviation of the data content in the raster layer
- Coefficient of Variance: the Relative Standard Deviation of the data content in the raster layer
- NoData: the value representing the NoData cells in the raster layer.
Figure 17. Output of the Summary Statistics for Raster on the Landsat8 image of Band 5.
Figure 18. Output of the Summary Statistics for Raster on the DTM.
Histogram Raster is a tool which calculates the information to be used to create a histogram of the content of the raster layer. In particular, this tool calculates and prints the number of cells of each different value in the map. These values can be used to display a histogram chart, just select and copy and paste them in a spreadsheet. It is available in the section:
Raster Tools → Descriptive Statistics
double click on the Histogram raster entry. The required input parameters are:
Input Raster layer: select the raster layer to elaborate from the list of the raster layers available in the Map
Band Index (optional): optional value of band index to use for multiple band layers, otherwise leave it at 0
Crop Geometry (optional): a reference for "clipping" a custom area of the raster layer if you need to extract the information on a subset of the complete raster; click the
at the end of the line to choose how to select it:Point
LineString: selects the boundary from the area_of_interest Layer’s Extent
Polygon: selects the polygon from the area_of_interest Layer’s Extent
Geometry from Layers…: selects the layer and feature ID of the polygon to use for delimiting the new vector layer.
Figure 19. Execution of the Histogram Raster.
Figure 20. Output of the Histogram Raster.
Figure 21. Graphic output of the Histogram Raster.
Raster Description is a tool which can be used to have a first description of the general metadata of the selected raster. The metadata visualized are: Name, Columns, Rows, Number of Bands, X-Y Cell Size, Pixel Type, Pixel Depth, NoData Value, the whole Extent and the CRS (Coordinate Reference System). This tool gives also the possibility to directly perform Summary Statistics.
Figure 22. Execution of the Raster Description.
Figure 23. Output of the Raster Description.