Training Material for 2019 UN Open GIS Challenge 2 - Open geospatial data and software for UN SDG 16, Peace justice and open institutions

Introduction
→ ⇒

United Nations (UN) have identified 17 interconnected goals to achieve a better and more sustainable future for all until the end of 2030. These Sustainable Development Goals (SDGs) range from ending poverty to ensuring gender equality. In order to address most of these goals, it is imperative to have a better understanding of geographic information science and technology, as any ‘event’ has a particular geographic location. The aim of this educational tutorial is to investigate the use of Free and Open Source Software (FOSS) to address SDG 16: ‘Peace, Justice and Open Institutions’.

Scope
The following educational material has been drafted within the framework of the 2019 OSGeo UN Committee Educational Challenge entitled ‘Open geospatial data and software for UN SDG 16, Peace justice and open institutions’. The seventh target of SDG 16 is to ‘ensure responsive, inclusive, participatory and representative decision-making at all levels’. Consequently, UN officers and patrols might need to collect geospatial data on the field to provide quantitative measures to address this target. In that respect, we have investigated the use of QField, which is an open source mobile data collection platform that could directly be connected to QGIS. The investigated tasks include real-time editing of geographic features, recording photos and developing a web-interface to display the collected data. There are several cloud based application development platforms and this tutorial relied on Heroku due to its support for PostgreSQL/PostGIS as well as different web programming languages. Any user could open a free account to test their ideas on Heroku.

Audience
We have prepared this educational material for researchers, educators and professionals in local, regional, national or international agencies with minimal geospatial information knowledge. We assume our audience has introductory knowledge of database management systems and programming. Nevertheless, the tutorial is intended to be self-contained meaning that anyone following the steps could understand and implement what had been achieved in this tutorial.

License
This educational material was prepared by Berk Anbaroğlu under the mentorship of Maria Brovelli, Serena Coetzee and Timur Obukhov. In addition, İhsan Buğra Coşkun and Mohammad Almasri contributed to the development of this tutorial. We would like to thank Agit Oktay for his support regarding the development of the web interface. Last, we would like to thank Matthias Kuhn who helped us with various issues regarding QGIS and QGField, and for his suggestion to use Syncthing. The material is distributed under Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. This entitles that the users of this tutorial could share and adapt this content as long as the attribution is provided and that the same license is used upon adaptations. This tutorial has been kindly supported by the 2019 OSGeo UN Committee Educational Challenge.

Software Used

 * PostgreSQL (Postgres) 11.5 is used to create the database that would store the collected data. PostGIS 2.5.3 extension is used to handle spatial data.
 * QGIS 3.10 is used to create the project based on the data stored in the Postgres.
 * QField 1.2.0 is used for mobile data collection and editing.
 * Syncthing is used to synchronize collected photos.
 * NodeJS 12.13.1 is used to visualize the collected data on the web.
 * Visual Studio Code 1.41.0 is the Integrated Development Environment for developing the JavaScript code.
 * Git is a version control system that is required to deploy the web based application.

Scenario
United Nations (UN) officers require mobile field data collection for various purposes, mainly of which is to provide reliable measures to the indicators of targets. It is necessary to provide an easy-to-follow tutorial that will ease the process of mobile spatial data collection of UN officers. This tutorial addresses some of the practical requirements of mobile spatial data collection including offline data collection, attaching photos to recorded events, editing of spatial features on the field and displaying the collected data on the internet through a simple scenario.

The scenario in this tutorial is recording the location and thematic information of trees in a region. Specifically, a lecturer specifies polygons for which each student is responsible. Thereon, students go to the field to collect the information regarding trees in that region. Some of the regions might have limited broadband coverage, which requires students collect offline data, and some students may capture photos of trees. Finally, it would be interesting to visualise the collected data on the internet so that decision makers (lecturers) could have a better understanding of over/under mapped regions. The scenario can easily be adopted for an operational task that necessiates mobile spatial data collection.

The overview of this tutorial is illustrated in Figure 1:



The scenario utilised in this tutorial is recording the location and thematic information of trees in a region. Specifically, a lecturer identifies polygons in which each student is responsible for. Thereon, students go to the field to collect the information regarding trees in that region.

Acquired Knowledge
After going through the entire educational material, one will be able to understand the process to develop a mobile geospatial data collection platform using technologies including QField, QGIS, Postgres/PostGIS and Syncthing.

Setting Up the Environment
This section describes the required steps to set up the development environment. In order to realise the steps described in subsequent sections in a smooth manner, it is recommended that the users:
 * Open a Heroku account
 * Install PostgreSQL and PostGIS on their local computer
 * Connect the Heroku database from QGIS
 * Set up the project in QGIS
 * Transfer the project from desktop PC (QGIS) to mobile data collection platform (QField).

Opening a Heroku Account
A database server should be available in the web to initiate a mobile geographic data collection project. There are several solutions that offer a free solution to this. One of the platforms supporting Postgres and PostGIS is Heroku, which is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud. Therefore, this tutorial relied on Heroku to set up the database on the cloud.

The sign up page of Heroku, as illustrated in Figure 2, requires some personal information such as name, surname and email address. Once the form is submitted, an email is sent to verify the account. These credentials is used to create a new app.

The second step is to create a Heroku app with a Postgres database. Steps are as follows:
 * 1) Log in to your Heroku account.
 * 2) Click "New" → "Create New App" (→ symbol is a left click and ⇒ is a right-click)
 * 3) Create New App
 * 4) * Provide a unique app name. Heroku acknowledges if the provided app name is unique.
 * 5) * There are two regions: "United States" (default) and "Europe". You may leave it as it is.
 * 6) * Finally, click "Create app".
 * 7) Go to https://dashboard.heroku.com/apps and choose the app you just created.
 * 8) Click on "Resources" tab.
 * 9) * Under "Add-ons" type "Heroku Postgres" and click on it. Finally, complete the Postgres deployment by clicking on the "Provision" button.
 * 10) To see the database credentials click on the settings symbol located on the top-right corner, and click "Data". Select your application, go to "Settings" → "View Credentials".

Accessing the Heroku Database
A user interface can be used to manipulate a Postgres database such as pgAdmin or DBeaver.

This tutorial relied on pgAdmin 4, and one it is installed, we can connect to our Heroku database with the following steps:


 * 1) Right click on Servers → Create Server
 * 2) Click on General and provide a name for your server (e.g. heroku).
 * 3) Go to the Connection tab, and enter the Heroku database credentials that you have previously obtained.
 * 4) * Host name/address: Host
 * 5) * Maintenance database: Database
 * 6) * Username: User
 * 7) * Password: Password (check the ‘Save password’ box for ease of login in future)
 * 8) Finally go the Advanced tab and state the name of the database as it appears on Heroku (i.e. the same value used for the Maintenance database), so that only the relevant database would be visible.



It is now possible to create the tables. To open the query window, right click on the database and select the Query Tool.

The scenario for this tutorial requires three tables: i) trees and ii) regions are tables with a spatial aspect, whereas tree_type do not have a geometry and are used to ease the process of data collection. In addition, multiple languages are supported in a more convenient way.

First, let us create the tree_type table. The following code creates two different types of trees (pine and oak) in two different languages (English and Turkish).



The other two tables possess a spatial component. Therefore, the first step is to activate the extension postgis, which is achieved using the create extension postgis statement.

The SQL statements to create the trees and regions tables are as follows.



The following attributes are included within the created tables:
 * trees
 * t_type is the type of a tree.
 * height is the height of a tree.
 * id is the unique identifier of a tree.
 * loc is the location of a tree in WGS 84.
 * observation_time is the date-time of an observation.

trees table has a point geometry indicating the latitude and longitude of the observed tree as well as the tree type, height and observation time. The second table, regions include a polygon geometry indicating the areas in which each student is expected to carry out data collection. Students’ names can be included in the 'responsible' attribute.

Connecting the Database with QGIS
The Postgres database created in the previous subsection can be included directly into a QGIS project, thanks to the seamless linkage between QGIS and Postgres.

Once PostGIS is right-clicked, and New Connection is clicked, the following interface opens in which you can provide database credentials of the Heroku database. You can provide any arbitrary Name, but the Host, Port and Database should exactly be the same as it appears on the Heroku page. It is also important to note that the SSL mode should be allow. Go to the Basic tab under Authentication, and you may store the user name and password in order to ease the process of real time data collection. Do not forget to check the box Also list tables with no geometry to obtain all the tables of the database.

If all the credentials of the Heroku database are entered correctly, clicking the Test Connection button pops up a window indicating the success of the connection.

Finally, once the OK button is clicked, the connection appears on the Browser panel as shown below, and the user can add all the relevant tables to the layer panel by double-clicking on the table.

Setting Up the Value Relations in QGIS
In order to ease the process of data collection, it is useful to explicitly state the value relationships between the feature layers and to improve the user experience by hiding some of the details. For instance, a student can only select one of the pre-defined tree types instead of typing it. This can be achieved by adjusting the Attribute Form specified under the Properties of the trees layer as shown below.



The tree type attribute t_type has a value relation to the tree_types table. In order to realise this relation. The following settings are specified: In addition, some of the attributes might better be hidden from the users such as the id attribute of the trees or regions as it is a serial number that auto increments as a tree/region is recorded. Therefore, this attribute shall not be editable. In addition, it can be hidden from the user to reduce clutter in the GUI of the QField. This process is illustrated as follows.
 * Widget type: Value Relation
 * Layer: trees_types
 * Key column: name
 * Value column: name



Actions to be taken:
 * Uncheck Editable
 * Widget Type: Hidden

Once an attribute is hidden, it means that during data collection that field is not displayed to the user. However, a user could still update it under the attribute table. In order to prevent this as well, we have unchecked Editable.

The last attribute of the trees layer is the observation time. It might be important to record the observation time in order to facilitate further spatio-temporal analysis. At the moment, it is possible to record the height of the same tree at two or more distinct time intervals.

Update the default value of the observation_time attribute to be now as shown in the figure below. In this way, whenever a tree is recorded, its observation time is also recorded.



Actions:
 * Widget Type: Date/Time
 * Default value: now

If the mobile data collection platform assumes internet connection, then including OpenStreetMap in the QGIS project may ease data collection process. Install the OpenLayers plugin, and include OpenStreetMap in your project.



Transferring the Project to QField
Once all the layers are added, the project is saved as a QGS file. This means that the file is a legible XML document. All of the details of the project are included in this document including the credentials of the Heroku database. Anyone possessing this QGS file can start collecting field data, which would have immediate effect on the Heroku database. In this way, multiple users can effectively collect field data into the same database. This file can directly be transferred to the smart phone or tablet possessing QField.



Once the QGS project file is transferred to the mobile device, it can be opened using QField as shown in (a). The layers appear just like they do in QGIS,as shown in (b). In order to start editing, the relevant layer is selected and then the pen icon is toggled in the upper-right corner of the screen. Once the horizontal lines button on the upper left corner is pressed, the user can start collecting data. The location could either be set by panning around the map and clicking on the '+' button located at the bottom-right of the screen or by clicking the locate button at the left-hand side of the screen, which uses the GNSS receiver of the phone to locate the position of the user. New features are added just like they are added in QGIS with the functionality of hiding some attributes (e.g. id in trees) or by offering a drop-down list to the user (e.g. user can select one of the pre-defined tree types).

Offline Data Collection
QField allows its users to collect data when there is no internet connection. This requires the use of the QField Snyc plugin, which can be installed from the QGIS plugin repository. The steps involved in offline data collection are as follows:
 * In QGIS, select Preferences as shown in (a) below. Clicking on this button opens the window presented in (b).
 * Set the Default Import Directory to be the path of the project.
 * Set the Default Export Directory to be the path of the files that are to be exported. This directory must be copied to the mobile device.




 * Select Project Configuration, which opens a window as shown below listing he available layers and actions that can to be taken for each of them.
 * For OpenStreetMap, select no action.
 * The other layers should be in the offline editing mode.




 * Select Package for QField, which opens the following window. Select the export directory, which can be under the path identified in Figure 16b. Once the Create button is clicked, the plugin creates the required Geopackage file and associates it with the QGS file. In this way, data are stored in Geopackage and each update to the project effectively update this Geopackage file.


 * Transfer the project to your mobile device and start collecting data using QField.
 * Once data collection is completed, synchronise your mobile device and update the Heroku database. Select Synchronize from QField, which opens up a window as illustrated in Figure 19. Once the Synchronize button is clicked, all of the content of the Geopackage is read and the Heroku database is updated by appending the new records. In this way, it is possible for multiple users to collect offline field data.

Collecting Photos in the Field
Photos can be used in mobile data collection projects to better describe a spatial object. Storing photos in a database is not trivial, and most of the time the path to the photo is stored. For this reason, an additional platform is needed to share photos where multiple users capture photos and synchronise them.

Setting Up the Snycthing Environment
Syncthing is a continuous file synchronisation program. It synchronizes files between two or more devices. The location of the shared folder is specified on Syncthing. It requires internet to synchronise the files. The aim of using Syncthing in this project is synchronising photo folders between mobile device and computer.

Firstly, Syncthing must be installed on all devices. It can be downloaded for Windows from here. There are two options as x64 and x86. The suitable option can be downloaded. Note that SyncTrayzor is the name of the software on the desktop.

Likewise, the app Syncthing must be installed on the mobile device. It can be downloaded to android devices through Google Play. Having installed Syncthing on PC and mobile devices, they are required to be connected. Open SyncTrayzor on your desktop. Go to Actions → Show ID as shown below.



The ID of a device is presented as a text and QR code as shown in Figure 23.



This code must also be entered to the mobile device. Press the '+' button in the Devices menu as shown below.



Device options will be shown. Enter the connection ID and connection name. Also, QR code can be used to add the PC as illustrated below.



Once the PC is added to the mobile, this action sends a request to the PC stating the details of the connection. Press Add Device as shown below.



After acceptance, connections will be established between the mobile device and the PC, which can be seen in SyncTrayzor as follows. The mobile device used in this tutorial is MI 8 SE.

Also, Syncthing on the mobile device displays the connection information as shown here:



Having reached this step means that the mobile device and PC are synchronised. One may add further mobile devices with the same procedure. Once the connection is established between the devices, the details regarding file synchronization must be provided.

First of all, both Syncthing and Synctrayzor needs to know that photos are saved in a folder named DCIM by default. Therefore, a folder named DCIM must be created in the project folder. It is important to remember transferring the DCIM folder when transferring the project from PC to the mobile device.

Updating the QGIS Project
The path of the captured photos must also be stored in the database. There are two different ways to add a ‘photo’ attribute to the ‘trees’ layer. First, the attribute can be added when creating the table in the database. The data type must be chosen as text. Second, it can be added from the QGIS project. This tutorial relies on the latter option. Open the attributes table. Right click the ‘trees’ layer and press open attributes table. Firstly, press the ‘Toggle Editing Mode’ which is shown as 1 in Figure 29. Secondly, press the ‘New Field’ button which is shown as 2 in the same figure. Figure 29. Adding a new attribute from QGIS After that, enter the column name and select the type. Select type as ‘Text, unlimited length (text)’. Then, press OK as shown in Figure 30. Figure 30. Adding a new field in QGIS Close the ‘Toggle Editing Mode’ and press save. The following adjustments have to be satisfied to display photos on QGIS: 1.	Right click to ‘trees’ layer and press ‘properties’ 2.	Go to Attributes Form 3.	Select the ‘Photo’ column 4.	Select ‘Widget Type’ as ‘Attachment’ 5.	Enter default path of DCIM folder 6.	Select ‘Relative paths’ These steps are shown in Figure 31. Figure 31. Configuring the photos field – part 1 Continue with the following steps: 7.	Storage Mode select ‘file paths’ 8.	Display resources path option is selected 9.	Also, ‘Display button to open file dialog’ option is selected as shown in Figure 32. Figure 32. Configuring the photos field – part 2 The last thing to do is: 10.	Selecting the ‘Integrated Document Viewer’ as image After that press the apply button. The project is now ready to take photos. Last step on photo synchronizing is that of sharing the photo folders. To synchronize the photo folder, press ‘Add Folder’ in the SyncTrayzor as shown in Figure 33. Figure 33. Adding the folder in SyncTrayzor for sharing photos with multiple users Enter the folder name and folder path. It is important to enter the file location correctly as shown in Figure 34 to make sure that photos are displayed on QGIS. Figure 34. Setting the file synchronisation folder.

Select the mobile devices you want to synchronize the folder that contains photos in the ‘Sharing’ option and press ‘Save’ button as illustrated in Figure 35. Figure 35. Selecting the devices to share the directory containing photos This sends a request to the mobile device(s) selected to receive the corresponding acknowledgment from the mobile device user as illustrated in Figure 36. Figure 36.Share request sent to mobile device Having accepted the request, the location of the 'DCIM' folder in the project must also be selected on the phone as illustrated in Figure 37. Figure 37. Setting up the mobile device to accommodate synched photos Now the synched mobile devices are ready to take photos, which will be synched with the PC, as well as with other synched mobile devices.

Web Interface
This section describes how to visualize the data stored in the database on the web. First, the section provides a brief introduction on the required technologies, including NodeJS, Visual Studio Code and GitHub. Second, the code is presented that can be used to visualize the collected field data.

Technologies
NodeJS NodeJS is an open-source, cross-platform, JavaScript runtime environment that executes JavaScript code outside of a browser. It can be downloaded from this link. Visual Studio Code Visual Studio Code is a source code editor that can be used with a variety of programming languages. It is developed by Microsoft for Windows, Linux and macOS. It can be downloaded from this link. The version used for this tutorial is 1.40.1. GitHub GitHub is a software development version control system using Git. In order to effectively upload our project to Heroku, Git must be downloaded from this link.

Create App on Heroku
Collected field data are kept on online data stores. In the previous sections, we have relied on Heroku, which allows opening free account. Heroku also allows publishing web based projects, and in order to achieve this, a project must be created in Heroku. Firstly, click the ‘new’ and ‘Create new app’ as shown in Figure 38. Figure 38 Creating a new app in Heroku

Then, enter the ‘App name’ and fill ‘Choose a region’. Press ‘create app’. You may choose ‘Europe’ as the region.

Figure 39. Providing an appropriate app name The app name should be acknowledged by Heroku. Once the provided app name is acknowledged, the statement ‘ is available’ would appear as shown in Figure 39. Let’s begin the coding of the website. Create an empty folder on the computer. After that,open the Visual Studio Code. Go to File  Open folder. Then go to Terminal  New Terminal. It opens a terminal screen below the screen. Write ‘npm init’ to terminal screen end press enter. It asks some questions to create ‘package.json’. This file holds various metadata relevant to the project. Please press enter the all questions until the end. The metadata file package.json is created at the end of this process. Once all the default values are accepted, the ‘package.json’ would look like as shown in Figure 40:

Figure 40 The metadata file: package.json Some JavaScript packages need to be installed. These packages will be installed via npm (Node Package Manager). Packages to install are listed as follows: •	Express Module Express is a minimal and flexible Node.js web application framework that provides a robust set of features to develop web and mobile applications. Write ‘npm install express --save’ to terminal and press enter.. •	File System Module File system module allows you to work with the file system on your computer. Write ‘npm install fs --save’ to terminal and press enter.

•	Installing Bluebird Module Bluebird is a fully featured library with focus on innovative features and performance. Write ‘npm install bluebird --save’ to terminal and press enter. •	Pg-Promise Module Pg-promise is a PostgreSQL interface for NodeJS. Write ‘npm install pg-promise --save’ to terminal and press enter. The package.json will be updated once these packages are installed, and will look as shown in Figure 41. Figure 41 Updated package.json file Project settings are ready. Firstly, the database connection can be established. Click the new file and add appConfig.js file as shown in Figure 42. Figure 42 Adding the appConfig.js file The appConfig.js would include the database connection settings (for now it is the localhost) and the entire content of the file is illustrated in Figure 43. Figure 43. appConfig.js file The only statement that requires to be changed regarding the credentials of the Heroku database is the connectionString. Credentials of the Heroku database can be found through ‘Settings’  ‘View Credentials’. After that, data stored in the database must be retrieved. For this, we need to create a new file ‘database.js: The connection is applied from appCongig.js and the query is applied in this script. The important thing is the query at this point. Check that your table name and queries correspond to those in your database. The content of the ‘database.js’ is shown in Figure 44. Figure 44 database.js file Finally, the web site can be created. Press ‘new file’ and create the Index.html file. The map is obtained from OpenLayers. The query response returns a GeoJSON file. Therefore, Jquery library will be used for the return query response. Also, the security policy is added to run the web site page as HTTPS protocol. Copy the code as an HTML file as shown in Figure 45. Figure 45 index.html file Finally, the website needs to be published through NodeJS. Click the new file and create index.js file. This file provides code to publish the website. It reads the HTML page and it sends a response. Figure 46. index.js file It can be checked whether the website is working or not. Go to terminal and write ‘node index.js’, then press enter. It starts the project in ‘http://localhost:4000/’. Go to a web browser and click the http://localhost:4000/. Once the browser opens, the data stored on the database would be retrieved and displayed on the map as shown in Figure 47. Figure 47. Displaying collected data on the browser

Deploying the Project on Heroku
Another file needs to be created before the website is published through Heroku. Go to new file and create ‘Procfile’ file. Heroku apps include a Procfile that specifies the commands that are executed by the app on startup. Only the starting file of the project is specified. There is one last thing needed before pub deploying the project on Heroku. The created ‘package.json’ file needs to be updated. It must be told by which file the website is started. Therefore, the start parameter must be added as shown in Figure 48. Figure 48. Adding the start parameter to the package.json Finally, the project is ready to be published on Heroku. Go to Github page and press New. Enter the ‘Repository name’ and press ‘Create repository’. Go to your file directory through cmd (Command Prompt). Write the command: •	git init Git is initialized in the file. Then write this command to add all the contents of the folder: •	git add. Files need to be committed with a command. Write the command: •	git commit –m “your message” After that, establish a remote connection. Note that you need to write the project link instead of ‘ProjectLink’. It can be done with: •	git remote add origin ProjectLink The last step is pushing the files to GitHub. Write this code: •	git push -u origin master Our files can be seen in the GitHub page. The last step to publish the website is deploying it to Heroku. Go to the Heroku project page and ‘Deploy’  Github. Find the repository on Heroku and press connect. There are two options. Go to manual deploy and press ‘Deploy Branch’. After the development stage, press ‘View’. The website is published in the given link. Whenever there is an update in QField, it is possible to observe the change on the web page as shown in Figure 49. Figure 49 Displaying the collected data

Concluding Remarks
Mobile spatial data collection is one of the common tasks that UN officers and patrols carry out on a regular basis. Therefore, easy-to-use and flexible means to achieve this improves the effectiveness of day-to-day operations. Some of the following requirements must be met to facilitate mobile data collection: •	Offline data collection: it might be possible that UN officers might be working on sites where there is limited or no access to internet. Even under such circumstances, the data collection framework must allow offline data collection. Whenever internet becomes available, officers could upload the collected data. •	Attaching multimedia evidence such as photos (and possibly other forms including voice records and videos) to events in order to promote peaceful and inclusive societies for sustainable development. By recording multimedia data on complex scenarios, such as an interview with a victim, it might be possible to have a better understanding on the linkage between undesired ‘events’ and their respective ‘locations’. •	Editing of spatial features on the field. One of the critical requirements to ensure responsive, participatory and representative decision-making at all levels is to have up-to-date data. Therefore, updating currently available databases might be as important as collecting new data. •	Defining user roles is important step towards ensuring quality assurance and quality control (QA/QC); since different users of the same system would usually have different privileges. •	Security of collected, both in terms of server infrastructure as well as providing the means to achieve anonymity, would be important to ensure a smooth data collection process. This tutorial addressed several of the aforementioned requirements. However, some of them remained unmet. For instance, it was not possible to define user roles due to the limitations of the Heroku platform providing free account. Specifically, it is restricted to execute the SQL command ‘create user’, which would be necessary to define the user privileges. Second, the credentials of the database connection is legible in the QGS file that is transmitted to the mobile devices possessing QField. Even though only confirmed officers would possess this file, it might still be a security breach. The developments on providing cloud technologies offer the potential to satisfy the aforementioned requirements. The readers of this tutorial are encouraged to follow the developments on QField Cloud. Integrating such developments into the current tutorial would substantially improve the data collection and update operations in the field.