New Virtual Raster Data Provider for Raster Calculator in QGIS
Introduction
QGIS and Tasks
QGIS is a powerful Free and Open Source GIS software, you can work with everything that has some spatial characteristics. The aim of this project is to add some tool to the existing raster calculator to facilitate some workflows.
Project proposal
The proposal for GSoC 2021 that I submitted can be found here.
Abstract
QGIS already has an integrated raster calculator, where user can enter an expression that will apply raster algebra functions to one or more input rasters and produce a new raster file. This project aim is to develop a new virtual raster data provider that can help to use the raster calculator without creating a lot of files on a disk. It will be possible to perform the computation on the fly when map gets rendered. It will be therefore possible for the users to see the results of raster algebra expression that has been typed without the creation of derived raster files. For example, it should be possible to compute the NDVI index on-the-fly.
Link to Github repository: Repository
Timeline
17th May - 6th June
Community bonding period:
Task
17th May - 30th May
- Get in contact with mentors, try to code something and look for some small bugfix I can work on;
- Join Slack channel created by mentors and plan the weekly meeting as they suggested;
- Set up of the development environment for QGIS (build the code from source, fork the official Repository and create a new branch, set Qt creator environment as it's described in the documentation for the developers );
- Start to dive into the code base by studying the existing QgsRasterDataProvider;
- Introduce myself over the channel and shared my proposal on the mailing list for suggestions and feedbacks;
- Create a wiki page for the project and my User page;
1st June - 6th June
- Sumbit a PR for simple bugfix to the official repository;
- Update of the proposal and share the updates with the community;
- Study the codebase, start to study and implement some simple unit test;
- Prototype the new virtual raster data provider with some basic methods, add the new provider to the provider registry class, start to write some unit test in c++, study the raster calculator class: I've done several commit on the forked repository so everyone can see what I am doing: it's possible to find the new provider and the test I added.
Plans for next week
- Have a meeting and confront with mentors;
- Work and add methods to the prototyped class I've created;
- Add some methods to the unit test I wrote accordingly to the new provider class;
- Study the raster calculator classes.
First Week: 7th June - 13th June
Coding Period:
Task
- Update with mentors: I received advices and some help;
- Move the provider I created from QGIS/src/core/providers/ to QGIS/src/providers/;
- I write a CMakeLists.txt to include the class I created and to target the libraries I will use in my class;
- Add, under mentors advice, a function to add the provider to the provider registry dynamically;
- Study the raster calculator and add some functionalities of the raster calculator to the new provider;
- Work on the test part, study the other existing tests and adding minor changes to the my test class;
- create a PR to my personal main repo from the branch in which I'm committing the changes, in this way it will be easier to check the changes and add comments or advices [1];
[1] https://github.com/Franc-Brs/QGIS/pull/3
Plans for next week
- Add some methods to the test class;
- Refining the prototyped class and add some checks in the methods I added;
- Have the meetings with mentors.
Second Week: 14th June - 20th June
Coding Period:
Task
- Update with mentors about some technical issues of my code;
- Add a method to the test class in order to call the virtualrasterprovider class I created;
- Update the CMakeList.txt in order to target all the required libraries in the test;
- Develop the raster calculator functionalities in my provider in the block() method, using some hardcoded data;
- Debug my code to check where it fails in returning the raster block with the hardcoded data;
Plans for next week
- Adjust the code to return the correct computed data from the block method;
- Start to look at and design the methods for taking the uri with all the parameters;
Third Week: 21th June - 27th June
Coding Period:
Task
- Refine the block() method in order to compute some easy map algebra with hardcoded data and an hardcoded expression;
- Add some specific function in the unit test in order to inspect the values of the computed raster of the code;
- Study of the existing parsing and decoding method for the URI and study QUrl in particular;
- Pseudocode the methods for the parse and decode of the URI;
Plans for next week
- Develop the methods in some class of qgis_core to parse and to encode the uri that will be given to my provider (that was moved from qgis_core to providers);
- Refine the methods of the prototyped provider in order to add some check on the parameters given by the uri;
Fourth Week: 28th June - 4th July
Coding Period:
Task
- Define a syntax for the uri that will be passed to my provider with all the information that it needs (extent of the block, width, height, output crs, formula defined in the calculator and raster entries that will be used in the formula)
- Add encode and decode methods to qgsRasterDataProvider class, so that they are available in qgis core;
- Add the above methods to my virtual raster provider in order to deal with the given uri;
- Add a method in the unit test class of virtual raster provider to test the decode method;
Plans for next week
- Refine the encode method accordinggly to the defined syntax for the uri;
- Add some test method to verify that my code works in the correct way for the encode part;
Fifth Week: 5th July - 11th July
Coding Period/Evaluation period:
Task
- According to the defined syntax of the URI and after a meeting and some advices from mentors I changed the behaviour of the decoding/encoding methods in qgsrasterdataprovider, defining a simple struct that contains the different elements that I should pass to my provider with the URI (instead of using a QVarianMap as output in the decode and input in the encode method);
- Add the decode functionality in the constructor of my provide;
- Add a test for the decode/encode method in the unit test class;
- Add a test for the constructor in the unit test class;
- Add some checks in the decode method to detect invalid or missing string element that should be, instead, in the URI;
- Add some checks in the provider constructor to see if the decoded elements are valid.
Plans for next week
- Continue to develop the checks both in the provider constructor and in the decode method;
- Work on the block method of the virtual raster provider, avoiding hardcoded data and try to use the provider via python console or in a more general unit test;
- Submit the requested evaluation.
Sixth Week: 12th July - 18th July
Coding Period/Evaluation period:
Task
- Submit the evaluation;
- Do a draft pr with some documentation and informations to the official QGIS repository;
- Adjust and refine both my provider constructor and decode methods in order to check the URi correctly and to be able to use the provider without hardcoded data;
- Modify a small part of my block method and complete the copy constructor in orde to use the provider from the python console inside QGIS;
- Change some test methods;
Plans for next week
- Design a method (some methods) to write the URi in an easy way;
- Start to look at the implementation of the UI;
Seventh Week: 19th July - 25th July
Coding Period:
Task
- Work on the UI dialog logic;
- Connect the UI with my provider and make the new work as a draft;
- Write a method in qgsrastercalcnode class to obtain raster name and raster references in the qgsrastercalcualtor;
- Start to worok on the first part of the review submitted by mentors.
Plans for next week
- Continue to work on the UI and modify it accordingly to mentors advices;
- Work on the review given by my mentors;
Eight Week: 26th July - 1st August
Coding Period:
Task
- Work on the UI and make it works;
- Finish to make the change for the review;
- Add some code to deal with relative and absolute path of the URI of the provider;
- Start to look into the possibility of adding conditional statements in the raster calculator.
Plans for next week
- Work on the addition of conditional statements in the raster calculator;
- Usual meeting with mentors and global check on the work done till now.
Ninth Week: 2nd August - 8th August
Coding Period:
Task
- Clean the code, according to the reviews of mentors;
- Start to add some documentation;
- Work on the conditional statement in the raster calculator in another branch;
Plans for next week
- Continue to work on the addition of conditional statements in the raster calculator;
- Usual meeting with mentors and checks on some of the possible issues that can affect the feature I've developed.
Tenth Week: 9th August - 15th August
Coding Period:
Task
- Clean the code, according to the reviews of mentors;
- Small modification on the qgisapp.cpp in order to allow creation of different order of virtual raster (i.e. a virtual raster from a virtual raster)
- Work on the conditional statement in the raster calculator in another branch;
- Add test method to test the woek I've done for the conditional statement.
Plans for next week
- Continue to work on the addition of conditional statements in the raster calculator;
- Final check on the PR;
- Work on the final evaluation.
GSoC 2021 Final Report
Title: New Virtual Raster Data Provider for Raster Calculator in QGIS
Before the coding period
- I set up the work environment;
- I read the existing documentation for QGIS developers and I started to dive into the codebase;
- I made a little bugfix, you can find the link to the PR in [1].
Abstract
The Raster Calculator in the Raster menu of QGSIS allows the user to perform calculations on the basis of existing raster pixel values. The results are written to a new raster layer in a GDAL-supported format.
The feature I've worked on allow the user to use the QGIS raster calculator without the need of writing the result on a file. I developed a virtual data provider for rasters used by the raster calculator. With this feature it is possible to add a virtual (raster) layer to QGIS avoiding the creation of derived files from the geo-processes and without occupying disk space. The new raster is created on-the-fly, and it is added to the TOC of QGIS, the same raster can be used for analysis purposes or only for rendering, and it can be also used other times in the raster calculator.
State of the QGIS Project before my GSoC
As it is written in the documentation [2] and and in the abstract above, in QGIS it is possibile to perform raster algebra but an output file is needed to write the results. Before this work there wasn't the possibility to use this calculator withouth the need of writing a file to the disk.
The addition that my project brought to QGIS
The Pull Request I submitted and that covers all the work I've done during my GSoC can be found in [3] and the changes in the code can be found in [4], precisely. In [6] it's possible to find the new branch I created for the development of this feature. As I've described in the proposal [5] and in the above chapters, the new feature I've worked on in the last two months involves the usage of the existing raster calculator of QGIS with the possibility of computing a new raster on-the-fly without the need to write the result on a file. To achieve and to develop this functionality I accomplished the following steps:
- Create a data provider that takes the raster data from files or from layers loaded in QGIS and use them to perform the computation. So I subclassed the QgsRasterDataProvider class and I create a QgsVirtualRasterProvider class that will takes the data and load them in QGIS. This new virtual raster data provider takes advantage of some properties of the raster calculator and therefore providing to it the right information, it's possible to obtain the desired result;
- During the process of development I also added a test class to verify that my work was consistent and the QgsVirtualRasterProvider behaved in the right way. Moreover the TestQgsVirtualRasterProvider I created helped me to understand where the code had some flaws. The test class can be found in [4];
- After the development of the new virtual raster data provider it was possible to use the new feature via python console, since the existing raster calculator already has an amazing dialog for the user interface I made some changes to it in order to enable the user to work with my new feature through the dialog.
As it is reported by the PR at [3] it is possible to use and to test this new feature by choosing "raster calculator" under the "Raster" tab and by checking the first checkbox of the dialog near the label: "Create on-the-fly raster instead of writing layer to disk". After the check the next things will happen:
- It will be given the possibility to choose a name for the layer that will be automatically added to the project, if nothing is typed the name is automatically taken from the formula;
- The possibility to choose a folder and the format for the output file will be hidden;
- The "Add result to project" checkbox will be checked without any other possibility.
In the next screen is shown a simple example with the raster calculator dialog, where in [10] it is possible to find the animated GIF.
By selecting the checkbox "Create on-the-fly raster instead of writing layer to disk" as it has been described, there will be given the possibility to choose the name, it is also possible to compute a raster that is created by using this feature, so starting from a on-the-fly computed raster it is possible to derive another on-the-fly raster.
The new data provider accepts a uri that must contains all the informations needed, the parameters are the following:
- crs;
- extent;
- width;
- height;
- formula;
- rasterX:uri;
- rasterX:provider;
- (other rasters uri and providers if needed).
Where rasterX:uri is uri the of the raster it's referenced in the formula and rasterX:provider is its provider. If there is the need to target multiple rasters, the uri and the provider for every raster (in the test class there are some examples of uri) should be given. At [10] you can find the final report.
Potential future Work
- Since I had some time left I start a new branch [7] for enhance the raster calculator capabilities by adding the possibility of using a conditional statement in the following form:
if(raster@1>100, 400, 10)
- Where the computed raster pixel will have value 400 if raster@1 is greater than 100 or it will have value 10 in any other case.
- This enhancement was not part of the initial proposal and therefore it's not included in the final submission, from the PR [10] it is possible to check what I changed and what I've done so far;
- An enhancement for the developed provider can be the possibility to take advantage of OpenCL acceleration as it has also been suggested in the dev mailing list;
- Another enhancement that concerns more the raster calculator and only partially the virtual raster provider would be the possibility to support the creation of output raster with multiple bands with the declaration of multiple formulas.
Useful Links
I've collected all the links that I mentioned in this report and all the links that can be useful to understand my work.
[1] https://github.com/qgis/QGIS/pull/43469#event-4832189934 (small bug-fix)
[2] https://docs.qgis.org/3.16/en/docs/user_manual/working_with_raster/raster_analysis.html#raster-calculator (state of art before this GSoC)
[3] https://github.com/qgis/QGIS/pull/44195 (Pull request to the main QGIS repository)
[4] https://github.com/qgis/QGIS/pull/44195/files (Files changed and created during my GSoC)
[5] https://docs.google.com/document/d/1u8L_L1IJjGCZ3d8Vq8duxHbsHZeyzwqOe3tXa1JcKbs/edit (GSoC Proposal)
[6] https://github.com/Franc-Brs/QGIS/tree/VRProvider (New branch for the Virtual Raster Provider)
[7] https://github.com/Franc-Brs/QGIS/tree/rastercalc-conditional-statements (New branch for conditional statement in raster calculator)
[8] https://wiki.osgeo.org/wiki/New_Virtual_Raster_Data_Provider_for_Raster_Calculator_in_QGIS (Main project page)
[9] https://summerofcode.withgoogle.com/projects/#5286802463653888 (Official Google page)
[10] https://gist.github.com/Franc-Brs/b533fde3bec4ba16978c8fd2ef5e064f#file-gsoc2021-md (Final report, Gist)
[11] https://github.com/Franc-Brs/QGIS/pull/4 (PR to my personal remote repository for the new branch for the conditional statement)
Student's Biography
I am Francesco Bursi and I’m currently enrolled at Advanced Master on "GIScience and Unmanned System for the integrated management of the territory and the natural resources - with majors" with specialization in GISCIENCE and GEOINFORMATICS (Level 8 EQF) at the University of Padua, in Italy. I took my master’s degree in civil engineering attending the Polytechnic of Turin in 2017. I'm passionate about informatics and GIS, and I really love OS environment. Some more information about me can be found on Linkedln and on Github