GSoC 2020 Data transaction implementation on pygeoapi

Introduction

Idea

pygeoapi is a python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.

This GSoC project aims to add support for simple transactions in pygeoapi. This project will implement Create, Update, Delete operations in pygeoapi data providers, thereby enabling all standard CRUD transactions in pygeoapi. This will enable an application developer to perform a transaction in a pygeoapi data provider through an HTTP request containing the resource id and suitable HTTP verb (POST/PUT/DELETE/PATCH). The project proposal is based on OGC API - Features - Part 4: Simple Transactions.

Project proposal

My proposal for GSoC 2020 can be found at Proposal.

Advantages from this project

pygeoapi implements OGC WFS 3.0 (OGC Feature API) which is fully RESTful. This makes it much easier for the application developers to work with. This project idea implements 'Part 4 : Simple Transactions' of this OGC standard. It adds transactional capabilities to the pygeoapi data providers by which application developers can perform transactions via the RESTful API itself. A RESTful API supporting complete CRUD transactions will be preferred by most application developers over other non-RESTful APIs. So this will make pygeoapi a more popular choice among application developers.

Link to Github repository: Repo

Timeline

4th May - 31st May

What I have done during this period?

• Joined pygeoapi Gitter room
• Went through the current features of pygeoapi
• Went through OGC API - Features - Part 4: Simple Transactions [1] draft
• Forked the pygeoapi GitHub repository Repository
• Introduced myself over the channel and shared my proposal over mailing list for suggestions
• Created a wiki page for the project "Data transaction implementation on pygeoapi"
• Updated wiki user page and added my personal information [Link]
• Updated links on the wiki Google_Summer_of_Code_2020_Accepted page [Link]
• Had a discussion with mentors regarding the expectations and development workflow
• Discussed the openapi specification document with the mentors
• Started working on the openapi document

Week 1(1st June - 7th June)

What I have done during this period?

• Defined openapi document schemas for remove(DELETE), insert(POST), modify(PATCH), replace(PUT) transactions
• Pushed the proposed openapi document sample to the openapidoc branch in my forked repo
• Shared the doc with pygeoapi community for suggestions

What do I plan on doing next week?

• Modify openapi.py file so that it generates the proposed openapi document
• Test the code for compliance

Blocking issues

None

Week 2(8th June - 14th June)

What I have done during this period?

• Corrected schemas for insert(POST) & replace(PUT) transactions as per suggestions from mentors
• Modified the code to generate the proposed openapi doc
• Pushed the changes to my forked repo

What do I plan on doing next week?

• Test the code for compliance

Blocking issues

None

Week 3(15th June - 21st June)

What I have done during this period?

• Started writing test cases for get_oas30() function
• Started testing and made corrections to the code
• Pushed the changes to my forked repo

What do I plan on doing next week?

• Complete writing test cases and compliance test
• Start documentation

Blocking issues

None

Week 4(22nd June - 28th June)

What I have done during this period?

• Refactored code and passed flake8 linting 
• Integrated travis CI and read the docs to my branch
• Pushed the changes to my forked repo

What do I plan on doing next week?

• Add more test cases and pass travis build
• Mockup REST endpoints on flask_app.py

Blocking issues

None

Week 5(29th June - 5th July)

What I have done during this period?

• Refactored code to pass the community standards
• Added details about transaction functionality to the configuration section in the documentation
• Raised PR for data-transaction-GSoC-1 branch
• Started researching on generic implementations for routing

What do I plan on doing next week?

• Fix a generic implementation for routing
• Mockup REST endpoints on flask_app.py

Blocking issues

None

Week 6(6th July - 12th July)

What I have done during this period?

• Performed suggested changes on the PR for data-transaction-GSoC-1 branch
• Designed a basic solution for generic flask endpoint to add support for transactions
• Successfully implemented a flask endpoint for POST and tested it with the available geojson provider implementation

What do I plan on doing next week?

• Find a better solution for the generic endpoint
• Implement rest of the flask endpoints
• Refactor code by adding exceptions and write unit tests

Blocking issues

None

Week 7(13th July - 19th July)

What I have done during this period?

• Implemented flask and starlette routings for transaction requests
• Implemented  transaction support for the geojson data provider

What do I plan on doing next week?

• Add transaction unit tests for flask_app.py and starlette_app.py
• Add transaction unit tests for the geojson data provider 
• Start implementing transactional support in other data providers

Blocking issues

None

Week 8(20th July - 26th July)

What I have done during this period?

• Implemented  transaction support for csv data provider
• Refactored geojson data provider code
• Added new unit tests for geojson and csv data providers

What do I plan on doing next week?

• Write unit tests for transactional support in remaining data providers
• Start implementing transactional support in remaining data providers

Blocking issues

None

Week 9(27th July - 2nd August)

What I have done during this period?

• Modified schemaless providers to fully support PATCH verb
• Completed unit tests for geojson and csv providers 
• Completed functional tests for flask and starlette endpoints

What do I plan on doing next week?

• Write unit tests for transactional support in remaining data providers
• Start implementing transactional support in remaining data providers

Blocking issues

None

Week 10(3rd August - 9th August)

What I have done during this period?

• Improved openapi with examples
• Improved descriptions for CRUD operations
• Created unit tests for postgresql and sqlite providers

What do I plan on doing next week?

• Implement transactional support for postgresql and sqlite providers

Blocking issues

None

Week 11(10th August - 16th August)

What I have done during this period?

• Added documentation for data transactions
• Updated id selection rule for POST transaction based on data type of id
• Updated request examples in openapidoc
• Started working on transactional support for postgresql and sqlite providers

What do I plan on doing next week?

• Complete transactional support for postgresql and sqlite providers

Blocking issues

None

Week 12(17th August - 23rd August)

What I have done during this period?

• Updated documentation for data transactions
• Updated csv provider to handle type of attributes
• Updated id selection rule in POST to generate type specific ids
• Updated request examples in openapidoc
• Started writing unit tests for postgresql and sqlite providers

What do I plan on doing next week?

• Complete transactional support for postgresql and sqlite providers

Blocking issues

None

Student's Biography

My name is Alex Mathew. I am a computer science graduate now pursuing my masters in geoinformatics under the Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India. Some more information about me can be obtained by following the link: [Link]

Mentors

• Jorge de Jesus
• Francesco Bartoli