GSoC 17: istSOS-Data analysis and statistical tools suite
| This is where I'll be posting information about istSOS-Data analysis and statistical tools suite GSoC 2017 project to the OSGeo community. |
Introduction
istSOS is sensor data management tool that allows collection, maintenance and publishing of monitoring observations using the Open Geospatial Consortium (OGC) Sensor Observation Service (SOS)standard.
OAT(Observation Analysis Tools) can be used as a pre-processor for calibration observations, integrating the creation of observations for calibration directly from sensor time-series. The tool consists in an expandable Python library of processing methods and an interface integrated in the QGIS FREEWAT plug-in which includes a large number of modelling capabilities, data management tools and calibration capacity.
OAT library is an expandable Python library that can easily interact with istSOS.
The goal of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticate documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).
Idea
istSOS has gone through a lot of improvements since 2009. This has somehow affected the simplified interaction with measurements and SOS service settings.
The aim of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticat documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).
Project
Title: istSOS-Data analysis and statistical tools suite.
Student: Rahul Chauhan
Mentors: Massimiliano Cannata, Milan Antonovic.
Details
Development process can be monitored at the following links:
- https://summerofcode.withgoogle.com/projects/#6301175977607168 - Project GSoC 2017 application Link
- https://github.com/istSOS/istsos2 - Project Main Repo. Link
- https://github.com/rahulworld/istsos2 - Forked Github Repo. Link
- https://github.com/rahulworld/Data-analysis - Project GitHub testing Repo. Link
Implementation details
OAT (Observation Analysis Tools)
OAT is designed to facilitate the import, analysis and visualization of time-series data and the use of these data to support model construction and advanced model calibration.
Observation Analysis Tool was developed to enable time-series visualisation, pre-processing of data for model development, and post-processing of model results. Observation Analysis Tool can act as a pre-processor for calibration observations, and will be expanded to incorporate its processing capabilities directly into the calibration process. The tool consists in an expandable Python library and in an interface integrated in the QGIS FREEWAT plug-in which include a large number of modelling capabilities, hydro-chemical data management tools and calibration capacity. The tool has been extensively used and tested in different european institutions, to collect a number of indications to drive the future development.
- The OAT library implements two class:
- 1. Sensor class that is designated to handle time series data and metadata.
- 2. Method class which is designated to represent a processing method.
- OAT takes advantage of the PANDAS (McKinney, 2010),NUMPY and SCIPY (Van der Walt et. al. 2011) packages.
- Most of the currently available OAT.Method objects are based on TSPROC processing capabilities. They enable the analysis of raw data (resampling, regularization, data interpolation, fitting, filling,validation and data quality assessment); decomposition or filtering of timeseries (low, medium, highfrequencies); aggregation and exceedancetime calculation, summary statistics and period statistics,integration and hydrograph separation among others.
Schedule
| Dates | Task |
|---|---|
| Week 1 (May, 30 - June, 4) | User interface implementation. |
| Week 2 (June, 5 - June, 11) | Implementation of resample method(OAT) in istSOS api. |
| Week 3 (June, 12 - June, 18) | Testing resample(OAT method) with istSOS api. |
| Week 4 (June, 19 - June, 25) | Implementation of regularisation time series analysis method(OAT) in istSOS api. |
| Week 5 (June, 26 - July, 2) Phase 1 Evaluation | Test the working of these operations. |
| Week 6 (July, 3 - July, 9) | Implement OAT.methods i.e. , interpolation, digital filter. |
| Week 7 (July, 10 - July, 16) | Implement OAT.methods i.e. filling, validation, data quality assessment. |
| Week 8 (July, 17 - July, 23) | Implement remain OAT.methods. |
| Week 9 (July, 24 - July, 30) Phase 2 Evaluation | Testing on OAT method in istSOS api. |
| Week 10 (July, 31 - August, 6) | Implement CKAN (Open Data Portals) plugin. |
| Week 11 (August, 7 - August, 13) | Implement publish datasets ready to be harvested using CKAN. |
| Week 12 (August, 14 - August, 20) | Testing, Debugging, Cleaning and Optimizing the codes. |
| Week 13 (August, 21 - August, 27) | Take a week to scrub code, write a few test cases and test them, improve the current documentation. |
| Week 14 (August, 27 - August, 29) Final evaluation | Buffer period. ( Final Week. Prepare documentation for final submission.) |
Reports
May 4 - May 14
- created github repo. https://github.com/rahulworld/Data-analysis
- Installed all devlopment requsites.
- Tried to understand the code flow of istSOS and OAT.
- Created Project wiki.
May 15 - May 29
What did I complete this week?
- Designed GUI for all Process time series methods.
- Also Designed GUI for
- Add Time Series
- Process Time Series
- Compare Sensor
- Manage Sensor
- This week, I’ve been indentifying all the process time series methods in OAT(freewat), I mean:
- digital filter
- exceedance
- hydro events
- hydro indices
- hydro quality
- resample
- data values
- hydro separation
- integrate
- compare
- subtract
- fill
- statistics
- hargreaves
What am I going to achieve for next week?
- Adding GUI(OAT extension and its sub-extension) in istSOS.
- Adding functionality of OAT method resampling in istSOS.
Is there any blocking issue?
- Currently, everything is going according to plan.
May 30 - June 4 (Week 1)
What did I complete this week?
- User interface design completed.
- Adding user interface implementation in istSOS api.
What am I going to achieve for next week?
- User interface implemented of OAT method resampling in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
June, 5 - June, 11 (Week 2)
What did I complete this week?
- User interface implemented completely.
- Implementation of resample method.
What am I going to achieve for next week?
- Completing functionality of resample method(OAT) in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
- Yes, I blocked on dygraph for plotting resampled data in istsos api.
June, 11 - June, 18 (Week 3)
What did I complete this week?
- Implemented resample functionality in istsos api.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Adding functionality of regularization method(OAT) in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
- Yes, I blocked on dygraph for plotting resampled data in istsos api.
June, 19 - June, 25 (Week 4)
What did I complete this week?
- Implemented regularization functionality in istsos api.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Testing on these methods.
- Documentation.
Is there any blocking issue?
- Currently, everything is going according to plan.
June, 26 - July, 2 (Week 5)
What did I complete this week?
- Testing on these methods functionality(implemented oat methods).
- Documentation.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Implementing digital-filter oat method in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
- this week my class test is going on so i am not able to work in full pace.
July, 3 - July, 9 (Week 6)
What did I complete this week?
- I have also updated the project wiki page.
- Implementing digital-filter oat method in istsos api.
- Implemented exceedance, integrate oat method in istsos api.
What am I going to achieve for next week?
- Implementing fill, statistics, data values oat method in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
July, 10 - July, 16 (Week 7)
What did I complete this week?
- I have also updated the project wiki page.
- Implementing digital-filter, data values oat method in istsos api.
- Implemented String response in Json in resample oat method.
- Implemented fill, statistics oat method in istsos api.
What am I going to achieve for next week?
- Implementing Hargreaves, Hydro events oat method in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
July, 17 - July, 23 (Week 8)
What did I complete this week?
- I have also updated the project wiki page.
- Implemented digital-filter, data values, hydro-sepration, hargreaves,fill oat method in istsos api.
- Implementing hydro-event, hydro-indicies oat method.
What am I going to achieve for next week?
- Implementing remain oat method(compare and subtract) in istsos api.
- Implement add sensor name(from raw,istSOS,csv).
Is there any blocking issue?
- Currently, everything is going according to plan.
July, 24 - July, 30 (Week 9)
What did I complete this week?
- I have also updated the project wiki page.
- Implementing hydro-event, hydro-indicies oat method.
What am I going to achieve for next week?
- Implementing result of oat methods in gui in istsos api.
- Implementing download result button in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
July, 31 - August, 6 (Week 10)
What did I complete this week?
- I have also updated the project wiki page.
- Implemented Download result.
- Implemented Show result.
What am I going to achieve for next week?
- Implementing Download result and show result all oat methods in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
August, 7 - August, 13 (Week 11)
What did I complete this week?
- Implemented Download result and show result in all implemented methods.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Complete the documentation, Cleaning and Optimising the codes.
- Implementing remain oat methods in istsos api.
Is there any blocking issue?
- Currently, everything is going according to plan.
August, 14 - August, 20 (Week 12) Final Report
Hello everyone,
My name is Rahul and this is my final report of my project.
Title: istSOS-Data Analysis and statistical tool suite Software Community: istSOS
Abstract: The goal of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticate documents using OAT library and harvesting the data from an istSOS server.
The state of the art BEFORE your GSoC: Before my project istSOS web-api didn't support Data analysis using OAT (Observation Analysis Tools).
The added value that your project brought to the software: Using OAT library now istSOS web-api easily support data analysis using implemented OAT methods and statisticat different type data. OAT is a Python package which is integrated in the istSOS through an interface exposing its features to non programmer users.
Work to be done on the project: Digital Filter- not much left, just some bug fixes Update documentation
Links:-
1 Commits : https://github.com/rahulworld/Data-analysis/commits/master
2 Wiki : https://wiki.osgeo.org/wiki/GSoC_17:_istSOS-Data_analysis_and_statistical_tools_suite
3 Images and examples : https://github.com/rahulworld/Data-analysis/tree/master/examples
4 Blog and documentation : https://rahulworld.github.io/GSoC.html
Thanks for support
Best regards, Rahul
Student's Biography
My name is Rahul Chauhan. I am 3rd year Computer Science & Engineering student at National Institute of Technology, Srinagar, India. Some more information about me can be obtained by following the link: http://rahulchauhan.net