GSoC 17: istSOS-Data analysis and statistical tools suite
| This is where I'll be posting information about istSOS-Data analysis and statistical tools suite GSoC 2017 project to the OSGeo community. |
Introduction
istSOS is sensor data management tool that allows collection, maintenance and publishing of monitoring observations using the Open Geospatial Consortium (OGC) Sensor Observation Service (SOS)standard.
OAT(Observation Analysis Tools) can be used as a pre-processor for calibration observations, integrating the creation of observations for calibration directly from sensor time-series. The tool consists in an expandable Python library of processing methods and an interface integrated in the QGIS FREEWAT plug-in which includes a large number of modelling capabilities, data management tools and calibration capacity.
OAT library is an expandable Python library that can easily interact with istSOS.
The goal of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticate documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).
Idea
istSOS has gone through a lot of improvements since 2009. This has somehow affected the simplified interaction with measurements and SOS service settings.
The aim of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticat documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).
Project
Title: istSOS-Data analysis and statistical tools suite.
Student: Rahul Chauhan
Mentors: Massimiliano Cannata, Milan Antonovic.
Details
Development process can be monitored at the following links:
- https://summerofcode.withgoogle.com/projects/#6301175977607168 - Project GSoC 2017 application Link
- https://github.com/istSOS/istsos2 - Project Main Repo. Link
- https://github.com/rahulworld/istsos2 - Forked Github Repo. Link
- https://github.com/rahulworld/Data-analysis - Project GitHub testing Repo. Link
Implementation details
OAT (Observation Analysis Tools)
OAT is designed to facilitate the import, analysis and visualization of time-series data and the use of these data to support model construction and advanced model calibration.
- The OAT library implements two class:
- 1. Sensor class that is designated to handle time series data and metadata.
- 2. Method class which is designated to represent a processing method.
- currently available OAT.method:
- Resampling:
- OAT takes advantage of the PANDAS (McKinney, 2010),NUMPY and SCIPY (Van der Walt et. al. 2011) packages.
- Pandas use upsample/interpolation time series data to a higher frequency and interpolate the new observations.
- Pandas use downsample/Aggregation time series data to a lower frequency and summarize the higher frequency observations.
- In the case of upsampling, care may be needed in determining how the fine-grained observations are calculated using interpolation. In the case of downsampling, care may be needed in selecting the summary statistics used to calculate the new aggregated values.
- Resampling:
Schedule
| Dates | Task |
|---|---|
| Week 1 (May, 30 - June, 4) | User interface implementation. |
| Week 2 (June, 5 - June, 11) | Implementation of resample method(OAT) in istSOS api. |
| Week 3 (June, 12 - June, 18) | Testing resample(OAT method) with istSOS api. |
| Week 4 (June, 19 - June, 25) | Implementation of regularisation time series analysis method(OAT) in istSOS api. |
| Week 5 (June, 26 - July, 2) Phase 1 Evaluation | Test the working of these operations. |
| Week 6 (July, 3 - July, 9) | Implement OAT.methods i.e. , interpolation, digital filter. |
| Week 7 (July, 10 - July, 16) | Implement OAT.methods i.e. filling, validation, data quality assessment. |
| Week 8 (July, 17 - July, 23) | Implement remain OAT.methods. |
| Week 9 (July, 24 - July, 30) Phase 2 Evaluation | Testing on OAT method in istSOS api. |
| Week 10 (July, 31 - August, 6) | Implement CKAN (Open Data Portals) plugin. |
| Week 11 (August, 7 - August, 13) | Implement publish datasets ready to be harvested using CKAN. |
| Week 12 (August, 14 - August, 20) | Testing, Debugging, Cleaning and Optimizing the codes. |
| Week 13 (August, 21 - August, 27) | Take a week to scrub code, write a few test cases and test them, improve the current documentation. |
| Week 14 (August, 27 - August, 29) Final evaluation | Buffer period. ( Final Week. Prepare documentation for final submission.) |
Reports
May 4 - May 14
- created github repo. https://github.com/rahulworld/Data-analysis
- Installed all devlopment requsites.
- Tried to understand the code flow of istSOS and OAT.
- Created Project wiki.
May 15 - May 29
What did I complete this week?
- Designed GUI for all Process time series methods.
- Also Designed GUI for
- Add Time Series
- Process Time Series
- Compare Sensor
- Manage Sensor
- This week, I’ve been indentifying all the process time series methods in OAT(freewat), I mean:
- digital filter
- exceedance
- hydro events
- hydro indices
- hydro quality
- resample
- data values
- hydro separation
- integrate
- compare
- subtract
- fill
- statistics
- hargreaves
What am I going to achieve for next week?
- Adding GUI(OAT extension and its sub-extension) in istSOS.
- Adding functionality of OAT method resampling in istSOS.
Is there any blocking issue?
- Currently, everything is going according to plan.
May 30 - June 4 (Week 1)
What did I complete this week?
- User interface design completed.
- Adding user interface implementation in istSOS api.
What am I going to achieve for next week?
- User interface implemented of OAT method resampling in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
June, 5 - June, 11 (Week 2)
What did I complete this week?
- User interface implemented completely.
- Implementation of resample method.
What am I going to achieve for next week?
- Completing functionality of resample method(OAT) in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
- Yes, I blocked on dygraph for plotting resampled data in istsos api.
June, 11 - June, 18 (Week 3)
What did I complete this week?
- Implemented resample functionality in istsos api.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Adding functionality of regularization method(OAT) in istSOS api.
Is there any blocking issue?
- Currently, everything is going according to plan.
- Yes, I blocked on dygraph for plotting resampled data in istsos api.
June, 19 - June, 25 (Week 4)
What did I complete this week?
- Implemented regularization functionality in istsos api.
- I have also updated the project wiki page.
What am I going to achieve for next week?
- Testing on these methods.
- Documentation.
Is there any blocking issue?
- Currently, everything is going according to plan.
Student's Biography
My name is Rahul Chauhan. I am 3rd year Computer Science & Engineering student at National Institute of Technology, Srinagar, India. Some more information about me can be obtained by following the link: http://rahulchauhan.net