GSoC 17: istSOS-Data analysis and statistical tools suite

From OSGeo
Revision as of 22:52, 25 June 2017 by Rahulworld (Talk | contribs) (Updated wiki page)

Jump to: navigation, search
This is where I'll be posting information about istSOS-Data analysis and statistical tools suite GSoC 2017 project to the OSGeo community.

Introduction

istSOS2 Main Page

istSOS is sensor data management tool that allows collection, maintenance and publishing of monitoring observations using the Open Geospatial Consortium (OGC) Sensor Observation Service (SOS)standard.

OAT(Observation Analysis Tools) can be used as a pre-processor for calibration observations, integrating the creation of observations for calibration directly from sensor time-series. The tool consists in an expandable Python library of processing methods and an interface integrated in the QGIS FREEWAT plug-in which includes a large number of modelling capabilities, data management tools and calibration capacity.

OAT library is an expandable Python library that can easily interact with istSOS.

The goal of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticate documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).

Idea

istSOS2 Added extenstion Data analysis

istSOS has gone through a lot of improvements since 2009. This has somehow affected the simplified interaction with measurements and SOS service settings.

The aim of my project primarily is to create OAT extension in RESTFull Web api and OAT extension having data analysis and statistical tools for istSOS which will be used to automate the creation of statisticat documents using OAT library and harvesting the data from an istSOS server and then publishing the result on Open Data Portals(CKAN).

Project

Title: istSOS-Data analysis and statistical tools suite.

Student: Rahul Chauhan

Mentors: Massimiliano Cannata, Milan Antonovic.

Details

Development process can be monitored at the following links:

Implementation details

OAT (Observation Analysis Tools)

OAT methods

OAT is designed to facilitate the import, analysis and visualization of time-series data and the use of these data to support model construction and advanced model calibration.

Observation Analysis Tool was developed to enable time-series visualisation, pre-processing of data for model development, and post-processing of model results. Observation Analysis Tool can act as a pre-processor for calibration observations, and will be expanded to incorporate its processing capabilities directly into the calibration process. The tool consists in an expandable Python library and in an interface integrated in the QGIS FREEWAT plug-in which include a large number of modelling capabilities, hydro-chemical data management tools and calibration capacity. The tool has been extensively used and tested in different european institutions, to collect a number of indications to drive the future development.

  • The OAT library implements two class:
    • 1. Sensor class that is designated to handle time series data and metadata.
    • 2. Method class which is designated to represent a processing method.
  • OAT takes advantage of the PANDAS (McKinney, 2010),NUMPY and SCIPY (Van der Walt et. al. 2011) packages.
  • Most of the currently available OAT.Method objects are based on TSPROC processing capabilities. They enable the analysis of raw data (resampling, regularization, data interpolation, fitting, filling,validation and data quality assessment); decomposition or filtering of time­series (low, medium, highfrequencies); aggregation and exceedance­time calculation, summary statistics and period statistics,integration and hydrograph separation among others.

Schedule

Dates Task
Week 1 (May, 30 - June, 4) User interface implementation.
Week 2 (June, 5 - June, 11) Implementation of resample method(OAT) in istSOS api.
Week 3 (June, 12 - June, 18) Testing resample(OAT method) with istSOS api.
Week 4 (June, 19 - June, 25) Implementation of regularisation time series analysis method(OAT) in istSOS api.
Week 5 (June, 26 - July, 2) Phase 1 Evaluation Test the working of these operations.
Week 6 (July, 3 - July, 9) Implement OAT.methods i.e. , interpolation, digital filter.
Week 7 (July, 10 - July, 16) Implement OAT.methods i.e. filling, validation, data quality assessment.
Week 8 (July, 17 - July, 23) Implement remain OAT.methods.
Week 9 (July, 24 - July, 30) Phase 2 Evaluation Testing on OAT method in istSOS api.
Week 10 (July, 31 - August, 6) Implement CKAN (Open Data Portals) plugin.
Week 11 (August, 7 - August, 13) Implement publish datasets ready to be harvested using CKAN.
Week 12 (August, 14 - August, 20) Testing, Debugging, Cleaning and Optimizing the codes.
Week 13 (August, 21 - August, 27) Take a week to scrub code, write a few test cases and test them, improve the current documentation.
Week 14 (August, 27 - August, 29) Final evaluation Buffer period. ( Final Week. Prepare documentation for final submission.)

Reports

May 4 - May 14

May 15 - May 29

What did I complete this week?

  • Designed GUI for all Process time series methods.
  • Also Designed GUI for
    • Add Time Series
    • Process Time Series
    • Compare Sensor
    • Manage Sensor
  • This week, I’ve been indentifying all the process time series methods in OAT(freewat), I mean:
    • digital filter
    • exceedance
    • hydro events
    • hydro indices
    • hydro quality
    • resample
    • data values
    • hydro separation
    • integrate
    • compare
    • subtract
    • fill
    • statistics
    • hargreaves

What am I going to achieve for next week?

  • Adding GUI(OAT extension and its sub-extension) in istSOS.
  • Adding functionality of OAT method resampling in istSOS.

Is there any blocking issue?

  • Currently, everything is going according to plan.

May 30 - June 4 (Week 1)

What did I complete this week?

  • User interface design completed.
  • Adding user interface implementation in istSOS api.

What am I going to achieve for next week?

  • User interface implemented of OAT method resampling in istSOS api.

Is there any blocking issue?

  • Currently, everything is going according to plan.

June, 5 - June, 11 (Week 2)

What did I complete this week?

  • User interface implemented completely.
  • Implementation of resample method.

What am I going to achieve for next week?

  • Completing functionality of resample method(OAT) in istSOS api.

Is there any blocking issue?

  • Currently, everything is going according to plan.
  • Yes, I blocked on dygraph for plotting resampled data in istsos api.

June, 11 - June, 18 (Week 3)

What did I complete this week?

  • Implemented resample functionality in istsos api.
  • I have also updated the project wiki page.

What am I going to achieve for next week?

  • Adding functionality of regularization method(OAT) in istSOS api.

Is there any blocking issue?

  • Currently, everything is going according to plan.
  • Yes, I blocked on dygraph for plotting resampled data in istsos api.

June, 19 - June, 25 (Week 4)

What did I complete this week?

  • Implemented regularization functionality in istsos api.
  • I have also updated the project wiki page.

What am I going to achieve for next week?

  • Testing on these methods.
  • Documentation.

Is there any blocking issue?

  • Currently, everything is going according to plan.

Student's Biography

My name is Rahul Chauhan. I am 3rd year Computer Science & Engineering student at National Institute of Technology, Srinagar, India. Some more information about me can be obtained by following the link: http://rahulchauhan.net