VistSOS Data Visualization Framework

This wiki will be in continous change specially during the summer 2016. If you have any suggestion, complain or idea that you want to share, please write me felipe07 at gmail dot com.

Welcome. This is the wiki of Vistsos: The istSOS Data Visualization Framework project.

Proposal summary

In order to extent the available characteristic offered by the istSOS server, I propose the use of an open source Javascript library for the implementation of VistSOS: the istSOS Data Visualization Framework. The proposed solution shall allow users of the istSOS ecosystem to configure, customize and include a wide set of chart, bar, grid and map visualization types in their websites. The framework will offer complete support for the connection of the visualization widgets with multiple data-exchange formats like CSV, TSV and JSON. Considering the real-time nature of the data sources the framework will support the generation of real-time visualization widgets specially crafted to improve the readiness to display data of this kind. The framework shall be implemented with robustness, reliability and customization as its core development principles. Recommended practices for software development like Version Control Systems use and continous testing will be followed.

Implementation details

• There are several candidates libraries for the implementation of the framework, many of them based on Javascript. Considering the goal and requirements of the project, one suitable candidate for the implementations of the framework can be the D3.js library which offers an important set of data visualization types both interactive and non-interactive.

• Besides the fact that D3.js is subject to the terms of an open license like BSD-3, two important points in favor of D3.js is its well documented API and its capacity of rendering of real-time data. For a detailed comparison of charting frameworks please refer to: [1].

• In order to improve the compatibility with the majority of browsers, standardized languages like HTML and CSS should be properly used. The proposed library D3.js support the generation of graphs in SVG format, supported by the majority of browsers that includes an SVG rendering engine. The manipulation of the browser’s DOM supported by D3.js is also tested against Firefox, Chrome (Chromium), Safari (WebKit), Opera and IE9 [2].

• The multiple support of data-interchange formats like CSV, TSV or JSON should be implemented with any pre-processing routine needed to normalize the incoming data. However, many charting frameworks support by default this formats and D3.js is not the exception.

Schedule

There is an initial schedule for the project which should be re-defined with the tutors during next weeks. Provisionally, I would like to share the activities and milestones defined by me chronologically ordered (also subject of change during coming weeks):

• Project planning with tutor:
  • Timeline adjustments
  • Definition of charts to be implemented (divided in 3 groups)
  • Definition of the technical limitations of the framework

• First prototype implementation:
  • Real-time data unsupported
  • Non-interactive widgets
  • Minimum customizable options: Color, size and position.
  • Implementation of
  • 1st group of charts
  • Hard-coded data

• Prototype evaluation

• Addition of feature:
  • Processing of CSV data
  • Processing of TSV data

• Prototype evaluation

• Addition of feature:
  • Processing of JSON data

• Prototype evaluation

• Second prototype implementation:
  • Implementation of 2nd group of charts
  • Interactive widgets
  • Customizable options: Color, size, position and scale

• Prototype evaluation

• Third prototype implementation:
  • Implementation of the 3rd group of charts
  • Real-time data supported

• Prototype evaluation

• Prototype evaluation

• Final product presentation

Competitors

Proprietary applications like: Arc2Earth have similar functionality. Several (mostly proprietary) generators for Google Earth tiles were mentioned on http://www.ogleearth.com/.

Microsoft Research project MapCruncher also generates tiles, but with metadata only for their MS Virtual Earth.

Zoomify could be now used as tiling program for SuperOverlay in GoogleEarth with free ZoomifyerEZ and generator ZoomifyKML (which I created as a part of my Master Thesis).

What new functionality this project brings

• The framework will provide a wide set of interactive and non-interactive data visualization charts, grids, plots and maps able to encode data coming from a network of sensors, external databases or files.

• The framework will allow the user to customize the color, scale, size or position of any available widget.

• Real-time data visualizations shall be available.

• The framework will support the processing of incoming data represented as CSV, TSV and JSON.

Who will use results of this project

Several organizations and individuals who handle processing of maps.

Institutions which will use the 'gdal2tiles' utility from this project, as I am cooperating with them:

• Moravian Library in Brno, Czech Republic
• Historical institute of Czech Academy of Sciences
• Czech National Library
• National Library of Scotland
• University of Connecticut Library (with their grant "Building a Globally Distributed Historical Sheet Map Set of Austro-Hungarian Topographic Maps, 1877-1914", cooperating with New York Public Library, University of Connecticut Library, and Library of the American Geographical Society. It is a grant about building a scalable workflow and toolset for map libraries to add cartographic content to the Web.)

Student's Biography

My name is Klokan Petr Přidal, I have finished Master degree in Applied Informatics at Masaryk University in Brno, now I am in the first semester of postgraduate study of Cartography at Czech Technical University in Prague. My Master thesis subject was "Processing and Digital Publishing of Historical Documents" (PDF of thesis in Czech).

I am familiar with open-source projects. I have experience with Linux/Solaris system administration. I like Python language, but I have also used other languages (C language, PHP). I have supplied some patches into Gnumeric (#330129, #334257). I founded a http://mplayerosx.sf.net/ project and I also won a competition in programming application for Mac using http://dictosx.sf.net/ (Objective-C). My primary desktop is Gnome and Linux, but I also use Mac, Windows and Solaris. I did programming for all of these platforms (mostly smaller projects).

I am interested in historical cartography, I have several presentations about old maps and free software. I made a czech web page about it at http://www.staremapy.cz/ and registered domain http://www.oldmapsonline.org/ (which is not in use yet).