GSoC 2019 Adding Augmented Reality (AR) support to the MapMint4ME Android Application

Introduction

Idea

This project aims to include the Augmented Reality (AR) support to MapMint4ME Android Application. This will allow the user to capture images with different objects marked on the image frame in real time. This will lead to storing more information about the location.MapMint4ME is an android application for MapMint web-service. MapMint is an internet-based Geographic Information System(GIS), designed to facilitate the deployment of Spatial Data Infrastructure(SDI). In an SDI, geographic data, metadata, tools, and the users are connected in an interactive manner in a framework so as to use the spatial information in an efficient and flexible way.

The MapMint is built on top of the ZOO-Project. The ZOO-Project is an SDI manager which gives the users the capability of building their own map and web-applications.

Project proposal

My proposal for GSoC 2019 can be found at: Proposal.

Advantages from this project

While capturing images most of the time people want to add some extra visual information with the image captured. Presently if we want to add extra features we have to take image edit image by placing the object and save the image. After addition of Augmented Reality (AR) support to the Android application, people can see the changes in realtime and capture image by placing an object exactly where they want. This improves the process of information gathering.

Link to Github repository: Repo

Timeline

6th May - 27th May

Community bonding period:

1. What I have done during this period?

• Introduced myself over the channel and shared my proposal over mailing list for suggestions
• Communicated with mentors and learned about community, working, etc. It was a great experience talking with experts in the domain
• Created a wiki page for the project "Adding Augmented Reality (AR) support to the MapMint4ME Android Application"
• Forked the repository of MapMint4ME android application Repo
• Updated wiki User page and added my personal information User
• Updated links on the wiki Google_Summer_of_Code_2019_Accepted page Link
• Joined MapMint Slack Workspace created by mentors which we will be used as a mode of communication over the GSoC 2019 period
• Environment setup of MapMint4ME android application on my personal system on Android Studio.
• Gone through the working of the application user interface and code
• Discussed with the mentors their expectations over the GSoC 2019 project period
• Mentors suggested that we could use Cloud Anchor for registering the change in water level and share those images between different devices

Week 1 - 4 (27th May - 24th June)

Coding Phase 1:

• Week 1 (27th May- 2nd June)  :

1. What I completed this week?

• Studied Cloud Anchor to add support to MapMint4ME Cloud Anchor Overview
• Tested the simple ARCore support added to MapMint4ME ARCore Test

1. What am I going to achieve for next week?

• Complete the work for Cloud Anchor support to transfer data on Google cloud and load to a different device
• Test the Cloud Anchor support

1. Is there any blocking issue?

• Android Virtual Device is being used to test the app but it's slow and sometimes doesn't even works.

• Week 2 (3rd June - 9th June)  :

1. What I completed this week?

• Added Cloud Anchor support to MapMint4ME Repo link
• Completed the work for Cloud Anchor support to transfer data on Google cloud

1. What am I going to achieve for next week?

• Test the Cloud Anchor support for different devices
• Use cloud Anchors for identifying water surface level changes.

1. Is there any blocking issue?

• I have to test the app on different devices which are presently not available.

• Week 3 (10th June - 16th June)  :

1. What I completed this week?

• Tested Cloud Anchor support for MapMint4ME on different devices
• Created a sample App to draw features with AR
• Tried adding drawing and cloud anchor support to MapMint4ME
• I have cleared all previous bottleneck for testing onto a different device

1. What am I going to achieve for next week?

• Add 3D line object to the surface and use cloud Anchors for identifying water surface level changes.
• Test the feature on different devices

1. Is there any blocking issue?

• Was unable to combine ARDrawing feature with Cloud Anchors.

• Week 4 (17th June - 23rd June)  :

1. What I completed this week?

• Created an app to test the water surface level change using cloud anchor Repo
• Tested cloud anchor java app on different devices and it runs smoothly with no crashing
• Test video link of cloud anchor java app Video link
• Here is the firebase database screenshot which includes all the anchor created for testing Screenshot

1. What am I going to achieve for next week?

• Combine cloud anchor java with MapMint4ME
• Test the feature on different devices for identifying water surface level changes.

1. Is there any blocking issue?

• MapMint4ME have different language support while combining new part I have to take care of that.

24th June - 28th June

• Evaluation Phase 1: First evaluation period. Mentors have evaluated my code written in week 1-4.

Week 5 - 8 (24th June - 22nd July)

Coding Phase 2:

• Week 5 (24th June- 30th June)  :

1. What I completed this week?

• Tried adding firebase support to MapMint4ME
• Studied string (values) for different languages supported in MapMint4ME

1. What am I going to achieve for next week?

• Remove warning and errors of Gradle
• Add Firebase support to MapMint4ME
• Study to draw and measure using AR

1. Is there any blocking issue?

• Unable to add firebase support as the Gradle file has some warning which I was unable to find the solution to remove.

• Week 6 (1st July - 7th July)  :

1. What I completed this week?

• Study to draw and measure using AR
• Setup Unity environment to add functionality to draw and measure distances
• Tried adding firebase support to MapMint4ME

1. What am I going to achieve for next week?

• Add AR-Ruler feature to the MapMint4ME
• Add Firebase support to MapMint4ME

1. Is there any blocking issue?

• Unable to add firebase support to MapMint4ME

• Week 7 (8th July - 14th July)  :

1. What I completed this week?

• Worked on Unity to implement ARScale feature
• AR Scale code completed using unity but was unable to merge with MapMint4ME
• Shifted back to work using JAVA
• Completed the code of ARScale feature for Mapmint4ME using JAVA
• Committed the written code by creating new branch over Git Repo

1. What am I going to achieve for next week?

• Increase ARScale functionality to measure multiple distances by placing multiple dots on the surface
• Test the AR Scale functionality of the app on different devices

1. Is there any blocking issue?

• No blocking issue

• Week 8 (15th July - 21st July)  :

1. What I completed this week?

• Successfully tested the AR Scale functionality of the app on different devices
• Added functionality to measure multiple distances by placing multiple cubes on the surface
• Here is a link to working of the AR Scale in the app Video
• Link of the repository with all code committed Repository
• Added firebase support to the MapMint4ME application

1. What am I going to achieve for next week?

• Start working on UI of MapMint4ME for the new options added
• Setup environment to add the feature of AR Drawing to the MapMint4ME
• Start working on the documentation of the work done

1. Is there any blocking issue?

• No blocking issue

22nd July - 26th July

• Evaluation Phase 2: Second evaluation period. Mentors evaluate my phase 2 code written in week 5 -8.

Week 9 - 12 ( 22th July - 19th August)

Coding phase 3:

• Week 9 (22nd July - 28th July)  :

1. What I completed this week?

• Understand the code of ARcore sample example app code working Code
• Studied code for UI of MapMint4ME
• Looked at previous year GSoC documentation for MapMint4ME

1. What am I going to achieve for next week?

• Test the sample AR Drawing app on different devices
• Combine the AR Drawing app code with MapMint4ME
• Find a way to test feature on Emulator

1. Is there any blocking issue?

• No blocking issue

• Week 10 (29th July - 4th August)  :

1. What I completed this week?

• Tested the ARDrawing sample app on different devices
• Link to the working of the test app Video_Link
• Integrated code of MapMint4ME and ARDrawing sample app Repo
• Steps to run app over the Emulator

  1) Create a virtual device with apk version >=27 
  2) Download arcore.apk Link
  3) Install the apk to the android virtual device by dragging it over the emulator 
  4) Restart the emulator and install MapMint4ME to the emulator

1. What am I going to achieve for next week?

• Start working on the UI code of the MapMint4ME
• Start writing final report for the project
• Test for all features together in MapMint4ME

1. Is there any blocking issue?

• No blocking issue

• Week 11 (4th August - 11th August)  :

1. What I completed this week?

• Started working and Updated the UI code of the MapMint4ME Repo
• Started writing final report for the project
• Successfully tested for all features together in MapMint4ME

1. What am I going to achieve for next week?

• Complete final report
• Final update of UI based on the review
• Complete all detailed documentation

1. Is there any blocking issue?

• No blocking issue

• Week 12 (11th August - 18th August)  :

1. What I completed this week?

• Cleaned up the code by removing unnecessary data & code
• Link to committed code repository Repo
• Completed update of UI based on the review to add under Index block
• Started working on documentation
• Completing final report for the project

1. What am I going to achieve for next week?

• Complete final report for the project
• Test UI links to activities and cleanup UI
• Complete all detailed documentation

1. Is there any blocking issue?

• No blocking issue

GSoC 2019 Final Report

Augmented Reality support is added to the MapMint4ME along with the previous features. Under Augmented Reality stream mainly three different features were added to the MapMint4ME using Google ARCore. The added features are as follows:

• Cloud Anchor: Cloud Anchors gives you the ability to create AR apps that share a common frame-of-reference, enabling multiple users to place virtual content in the same real-world location. A fundamental concept in ARCore is that of an Anchor, which describes a fixed position in the real world. The value of an Anchor's pose is automatically adjusted by ARCore as its motion tracking improves over time. Cloud Anchors are Anchors that are hosted in the cloud and can be resolved by multiple users to establish a common frame of reference across users and their devices. To share Cloud Anchor Id's among different user Google Firebase is used.

• AR Draw: AR Drawing allows the user to draw white lines in 3d space. It was initially made as a quick experiment of how to combine ARCore with openFrameworks but was expanded with native implementation in Java. It mainly has the functionality of detecting the 3d plane the user is drawing. For lines, there are three settings line width, smoothness and distance scale.

• AR Scale: Add functionality of measuring distance in realtime using an Augmented Reality ruler. The feature added can measure distance in horizontal along the plane detected or perpendicular to the surface detected. The distance measure is displayed in realtime. It also has the functionality to measure multiple distances in random directions by adding various drop points in the path of measurement.

Here is a link to the Final report of GSoC 2019. Final Report

Here is a link to a working example with a tutorial video for using each feature Example

Student's Biography

My name is Aditya Vitthalrao Chondke. I am a Computer Science graduate, pursuing my Master's degree in Geoinformatics and Natural Resources Engineering under Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India. Some more information about me can be obtained by following the link: User

Mentors

• Fenoy Gerald Fenoy Gerald
• Rajat Shinde Rajat Shinde