Geoservices REST API

This wiki page aims to collate community concerns related to the proposed acceptance of the "Geoservices REST API" becoming an OGC standard. It is being collaboratively edited, targeting completion before the end of May 2013.

Cover Letter from the OSGeo Board

Please don't edit this "Cover Letter" statement, which has been approved by the OSGeo Board.

The board of the Open Source Geospatial Foundation (OSGeo) is presenting this letter to the OGC, which highlights concerns about the "GeoServices REST API" from many people within the OSGeo community. As always, if there is anything the OSGeo board can do to help, then please let us know.

Signed: Jeff McKenna (OSGeo president), Peter Batty, Jáchym Čepický, Michael Gerlek, Anne Ghisla, Mark Lucas, Daniel Morissette, Cameron Shorter, Frank Warmerdam

Open Letter to OGC and voting members

Please don't edit this "Open Letter" statement, comments and discussion should go below.

May 2013

We, the undersigned, have concerns that approving the "Geoservices REST API" as an OGC standard, would have detrimental impacts on interoperability within the spatial industry.

We strongly urge that the proposed "Geoservices REST API", as it stands in May 2013, be rejected as an OGC standard.

People have listed different reasons for concern. They are described below.

Signed

Please add your name here if you agree with the above statement. Include name, work title (if appropriate), very brief title/involvement in OSGeo if appropriate. (Link to OSGeo profile if appropriate). You may sign as a group, such as the Project Steering Committee of XXX project if you wish, or as Your Name on behalf of YYY company.

1. Cameron Shorter, Geospatial Solutions Director at LISAsoft, core contributor & coordinator of OSGeo-Live, OSGeo Board member
2. Mark Lucas, Founding member and board of directors for OSGeo foundation, Prinicipal Scientist for RadiantBlue Technologies Inc.
3. Stephen Woodbridge, Director of iMaptools.com, Contributor and/or PSC of Mapserver, pgRouting, PAGC, and PostGIS
4. Even Rouault, Geospatial developer, OSGeo Charter Member, core contributor and PSC member of GDAL/OGR, contributor of Mapserver, PROJ.4, libgeotiff, shapelib, libtiff
5. Gerhard Triebnig, Managing Director at EOX IT Services GmbH
6. Brent Wood, Environmental Information Delivery Programme Leader, NIWA, New Zealand. OGC member, Aust/NZ OSGEO chapter member, NZOSS Council member
7. Stephan Meissl, CTO at EOX IT Services GmbH, contributor to Mapserver, PSC chair of EOxServer
8. Jeroen Ticheler, Director of GeoCat, project founder and PSC chair of GeoNetwork opensource
9. Just van den Broecke, Director at Just Objects, contributor to Heron Mapping Client, secretary of OSGeo Dutch Local Chapter, member at OpenGeoGroep
10. Milo van der Linden, member at OpenGeoGroep
11. Landon Blake, GIS Department Manager/Land Surveyor at KSN, OSGeo California Chapter Board Representative.
12. Daniel Morissette, President at Mapgears, core contributor and PSC member of Mapserver and GDAL/OGR. Former OGC TC member and involved in the implementation of several OGC WxS services specs in MapServer.
13. Bob Basques, GIS Systems Developer at the City of Saint Paul, MN. Public Works GIS (GISmo), Technical Director at SharedGeo, OSGeo Charter Member, OSGeo TCMUG local chapter member, Co-founder and PSC member of GeoMoose project.
14. Pedro-Juan Ferrer Matoses, PM at Omnium Strategic Intelligence, Spain, OSGeo Charter Member, OSGeo Spanish Local Chapter Liaison officer.
15. Bevan Rudge, Director Lucion Limited, IT Advisor at Conservation Strategy Fund, Esri client
16. María Arias de Reyna, software engineer at GeoCat, Spain, member of OSGeo Spanish Local Chapter.
17. Anne Ghisla, OSGeo Board Member, Italy, member of OSGeo Italian Local Chapter.
18. Micho Garcia, Freelance and member of geomati.co, Spain, member of Spanish Local Chapter
19. Margherita Di Leo, OSGeo Charter Member, Post-doctoral researcher at the European Commission, JRC, Italy
20. Jorge Sanz, GIS Consultant at Prodevelop, OSGeo Charter Member, OSGeo Spanish Local Chapter Member, Spain
21. Pablo Sanxiao, CTO and co-founder at iCarto, OSGeo Spanish Local Chapter Member, Spain
22. Frank Steggink, GIS software developer at Vicrea, The Netherlands, member of the Dutch Local Chapter
23. Olivier Courtin, Oslandia co-founder, core contributor or/and PSC member of Mapserver and PostGIS. OGC TC member.
24. Wladimir Szczerban, OSGeo Spanish Local Chapter Member, Spain
25. Anita Graser, GIS specialist with AIT Austrian Institute of Technology, OSGeo Charter member and QGIS team member.
26. Volker Mische, geospatial software engineer, creator of GeoCouch
27. Iván Sánchez, OSGeo Spanish Local Chapter Member, head of OpenStreetMap Spain, OpenStreetMap Foundation member, Humanitarian OpenStreetMap Team member, Spanish SDI working group member
28. Gabriel Carrión, Strategy Manager at gvSIG association
29. Sandro Santilli, OSGeo Charter Member, PostGIS and GEOS PSC member and core hacker.
30. Javier Diaz, member of Geoinquietos Bs As [1], member of the Organizing Committee FOSS4G Bs As 2013 [2]
31. Jo Cook, Consultant at Astun Technology, former Director of OSGeo, Charter Member, founder of UK Local Chapter, Deputy Chair of FOSS4G 2013
32. Francisco José Peñarrubia, CTO and co-founder at SCOLAB. Members of gvSIG Association
33. Shanmugam Ganeshkumar, Director of GeoICON, member OSGeo Malaysia Chapter
34. Barry Rowlingson, Senior Researcher, Lancaster University and Software Sustainability Institute Fellow
35. Stefan Keller, University of Applied Sciences, Rapperswil (Switzerland), Member of Swiss OSM (SOSM) and QGIS association and of organizing committees of pgConf.DE and FOSSGIS 2013, and member of eCH (e-government standards of Switzerland)
36. Andrew Bailey, OSGeo member, Astun Technology
37. Suchith Anand, OSGeo Charter member, OSGeo Education member, FOSS4G 2013 LOC member
38. Carlos Krefft, GIS software developer at CSTARS - University of Miami, OGC and OSGeo Member.
39. Stefano Costa, OSGeo member, GFOSS.it member and former board member, Ministero per i Beni e le Attività Culturali (Italy)
40. Peter Baumann, Jacobs University, OGC member, WCS.SWG chair, editor of 10+ specs (disclaimer: this is an expression of my personal opinion and not in any way endorsed by OGC)
41. Peter Batty, CTO of Geospatial Division at Ubisense, OSGeo board member, former CTO of Intergraph and GE Smallworld, Technical Committee member of OGC in its formative years c 1995-97
42. Barend Köbben, OSGEO Chartered Member, OSGeo.nl Dutch chapter treasurer, Senior Lecturer at ITC-University of Twente
43. Paolo Cavallini, Faunalia, OSGeo member, GFOSS.it member and former president, QGIS-PSC
44. FRans Thamura, Indonesia, OSGeo Indonesia, organizer]
45. Sanghee Shin, Founder and CEO of Gaia3D, OSGeo Charter Member, Representative of OSGeo Korean Chapter, Chairman of Open Source GIS Alliance Korea
46. Benni Purwonegoro,Indonesia, IT-Spatial Engineer @ Geospatial Information Agency .
47. Jachym Cepicky, Czech Republic, member of OSGeo Board of Directors
48. Pat Cappelaere, Vightel Corporation
49. Jürgen Fischer, norBIT GmbH, QGIS core developer
50. Maria Antonia Brovelli, OSGeo Charter member, OSGeo Education member, GIS Professor and Vice Rector for the Como Campus at Politecnico di Milano, Italy
51. Nacho Varela, GIS Consultant, OSGeo Spanish Local Chapter Member, Spain
52. Vasile Craciunescu, OSGeo Charter member, OSGeo Romania Local Chapter Leader, Researcher at Romanian National Meteorological Administration, Romania
53. Abbas Abdul Wahab, Asst. Director, Federal Department of Town & Country Planning, Peninsular Malaysia
54. Roy Braam, Software Engineer @ | B3Partners
55. Peteris Bruns, Latvia, GIS Consultant & Software Engineer @ | SunGIS
56. Giovanni Manghi, Portugal, Faunalia, OSGeo member, OSGeo-Portugal
57. Hugo Martins, UK, Lutra Consulting, WebGIS Developer, OSGeo-Portugal Member
58. Sidney Gijzen, The Netherlands, Researcher GIS @ Alterra, Wageningen UR
59. Miles Fidelman, US, Principal, Protocol Technologies Group, LLC
60. Puneet Kishor, OSGeo Charter Member; Geology, Univ. of Wisconsin-Madison; Creative Commons

Summary

As at May 2013, OGC members have been asked to decide whether to accept the "GeoServices REST API" as an OGC standard. This is a contentious issue, with many people arguing that introduction of the “GeoServices REST API” will have costly, far reaching, negative impacts on interoperability, and significantly tarnish the OGC's reputation as a champion of interoperability.

The key points of contention revolve around the fact that the proposed "GeoServices REST API" does not build upon or extend existing OGC standards, but rather addresses similar requirements using an alternative API. In particular, the overlap and/or duplication of existing standards is widespread: OGC's core standards of WMS, WMTS, WFS, SE/SLD, WCS, CS/W are all duplicated to a significant extent. This defeats the purpose of having standards in the first place.

Duplication of standards will likely result in a combination of the following:

1. The cost to application developers, systems integrators, testers and sponsors to support all relevant OGC standards will be substantially increased.
2. Consequently, organisations and/or applications may choose to only support one standard, or only support one standard fully.
3. Sponsors (such as governments) who require compliance with OGC standards will discover that applications don't communicate together, due to applications supporting different OGC standards that essentially do the same thing.
4. This will result in a diminished importance of OGC, as the "OGC standards" stamp of approval will not equate interoperability.
5. After a while, in order to solve interoperability issues, a respected international organisation or program will likely take the initiative to mandate one standard as the preferred standard for all agencies to follow. To date, the OGC has provided this leadership.
6. One standard taking prominance over the other will likely lead to the other being neglected or deprecated, resulting in many OGC compliant systems becoming legacy systems in the process. This should be considered an undesirable outcome for a standards organisation.

Analysis

This section is derived from an email from Adrian Custer, and has been singled out as he has provided a good technical summary of concerns: http://lists.osgeo.org/pipermail/discuss/2013-May/011667.html.

Please don't update this section. Instead, forward corrections or suggestions to Adrian.

General overview

The pros

• The OGC should be in the business of developing good standards, not of choosing which standards should be implemented.
• The proposers of the document want to make a standard and have followed all the rules of the consortium. The work of any such group of members deserves serious, good faith consideration.
• The need for an integrated suite of services using simple data, which is addressed (partially) by the document, is real. The proposed document is pushing the OGC on this issue.
• The proposed document could be useful to a number of people.
• The proposed document is not significantly more broken than the existing standards of the OGC. As an author of standards at the OGC, I know how totally impossible it is to write a good standard, so the weaknesses in the existing document seem more acceptable.

The cons

• The OGC is, de-facto, in the position of recommending the interoperable standards for geospatial services. The proposed document is not good enough, not widely enough implemented, and not publicly supported enough, to be considered at the same level as existing standards.
• Adopting a standard implies a desire to maintain the standard, but the desire to support this approach by the OGC membership is limited. The lack of collaboration on this version bodes ill for the future.
• The overlap in functionality between the proposed services and the existing services, notably with the ongoing work to modularize the existing services, is almost 100 percent. However, compatibility is low.
• There is already a published document: http://www.esri.com/library/whitepapers/pdfs/geoservices-rest-spec.pdf so there is no need for the document to be adopted as an OGC Standard merely for interoperability with the ESRI implemetation.
• The document, as a new, separate effort, repeats mistakes which were made and since solved by the other services.
• The document focuses on the past (notably with backwards compatibility and use of only GET/POST) not on the future.
• The document needs a comprehensive editorial rewrite. (see at end)

The problem for me, is that both simple answers are bad.

A simple acceptance of the standard would introduce a new set of 'OGC approved' open services. The OGC approval might enable governments to buy a XXXX-new-name-here-XXXX solution instead of a W*S or a S*S solution. (On that, I am inclined to let them make their own mistakes.) The path forwards towards harmonizing the services is unclear. There is little good will towards the standard on behalf of the membership. Fixing this document in addition to fixing the W*S services will be a pain.

Simply rejecting the solution would be bad for the OGC. It would place the OGC in the position of picking winners and losers in the standards business. It would mean that the OGC is stuck on the project of fixing the W*S standards to meet some nebulous future functionality without having any path to get there. It would discourage innovation and progress.

Is there a third way?

Well, actually, there is. The natural consequence of either decision is a renewed commitment towards trying to build this thing that we want, an integrated suite of simple services built using simple, well defined resources, accessible and usable using the core HTTP verbs, and discoverable through following links. That's the way forwards and why I have wasted a chunk of my life on WMS-Next.

A Quick Critique of the proposed standard document

The name

The document currently uses a name that is confusing in all discussions on the matter, both internally at the OGC and externally.

(Postnote: the Standards Working Group proposing the document agreed to accept proposals for new names; if they can find a good alternative, they may adopt it.)

The design

The document does not show an overall, coherent design to the service suite. This is mostly because the design reflects a suite that [initially] evolved piecemeal at ESRI rather than one designed from the start to meet the needs of a broad swath of users. This is also due because the experts at the OGC did not contribute to an improved design for a multitude of reasons.

The design of the service which does exist is focused on yesteryear. As an apology for the existing design, section~6.2.2 of part 1, Core says that the only HTTP verbs used are GET and PUT because "the API was originally developed several years ago" but that's okay because it enables support for "Microsoft Silverlight" and "Adobe Flex" two standards which are dying rapidly. In 2013, a new standard ought to be focused on the next 10 years not the past ten.

The writing

The document is poorly written.

The document is poorly introduced

The document should start with a discussion of the overall design of the suite of services and what each individual service provides within that suite. This would make it easy for a user to understand the focus of the suite. (The note between requirements~3 and requirement~4, stating that the spec makes no requirements on clients, is really part of the scope of the document and should be front and center.)

The document instead starts with a whole section on REST.

1. Scope

The GeoServices REST API provides a standard way for web clients to communicate with geographic information system (GIS) servers based on Representational State Transfer (REST) principles.

The scope claims that the document identifies resources and a way to use "structured URLs" to exchange those resources between clients and services. If that were true, I would expect

(1) a list of resources at least as an example, and

(2) an example of the structure and structuring principles of the URLs.

Section~6 stats this process. Section~6.1 introduces the concept of a root URL for each service and mixes in that that root URL is also a catalog and that the hierarchy offers resources. This all happens in a single, overly compact paragraph. This section needs an editorial rewrite for clarity separating

(1) the end point,

(2) the catalog,

(3) the URL hierarchy,

(4) the resources, and

(5) interaction of clients with the URLs through the exchange (get/post) of the resources to the different URLs.

Unfortunately, section~6 then decays into a meandering discussion of REST which mentions the buzzword a lot but does not discuss the aspects of REST design that have been met by the suite and those that have not. If REST is a form of design which offers certain benefits, OGC standards should be discussing the benefits achieved not the buzz word. To what extent are the resources cachable and how do clients and servers agree on that? To what extent are the endpoints and actions on those endpoints discoverable, and how? Those are the issues of RESTful design.

Section~6.3 discusses Resources but immediately then states:

Each resource has a unique URL. Resources MAY support parameters.

which is a discussion of the URL hierarchy. (And 'Resources' do not 'support parameters'! Services may 'handle requests containing parameters' if that is what is meant.) Then the standard introduces a new idea 'Controller Resources' which are able to 'edit' and 'query'!? A web search does not reveal any formal discussion of 'Controller Resource' so this new type of resource probably deserves some kind of formal introduction.

Section~6.3.3 is not really about resources but the fact that the URL hierarchy is not discoverable from the Resources themselves.

Section~6.3.4 does give a list of resources, but since 'image' is not part of it but will be used by the Service for Map Images, I surmise that the list is of the things 'we have designed a JSON representation for' rather than a list of 'the resources identified and exchanged in the service suite' which is what I expect in an introductory design.

This introduction therefore needs a substantial editorial rewrite to be effective in its goal of explaining what the suite of standards actually does.

This criticism is not limited to the core document.

The Service for Maps document also starts without setting out the five core elements (end pt, catalog?, URL hierarchy, resources, and interactions).

Section~6 of the Service for Maps part, in the middle of the second paragraph, introduces a new concept, with capitalized letters:

The Map Service Root resource includes a tables property ...

without giving us any clue what this is. It "provides basic information" (perhaps it is a data structure?) but it also "supports several operations" (maybe it's a magical 'Controller Resource'). To me, this just reads as 'the authors of the spec are confused and imprecise in their written language'.

Then, in the figure of 'Resources,' there is actually a resource called 'All layers and tables'. Sorry, but this reader needs some real help to understand what that resource might be.

The document requirements are poor

The very first requirement, Requirement~1 of the Core document, states:

Req 1 If a request uses the HTTP GET method, the processing of the request by the service SHALL be safe and idempotent as specified by RFC 2616, 9.1.

This is a *ridiculous* requirement, completely untestable, and explicitly what the Mod Spec should be preventing with its attempt to require all injunctive language to be accompanied by formal tests. The formal test indeed shows that the requirement is untestable.

Inspect the documentation of the implementation to identify, if requests specified in the GeoServices REST API standard that support the HTTP GET method, are all safe and idempotent, i.e. free of side- effects, as specified in HTTP (RFC 2616, section 9.1). Alternatively, ask the developer of the the service.

Note that it is not possible to run automated tests against a service to verify conformance with the requirement (and IETF provides no test for conformance with HTTP either). In the absence of such tests, a statement of the developers of the service is sufficient indication that the requirement was addressed.

So essentially this requirement is met if the developers say 'yeah, we thought about it.' In other words, requirement 1 of the whole shebang is a recommendation. Contrast this with requirement 2 which has a nice test defined for it and is a good, solid requirement.

Again, broadening the critique to the Services for Maps, its first requirement states:

Req 1 The Map Service Root resource SHALL accept requests that conform to the URI template in Table and use any HTTP method identified in the same table.

but the test is then completely irrelevant. At the protocol level, all web services 'shall accept' request messages, so we are not discussing that. A functional test for this 'shall accept' is to construct all valid requests and see that the service never returns an exception to those requests. Even worse, the first test is a catch all 'please test the service here' test. This is *not* a well written specification of the tests for the injunctions of the Service for Maps.

The requirements need a complete, systematic review to see that they work, they are testable, and have good tests.

The document has numerous other issues which need resolution

• There is no discussion of the overlap of the f= parameter and of the HTTP format header. What happens if they conflict?
• The lack of full integration of the four core HTTP verbs, notably DELETE and PUT, exists merely for backwards compatibility not for forwards progress.
• Requirement~3 blocks forwards progress and experimentation with no explanation as to its importance.
• This goes on and on. This is the work for an EDITOR. I am not interested in doing that work.

Further Concerns

--- DRAFT ____

Please add concerns as bullet points below. Try to be concise. Where appropriate, link to external web pages (such as email achieves)

Political Concerns

• Adopting the standard will expose the OGC to a strong suspicion of acting as a rubber stamp organization under ESRI weight, and will be detrimental to its recognized position as a reference organization for geospatial standards.
• It is a dubious practice that a standardization organisation promotes competing standards, without explicitely obsoleting (or at least recommending) some of them. How is a newcomer to the industry supposed to select the appropriate standard if several ones share the same scope : WFS or GeoServices REST API Feature Service, WMS or GeoServices REST API Map Service, etc. ?

Commercial Concerns

• Promoting standards from an existing implementation made by a single vendor leads to an obvious bias in competition.
• Supporting multiple overlapping standards greatly reduces usability while it increases complexity and cost of development and maintenance.
• Many SME's have invested in supporting existing OGC standards in their products. They will be forced to choose the standards they support (and can explain), resulting in decreased interoperability, confusion and frustration for clients.
• Confusing customers with new, overlapping OGC standards will lower the credibility of companies and of OGC, reducing business opportunities.

Technical Concerns

• The Geoservices REST API overlaps in large proportion with existing OGC standards such as WMS, WCS, WFS, WMTS, CSW, with no effort made to reconcile with those standards.
• The standardization of WKT for Spatial reference systems is unfortunately currently quite weak in OGC standards. Geoservices REST API is tied with ESRI's version of WKT, which is not properly specified in the Geoservices REST API documents, and is known to be incompatible with other OGC documents, which will lead to a larger confusion. See the following comment for more details on this issue.
• The Geoservices REST API is not particularly RESTful - it's a thinly disguised service call, not an address space for RESTful objects that can be operated on.
• At least as far as "imagery" is concerned, OGC standards arguably are substantially more mature, powerful, flexible, and modular then the ESRI "Geoservice REST API" Part 6 (and some design principles suggest that scalability may be hampered as well):
  • data model:
    • the ESRI "Geoservice REST API" appears constrained to 2-D imagery, plus optional time stamps. OGC has established a unified coverage model which fully supports n-D spatio-temporal data. It allows use and exchange of coverages between different services, such as WCS, WCPS, WPS, and SWE.
    • OGC coverages support both regular and irregular grids; the ESRI "Geoservice REST API" supports only regular grids, more specifically: only rectified grids with quadrilateral pixels.
    • the ESRI "Geoservice REST API" lacks support for temporal data; it only offers timestamps, measured in milliseconds; this is inconvenient for users and immediately excludes, e.g., geological dates. OGC has established uniform handling of horizontal, vertical, and temporal coordinate reference systems (CRSs), following a deep consensus process with GIS science and backwards compatible with EPSG. The ESRI "Geoservice REST API" specific way of handling coordinates is not known to support this, thereby excluding appropriate timeseries handling in remote sensing, air traffic, MetOcean, etc.
    • OGC coverages provide a concise, versatile model for supporting different binary formats; the ESRI "Geoservice REST API" supports only very few selected 2-D formats, excluding, e.g., JPEG2000, NetCDF, HDF, etc.
    • the ESRI "Geoservice REST API" lacks a clear model of their data structures, it can be deduced only implicitly from the operation mechanics.
  • service model:
    • The ESRI "Geoservice REST API" Part 6 lacks conciseness, thereby opening up ways for implementations that are not interoperable. For developers of alternative implementations this may mean they have to acquire ESRI licenses for finding out the intended behavior.
    • Functionality in the ESRI "Geoservice REST API" appears randomly chosen, with no clear concept visible; this burdens implementers while still leaving holes of functionality. For example, this functionality appears restricted to mapping applications and does not easily extend into other domains.
    • It has been said that the ESRI "Geoservice REST API" can be seen as a "wrapper around OGC W*S" services. This is not true for WCS (and WCPS), at least: the ESRI "Geoservice REST API" Part 6 is too poor in functionality and too different in mechanics to accomplish this.
  • In summary, the ESRI "Geoservice REST API" Imaging part is at a technological level where WCS departed from some 5 years ago. Inconciseness of the specification at large will make it difficult for third parties to come up with interoperable implementations. The components making up the ESRI "Geoservice REST API" provide natural blocks assignable to the matching SWGs. As for Part 6 of the ESRI "Geoservice REST API", if to become a standard it needs to be discussed in the WCS.SWG for harmonization, clarification, and improvement.

Methodological Concerns

• No public response (nor private to the authors of the comments) has been made to the various comments sent on the OGC Requests mailing list in [July 2012] and [August 2012] during the 30 day public comments period.
• The Geoservices REST API can not be amended (other than editorial changes in the specification document), because of a requirement of backward compatibility with ESRI implementation. Consequently, the standard is unlikely to improve, or its evolution will be only lead by ESRI.
• OGC standards normally require interoperability experiments and a richer process to ratify a standard such as this one. No explanation has been forthcoming as to why a simplified process is appropriate in this case.
• ESRI/OGC have specifically rejected requests to openly share their justifications document being selectively sent to OGC voters.

References

• Call for comments on GeoServices REST API: http://www.opengeospatial.org/standards/requests/89
• Email archive of OSGeo discussions about GeoServices REST API: http://lists.osgeo.org/pipermail/discuss/2013-May/thread.html
  • Adrian Custer's eloquent, unbiased, concise summary of the technical issues: http://lists.osgeo.org/pipermail/discuss/2013-May/011667.html
  • "Is OGC Loosing its way?", letter to OGC Voters, from OGC Interoperability Movement Team Leaders, http://lists.osgeo.org/pipermail/discuss/2013-May/011632.html