Difference between revisions of "LIDAR Format Letter"
Line 2: | Line 2: | ||
April 2015. | April 2015. | ||
− | We, the undersigned, are concerned that the current interoperability between LiDAR applications, through use of the open "LAS" format, is being threatened by ESRI's introduction and promotion of an alternative "Optimised LAS" proprietary format. This is of concern | + | We, the undersigned, are concerned that the current interoperability between LiDAR applications, through use of the open "LAS" format, is being threatened by ESRI's introduction and promotion of an alternative "Optimised LAS" proprietary format. This is of significant concern since the fragmentation of the LAS format will lead to reduced interoperability between applications and organisations, and introduce vendor lock-in. |
We request that: | We request that: | ||
− | # The OGC initiate the formalisation of an open standard for storing LIDAR data, and OGC | + | # The OGC initiate the formalisation of an open standard for storing LIDAR data, and that sponsors of OGC help to prioritise the development of this open LiDAR standard. |
− | # ESRI support the OGC in their [http://www.opengeospatial.org/ogc/vision mission] "to advance the development and use of international standards and supporting services that promote geospatial interoperability." In particular, ESRI join the OGC in consolidating | + | # ESRI support the OGC in their [http://www.opengeospatial.org/ogc/vision mission] "to advance the development and use of international standards and supporting services that promote geospatial interoperability." In particular, ESRI join the OGC in consolidating an Open Standard for use of LiDAR data. This might include proposing ESRI's "Optimised LAS" as an Open Standard to remove any technical or legal hurdle in use of "Optimised LAS" as an Open Standard. A simple test to determine if "Optimised LAS" can be used as an Open Standard would be if "Optimised LAS" can legally be implemented by Open Source software such as LibLAS or LASzip. |
# Users and sponsors of LiDAR data, publicly state their preference for the use of an open LiDAR format over closed when selecting software and services. | # Users and sponsors of LiDAR data, publicly state their preference for the use of an open LiDAR format over closed when selecting software and services. | ||
− | # The [http://www.asprs.org/LD-Division/LAS-Working-Group.html LAS Working Group (LWG)], who are | + | # The owners of LAS open format [http://www.asprs.org/LD-Division/LAS-Working-Group.html LAS Working Group (LWG)], who are part of [http://www.asprs.org/ American Society for Photogrammetry and Remote Sensing (ASPRS)], publicly request ESRI avoid incorporating the use of the word "LAS" in describing a proprietary format. |
=Signed= | =Signed= | ||
Line 24: | Line 24: | ||
[http://en.wikipedia.org/wiki/Lidar LiDAR] (Light Detection And Ranging) is a form of high precision range measurement unit much like a radar system that uses laser light instead of electromagnetic waves. The standard LiDAR product is a 3D point cloud that can be conceptualized as a series of point measurements representing distance between the sensor to a returned emission. A common storage format for these point cloud data is the LAS format. | [http://en.wikipedia.org/wiki/Lidar LiDAR] (Light Detection And Ranging) is a form of high precision range measurement unit much like a radar system that uses laser light instead of electromagnetic waves. The standard LiDAR product is a 3D point cloud that can be conceptualized as a series of point measurements representing distance between the sensor to a returned emission. A common storage format for these point cloud data is the LAS format. | ||
− | For over a decade (since 2003), there has been a common format for storing LiDAR data, the [http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html “LAS” format], and open source libraries have been developed to read, write and process these LAS files: [http://live.osgeo.org/en/overview/liblas_overview.html libLAS] and [http://rapidlasso.com/LASlib LASlib]. These libraries have been incorporated within many LiDAR applications, allowing read/write access to a common exchange format, and resulting in full interoperability between the applications. There is also an open source compression algorithm for the LAS format called [http://laszip.org LASzip] that many LiDAR data portals use to compress LAS into smaller files for faster download. [http://laszip.org LASzip] is not officially endorsed by the ASPRS | + | For over a decade (since 2003), there has been a common format for storing LiDAR data, the [http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html “LAS” format], and open source libraries have been developed to read, write and process these LAS files: [http://live.osgeo.org/en/overview/liblas_overview.html libLAS] and [http://rapidlasso.com/LASlib LASlib]. These libraries have been incorporated within many LiDAR applications, allowing read/write access to a common exchange format, and resulting in full interoperability between the applications. There is also an open source compression algorithm for the LAS format called [http://laszip.org LASzip] that many LiDAR data portals use to compress LAS into smaller files for faster download. [http://laszip.org LASzip] is not officially endorsed by the ASPRS yet is a de-facto standard for compressed LiDAR. |
==LAS limitations== | ==LAS limitations== |
Revision as of 21:15, 9 April 2015
Open Letter for the need for Open Standards in LiDAR
April 2015.
We, the undersigned, are concerned that the current interoperability between LiDAR applications, through use of the open "LAS" format, is being threatened by ESRI's introduction and promotion of an alternative "Optimised LAS" proprietary format. This is of significant concern since the fragmentation of the LAS format will lead to reduced interoperability between applications and organisations, and introduce vendor lock-in.
We request that:
- The OGC initiate the formalisation of an open standard for storing LIDAR data, and that sponsors of OGC help to prioritise the development of this open LiDAR standard.
- ESRI support the OGC in their mission "to advance the development and use of international standards and supporting services that promote geospatial interoperability." In particular, ESRI join the OGC in consolidating an Open Standard for use of LiDAR data. This might include proposing ESRI's "Optimised LAS" as an Open Standard to remove any technical or legal hurdle in use of "Optimised LAS" as an Open Standard. A simple test to determine if "Optimised LAS" can be used as an Open Standard would be if "Optimised LAS" can legally be implemented by Open Source software such as LibLAS or LASzip.
- Users and sponsors of LiDAR data, publicly state their preference for the use of an open LiDAR format over closed when selecting software and services.
- The owners of LAS open format LAS Working Group (LWG), who are part of American Society for Photogrammetry and Remote Sensing (ASPRS), publicly request ESRI avoid incorporating the use of the word "LAS" in describing a proprietary format.
Signed
Name, Affiliation(s), Optional comment on interest in Open LIDAR format
- Suchith Anand, Geo for All, committed to Open Principles in Geo Education and Policy.
- Martin Isenburg, founder of rapidlasso GmbH and creator of LASzip, LAStools, and PulseWaves.
- Cameron Shorter, GeoSpatial Director at LISAsoft, Core contributor and coordinator of OSGeo-Live, Contributor to numerous OGC testbeds, technical lead on a range of previous Australian and New Zealand Open Government initiatives.
- Stefan Keller, founder and director of Geometa Lab; researcher in GIS, databases, open (government) data and interoperability; maintainer of GeoConverter; contributor to open source software (GDAL/OGR, QGIS).
- Patrick Hogan, NASA World Wind Project Manager, committed to the US National Spatial Data Infrastructure principles, one being "to make geographic data more accessible to the public" NSDI. Open standards are essential for this.
Background
About LiDAR
LiDAR (Light Detection And Ranging) is a form of high precision range measurement unit much like a radar system that uses laser light instead of electromagnetic waves. The standard LiDAR product is a 3D point cloud that can be conceptualized as a series of point measurements representing distance between the sensor to a returned emission. A common storage format for these point cloud data is the LAS format.
For over a decade (since 2003), there has been a common format for storing LiDAR data, the “LAS” format, and open source libraries have been developed to read, write and process these LAS files: libLAS and LASlib. These libraries have been incorporated within many LiDAR applications, allowing read/write access to a common exchange format, and resulting in full interoperability between the applications. There is also an open source compression algorithm for the LAS format called LASzip that many LiDAR data portals use to compress LAS into smaller files for faster download. LASzip is not officially endorsed by the ASPRS yet is a de-facto standard for compressed LiDAR.
LAS limitations
Quoting Paul Ramsey:
- LAS format is not without its drawbacks:
- While it is a binary format and does not waste any space unnecessarily, neither does it apply any compression to the data it stores. That’s not good for archival use.
- Also, LAS stores points in scan order, so accessing any particular chunk of points involves reading the whole file. That’s not good for random access.
- Clearly there is a little more work to be done. Can LAS be improved? In fact, it already has been:
- An open source compression library LASzip can apply 20:1 lossless compression to LAS files, making them great for archival purposes.
- Other LAS users have experimented with re-ordering points in a LAS or LASzip file to allow random access to internal chunks of the LIDAR point cloud.
- Basically, making LAS smaller and faster is not rocket science, and if the work were incorporated into libLAS then the whole LIDAR community could leverage it together, and the user community would only have one file type to interchange.
Closed "Optimized LAS" format from ESRI
ESRI has announced the release of an "Optimized LAS" format which is claimed to provide faster access and smaller file sizes (similar to the open source LASzip format).
This announcement created a outburst of vocal protest in the LiDAR community [1 2 3 4 5 6].
One year after releasing "Optimized LAS", ESRI released a free Windows DLL that is claimed to convert between LAS and "Optimized LAS".
Note there is a distinct difference between "free" and "open". Unless ESRI release under an "open" license, libLAS and related products <Martin asks: IS THE FIRST PART TRUE???> will be legally prevented from incorporating ESRI's reader/writer into their products, or from fixing any underlying bugs or performance limitations which may exist in ESRI's product.
In particular, the "Optimized LAS" format is neither published, nor available under any open license, which provides both technical as well as legal barriers for other applications reading and/or writing to this proprietary format. This creates a vendor lock-in scenario which is contrary to the principles of the Open Geospatial Consortium, the OSGeo Foundation, and many government IT procurement policies.
OGCs efforts so far to enable Open standards in LiDAR
The Open Geospatial Consortium (OGC) have indicated their interest in developing a standard LiDAR format.
Carl Reed (formerly on OGC staff) provided the following information on theOGC's previous efforts to work towards enabling Open Standards in LiDAR:
- "Over the last 8 or so years, the OGC approached ASPRS at least twice regarding LAS and working with ASPRS to bring LAS into the OGC for consideration as both a Best Practice and an OGC standard. OGC member Rick Pearsall worked hard to try to make this work. Rick worked for NGA, was active in the OGC, and was also the Standards Committee chair at ASPRS. Rick and Carl tried and failed. These attempts go back to at least 2007.http://www.asprs.org/a/society/divisions/ppd/ppd_meetings/2007springppdreport.pdf. More recently, Carl Reed had an email dialogue with Lewis Graham about bringing LAS into the OGC as an OGC Best Practice. Carl thought progress was being made, but then for some reason all communication stopped."
Scott Simmons (Executive Director, Standards Program) has explained the OGC's continuing interest in pursuing point cloud encoding standards, including a member-initiated mechanism to extend LAS data with OGC-standard XML content.
- The OGC invites interested members who wish to work on this effort to please contact Scott Simmons (Executive Director, Standards Program E-mail : ssimmons@opengeospatial.org ) to register their interest and discuss details. OGC will also be holding an ad hoc session at the OGC's next Technical Committee meeting in Boulder, CO, USA in early June to bring together all interested from all sectors (government, industry, academia) for this and plan next steps.
History: LAS and "Optimized LAS"
Martin Isenburg, one of the leading experts in LIDAR formats, and creator of LASzip, LAStools, and PulseWaves explains the history LAS and "Optimized LAS":
- Summary:
- The LAS format has been used successfully for over a decade.
- Within last couple of years, ESRI has added native LAS support into ESRI products.
- ESRI initially opened dialogue suggesting to embrace open source LAZ, the de-facto community standard for compressed LIDAR.
- Simultaneously ESRI developed their own proprietary format presenting the community with a 'fait-accompli' with "Optimized LAS" or "zLAS".
- The community reached out and offered to work with ESRI on numerous occasions to avoid format fragmentation.
- ESRI's closed zLAS format is very similar in design and performance to the open LAZ format.
- 1998: Original LAS format definition
- The original development of the LAS format started in 1998 according to Lewis Graham. The effort was at first led by pioneers of the LiDAR industry until the format was donated to the ASPRS. Since then the LAS Working Group (LWG) has been maintaining the LAS format, guiding it from the initial LAS 1.0 version until today's LAS 1.4 version. This effort created an incredible successful open data exchange standard for discrete LiDAR points that is nowadays supported by practically every LiDAR software.
- 2011: ESRI joins LAS Working Group
- ESRI did not join the LWG of the ASPRS until rather late, the 24th of August in 2011 to be exact. At that time ESRI was not really a "player" in the LiDAR market as they did not have much support for LiDAR in any of their products. That was to change soon as they were planning to add LAS as a native type in ArcGIS 10.1.
- Shortly before that, in June 2011, Martin Isenburg received a personal message from the ESRI development team: "I have a question about LAS compression. I’m evaluating some potential enhancements and support for LAZ is one of them. Compression time and amount is impressive." When inquiring two month later about the status of LAZ integration Martin was told: "We’re in beta and working to finish the current release. I very much doubt LAZ will make it in because it’s too big a change at this time. So, we’ll be considering, for the following release, what we want to do regarding compression and spatial indexing."
- June 2012: ArcGIS 10.1 includes [closed] LAS Dataset file
- In June 2012 ESRI released ArcGIS 10.1 and introduced the "LAS Dataset file (*.lasd)" that would group collections of files into one logical unit. That was a useful addition but unfortunately ESRI did not share this improvement with others despite several private and eventually public requests.
- 2012/2013: Collaboration Discussions
- However, there were encouraging follow-ups from the ESRI team about compression in January 2012: "I would be interested in having a more in depth meeting with you on better understand the great work you are doing on with LAS and how we can possibly better partner on this." and again in December 2012: "If you have time, I'd like to set some time aside Tuesday afternoon to meet with <an important person> at the ESRI booth. Does 2pm sound okay?" The meeting went well and it looked as if ESRI was going to embrace the de-facto open standard because shortly after the meeting Martin got word that: "I hear from <an imporant person> that it was a success and that he had a good meeting with you. I wanted to see if we can have a telephone discussion on Wed or Thu this week related to the potential of incorporating LAZ into ArcGIS."
- Then the first signs of hesitation showed. First there were legal issues raised in February 2013: "Currently the legal aspects are being reviewed. They were having some issues separating the LPGL aspects from LASzip from the remainder of LAStools etc." and then in April 2013 concerns about the code were made: "We have started to look into the integration of LAZ, but come across some issues. We don’t want to copy files from the other package, mix and match or hack around. We were hoping for an API to stream points out of a LAZ file as well as write LAZ files. There should be a simple code sample for that?" and - following up on that - in June 2013 Martin was told "I see value in LASzip becoming a de-facto standard that provides compression to the LAS format. If this to happen then it needs to be bundled together with appropriate reference implementations, examples and documentation as a standard and so ensure that issues as defined earlier in mail to not occur."
- June 2013: ESRI asked to sponsor clean LASzip DLL
- At that time LASzip was an open source project without sponsorship. The original funding from USACE that had turned LASzip from an academic prototype into an industry strength compression engine had long run out. So right then Martin asked ESRI to become a sponsor to create the kind of clean, well-documented, and easy-to-use API for LASzip that ESRI was after. The answer was: "ESRI often helps in the financing of Open Source projects and I could foresee ESRI possibly helping in LASzip. [...] If you are interested in promoting LASzip as such a standard and doing the required work then I would recommend you put together a proposal and I can look to get Esri as a sponsor."
- Immediately Martin proposed the following to ESRI: "I hereby propose to write an easy to use open source DLL wrapper for LASzip that will make it easier to integrate LASzip in a standardized manner into other software products such as ArcGIS, LP 360, or Terrasolid. This will come with example code on how to use the DLL for reading and writing LAZ files and inlude a few compressed example files. I hereby ask ESRI to provide funding for this effort without imposing any limitations on the produced DLL API."
- July 2013: ArcGIS 10.2 extended to include indexing
- In July 2013 ESRI added an improvement to their LAS file handling with the release of ArcGIS 10.2, namely spatial indexing to speed up area-of-interest queries. Just like the LAS Dataset file (*.lasd) this was done with a closed format. Their new proprietary (*.lasx) files have a practically identical look and feel as the open source spatial indexing (*.lax) files introduced a year earlier and published at the European LiDAR Mapping Forum (ELMF) in December 2012.
- After more requests (in particular for USDA's FUSION) for an easier interface to LASzip Martin decided not to wait for funding from ESRI but "code it forward". On July 30th of 2013 Martin announced the release of a clean well-documented, and easy-to-use DLL with the expectation that developers at ESRI would use it to add read and write support for LAZ to their next release of ArcGIS.
- December 2013 : ESRI discovered to be secretly developing proprietary LAS format
- All this changed suddenly in December of 2013 when several LAStools users contacted Martin with suspicions that ESRI may be working on their own proprietary LAS compression. Martin broke the news as soon as it became evident that ESRI had used the time it needed to resolve "legal issues" and "code problems" to put together their own proprietary compressed format with near-identical performance and functionality to LASzip. ESRI's LAS compression scheme (that has since been dubbed the "LAZ clone") comes with one additional feature, that LASzip is obviously lacking: it is ESRI-controlled technology.
- Caught in an instant heated-debate (see comments) from the LiDAR community, ESRI released an official FAQ a few days later to confirm the rumors. From day one Martin has taken a very outspoken stance, informing the community about ESRI's actions and lobbying against the "LAZ clone". Martin tried to keep his critical stance "light-hearted" in order not to burn bridges with ESRI. Again and again he engaged in direct personal communication with decision makers and product managers of ESRI pleading to work with the LiDAR community and avoid format fragmentation.
- Martin's core argument for resolving this to everybody's benefit was that - coincidentally - a natural break was happening in the LAS format with the introduction of the new LAS 1.4 point types. Martin repeatedly outlined a detailed plan of how a joint development of LASzip for LAS 1.4 between rapidlasso and ESRI could exploit this natural break in the LAS format to accomplish two things at once: (1) add those unspecified technical additions that ESRI had hinted at being the reason for creating the own closed "Optimized LAS" instead of embracing the existing open LASzip formt and (2) extend the LASzip compression scheme to handle the new point types introduced with the LAS 1.4 specification. As there was no pressing need at the time to handle LAS 1.4 Martin had delayed the extension of LASzip to the new LAS 1.4 point types to make sure a cooperation with ESRI would remain a viable option. Unfortunately Martin's proposal fell on deaf ears at ESRI.
- April 2014: Last plea to avoid format fragmentation
- In a final attempt to convince ESRI management, Martin released an April Fools' Day press release to demonstrate how positively the community would react to news of ESRI embracing an open format. This prank press release was entirely true except for the claim that ESRI had agreed to such a cooperation. Many people fell for the "good news" and their jubilant reactions (see comments) left no doubt about the sentiment on this issue within the LiDAR communitiy.
- This seemingly positive press release was widely carried by geospatial news outlets that had not covered the controversy surrounding ESRI's release of "Optimized LAS". This has limited the exposure of the controversy over the new LiDAR format by ESRI, especially evident on LiDAR news that reported every advance of "Optimized LAS" [1 2 3 4 5 6 7] without any mention of the controversy.
- October 2014: Announcement of [open] LAS Compatibility mode
- Martin was able to win Digital Coast, NOAA Coastal Services Center as a gold sponsor for LASzip and announced in October 2014 the development of LAS 1.4 compatibility mode that would allow addition of immediate support for the new point types to LASzip without closing the door on a potential cooperation with ESRI for a joint LAS 1.4 compressor by utilizing a recoding of new point types into old ones.
- November 2014: ESRI announces proprietary extension to LAS 1.4
- The final signal from ESRI that they were not willing to cooperate on an open compression standard came in November 2014 with the announcement that ESRI had added their own extension for the new LAS 1.4 point types to "Optimized LAS". This ended all hopes of devising a joint compressor and avoid format fragmentation by exploiting this "natural break" in the LAS format as Martin had pleaded with ESRI earlier.
- Since then, ESRI has been agressively promoting their closed format. By calling it "Optimized LAS" and "zLAS" ESRI blurs the distinction between the open LAS standard and their own proprietary format. This is likely to miss-lead novice and future users endangering many years of standardization work. Although the ASPRS holds a clearly spelled-out copyright on the name "LAS", as of April 2015, the LAS Working Group (LWG) has yet to make an official statement regarding ESRI's exploitation of the "LAS" name to promote their new closed LiDAR format.
Value of Standards
The importance of Open Standards is described is most government IT policies. For instance, the United Kingdom policy states:
- … Government assets should be interoperable and open for re-use in order to maximise return on investment, avoid technological or supplier lock-in, reduce operational risk in ICT projects and provide responsive services for citizens and business. This should also lower barriers to entry for more diverse sources of IT services, including citizens and SMEs. [1]
The value of Open Standards have been described in numerous national studies on the effects of standards on economic growth.
- ... the national studies demonstrate that standards have a positive influence on economic growth due to the resulting improved diffusion of knowledge. The contribution of standards to the growth rate in each country is equivalent to 0.9% in Germany, 0.8% in 0.3% in the UK and 0.2% in Canada. [2]
References
<Cameron: The links below should be referenced under WIKI REF tags from above.>
- Paul Ramsey provides background to LAS vs Optimised LAS, http://boundlessgeo.com/2014/01/lidar-format-wars/
- Running commentary by Martin Isenburg, author of LASlib, http://rapidlasso.com/2015/02/22/lidar-las-asprs-esri-and-the-laz-clone/
- Earlier comment from Martin Isenburg, http://rapidlasso.com/2014/11/06/keeping-esri-honest/
- First call-to-action by Martin Isenburg, http://rapidlasso.com/2013/12/30/new-compressed-las-format-by-esri/
- ESRI Announces "Optimised LAS", http://blog.lidarnews.com/esri-announces-las-compression/
- ESRI description of "Optimised LAS", http://www.lidarnews.com/content/view/10214
- Discussion background on this topic at Geo for All list , http://lists.osgeo.org/pipermail/ica-osgeo-labs/2015-March/001225.html
- ↑ All about Open Source – An Introduction to Open Source Software for Government IT, Version 2.0, United Kingdom Cabinet Office https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/78959/All_About_Open_Source_v2_0.pdf
- ↑ Prof. Dr. Knut Blind, Prof. Dr. Andre Jungmittag, Dr. Axel Mangelsdorf "The Economic Benefits of Standardization", DINN, 2000. Retrieved March 2015.